Skip to main content

Large Blog Sitemaps Are Broken, And Lack Content

Several owners of large blogs (500 posts and up) are reporting problems with indexing, in Blogger Help Forum: Get Help with an Issue.
I have over 2500 posts, in my blog. Each sitemap page, which should contain 500 posts, shows only 150 posts. This reduces my indexed posts tremendously!
This blog owner did nothing to cause this problem.

Upon investigation, we have discovered that both blogs using the classic feed based sitemap, and the current automatically generated sitemap, are missing sitemap contents, equally.

Right now, sitemaps for blogs with over 500 posts - and a paged sitemap - are missing significant amounts of content.

The problem affects blogs using the classic sitemap, and the new sitemap.

Large blogs cannot be properly indexed, with the sitemaps containing only 150 entries, instead of 500 entries, per page. Both blogs which use the classic blog feed based sitemap, and blogs which use the new Blogger generated sitemap, appear to be equally affected.

The problem appears to involved both blogs published to BlogSpot, and to custom domains.

The effects of the problem can be easily seen, in Search Console reports.

The problem can be observed in the Search Console - Crawl - Sitemaps displays, where the owner will notice a significantly smaller number of indexed posts. Examine the pages in the sitemap.

Or, you may examine the blog posts newsfeed sitemap, using a text browser. Then, search for "<title·type='text'>".


Load a "500 posts" feed segment, in a text browser.



Search for "<title·type='text'>".

Right now, you'll get 151 hits - 1 for the blog, 150 for 150 posts.

That's 150 posts, out of 500. What is that doing, to the blog reputation?


This blog has just under 2,400 posts published. Right now, Search Console shows 1,157 Submitted, with 1,388 Indexed.


This blog has just under 2,400 published posts. 1,157 submitted is a significant drop.

Blogs with more than 2,500 posts (and a fully populated sitemap) are similarly affected.


This is a significant impact, on traffic to large blogs.

With the sitemap capable of hosting 2,500 posts for submission and indexing, this is a significant drop in indexing - and probably, in search engine reputation.

The problem has been reported to Blogger Support. If your blog is affected, and you wish to state the URL of your blog here, I will pass it on to Blogger.
(Update 11/28 1:00): Blogger Engineering is now offering a Blogger generated sitemap ("sitemap.xml"), with up to 20 pages of 150 entries / page. People using sitemaps based on blog feeds may have to add sitemaps, if feed segments are limited to 150 posts / segment.

Comments

Problem found. We're preparing a fix.
M. Fadhly said…
Same problem .. >.<
caraycaray.blogspot.com

We're not using the feeds to generate a sitemap, but we have over 10K posts organized by label, and the feeds for each label are cutting off after 150 posts.
I've noticed as well but figured it was something I had done.

stlouisrenewableenergy.blogspot.com
Tom Ferrio said…
http://www.progressive-charlestown.com/ (progressive-charlestown.blogspot.com)

9,230 posts
1,131 URLs submitted
1,120 URLs indexed

Thanks.
Maggie MacLean said…
I have the same problem:

http://www.womenhistoryblog.com/
Kiran Kumari said…
Am I right
Sitemap: http://www.staffnews.in/sitemap-pages.xml [for pages sitemap]
Sitemap: http://www.staffnews.in/sitemap.xml [for 20 sitemaps from http://www.staffnews.in/sitemap.xml?page=1 to http://www.staffnews.in/sitemap.xml?page=20 --- cover 3000 post]
Sitemap: http://www.staffnews.in/sitemap.xml?page=21
Sitemap: http://www.staffnews.in/sitemap.xml?page=22
Sitemap: http://www.staffnews.in/sitemap.xml?page=23
Sitemap: http://www.staffnews.in/sitemap.xml?page=24
Sitemap: http://www.staffnews.in/sitemap.xml?page=25
Sitemap: http://www.staffnews.in/sitemap.xml?page=26
Sitemap: http://www.staffnews.in/sitemap.xml?page=27
Sitemap: http://www.staffnews.in/sitemap.xml?page=28
Sitemap: http://www.staffnews.in/sitemap.xml?page=29
Sitemap: http://www.staffnews.in/sitemap.xml?page=30
Sitemap: http://www.staffnews.in/sitemap.xml?page=31
Sitemap: http://www.staffnews.in/sitemap.xml?page=32
Sitemap: http://www.staffnews.in/sitemap.xml?page=33
Sitemap: http://www.staffnews.in/sitemap.xml?page=34
Sitemap: http://www.staffnews.in/sitemap.xml?page=35
Sitemap: http://www.staffnews.in/sitemap.xml?page=36
Sitemap: http://www.staffnews.in/sitemap.xml?page=37
Sitemap: http://www.staffnews.in/sitemap.xml?page=38
Sitemap: http://www.staffnews.in/sitemap.xml?page=39
Chuck Croll said…
Hi Kiran,

Thanks for asking the question.

If your blog is over 3,000 posts, it appears that you will need more sitemaps, yes. That is a rather harsh workaround, no?

Popular posts from this blog

Custom Domain Migration - Managing The Traffic

Your blog depends upon traffic for its success.

Anything that affects the traffic to your blog, such as any change in the URL, affects the success of your blog. Publishing the blog to a custom domain, like renaming the blog, will affect traffic to your blog. The effects of the change will vary from blog to blog, because of the different traffic to every different blog.Followers. People who find your blog because of recommendations by other people.Search engines. Robotic processes which methodically surf your blog, and provide dynamic indexing to people who search for information.Subscribers. People who read your content from their newsfeed reader, such as the dashboard Reading List.Viewers. People who read your content from their browser.No two blogs are the same - and no two blogs will have the same combinations of traffic sources.

Stats Components Are Significant, In Their Own Context

One popular Stats related accessory, which displays pageview information to the public, is the "Popular Posts" gadget.

Popular Posts identifies from 1 to 10 of the most popular posts in the blog, by comparing Stats pageview counts. Optional parts of the display of each post are a snippet of text, and an ever popular thumbnail photo.

Like many Stats features, blog owners have found imaginative uses for "Popular Posts" - and overlook the limitations of the gadget. Both the dynamic nature of Stats, and the timing of the various pageview count recalculations, create confusion, when Popular Posts is examined.