Wednesday, July 09, 2008

Google Webmaster Tools And Label Searches

As a publisher of a Blogger blog, besides publishing articles, you have to know how those articles are being read.

Just as knowing who is reading your blog, you need to know who is indexing your blog. The Google Search Engine, which feeds 2/3 of the known major search engines, provides Google Webmaster Tools, so we can monitor how well our blogs are being indexed.

If you're going to use Google Webmaster Tools effectively, you need to know how to interpret the reports.

Specifically, you need to know what reports show problems needing attention, and what reports advise us of problems avoided. Representing the latter, we have a message in the list of URLs restricted by robots.txt, which can be found from a link in "Overview", or from "Diagnostics" - "Web Crawl".
http://blogging.nitecruzr.net/search/label/Search Engines URL restricted by robots.txt


This list, when run against a blog which has a lot of labels, will be rather long. This will cause panic in the hearts of many. Fortunately, this is unnecessary panic.

The keyword here is
/search/label/
This shows us a label search that is blocked, intentionally by Blogger, so the search engines won't follow it.

Were the search engines to index a label search, they would be indexing the same posts in the blog, that they find from the sitemap. They would then penalise both the sitemap indexed content - and the label search indexed content - in their indexing, for "duplicate content". When this happens, your posts appear lower in the search results - and your blog gets less new visitors.

So, when you see "... /search/label/ ... URL restricted by robots.txt" in your access report, don't panic. This restriction is to your advantage.

>> Top

33 comments:

Ananda said...

I was about to panick, but found the "/search/label/" on my warning messages. Thanks a bunch! (Great blog by the way! ... now I am going to look for something on changing a domain from Google Apps to blogger here to see if you can help me again. :-))

Laurie said...

Oh My Gosh, thank you so much for your help. You have been the only one that I found that was able to answer this issue and responded so quickly.

Patt said...

Thank you it'really helpful..I was very nervous that this might cause of limit indexing and dropped traffic.

top said...

So, this means that it is okay if there's crawler errors regarding to /search/label?

and this contained in robots.txt is normal?
User-agent: *
Disallow: /search
Noindex: /feedReaderJson

Chuck said...

Hey Top,

See What's In My "robots.txt" File?.

Vinit said...

Oh good.......I was about to panic, but seems like
"Disallow: /search" line in default robot.txt is doing this trick.

Rico said...

Excellent information...i wish I would understand better SEO...but I seem to get along better with food...yall invited to visit me.

http://ricocoffeeshop.blogspot.com

Owlman said...

I'm still a little confused. I thought that the tags were good and helped the Goolgle bots to categorize your post, so why would they then be blocked?

Chuck said...

Owlman,

When you setup a sitemap for your blog, the posts are indexed as the sitemap is processed. Indexing the posts by tag / label would give the same posts, that are already indexed through the sitemap. The search engines would see this as duplicate content.

AJ @ A Little Bit Nutty said...

Thanks for explaining that. I had no clue what to do. Now I know...nothing.

Chuck said...

AJ,

Knowing that you know ...nothing is the first step to appreciating when you learn something.

andy in china said...

Cheers mate, was worried abit then....kept getting alot of error messages...

akumaukerja said...

oh God, thanks!I very worry and panic when I check my sitemap. but now no more, because I already know any posts from you.Thank you very much,..!

wasay said...

But here my problem is that, all my posts with label nokia is restricted and all my posts with label nokia are disappeared from google search results. what's wrong i don't understand, few days back i labeled my posts again for categorizing my blog and for easy navigation to visitors. please can you help me...............

Chuck said...

Wasay,

Your posts with nokia are restricted, intentionally.

If your posts with that label are not being indexed in Google search, it's because the posts aren't being publicised properly. Indexing your posts by the labels won't help you.

The label search restriction is applied by Blogger, universally. Leave it.

best7android.com said...

Yes i was panicked just 2 minutes ago for the last couple of days. And then see your blog. It seems that this blog helps me on clearing another google releated confusion few days ago. Greateful to the writer. Special thanks to him/her.

Willy in Phuket said...

Sorry I'm new and still very confused, so it mean that a page with 1 story will never be indexed? what would be the proper way to post stories in order to have them on google? I would really appreciate if you would take a bit of your time and look at http://phuket101.blogspot.com and tell me if there is hope for individual pages... I put so much effort in it.

Chuck said...

No Willy, it does not mean that.

A page with 1 story (post?) will be indexed using the page URL, and using the URL for the 1 story (1 post?). There simply will be no indexing using any label searches.

Ari Ibnu Umar said...

thanks...

engglina said...

Thank God, You've brought me to this blog. I almost do not want to write anymore on my blog. Thank you, sir, you've become a blessing for us.

A Proper Bostonian said...

Thanks, this is the first thing I've read about this that was written clearly and made sense to a newbie like me.

Women Love Style said...

I'm a little confused. If i do a site: search on google and look at omitted results all my labels are listed. Does this mean for some strange reason Google is now indexing them? My site was performing well but in the last week the majority of my posts have dropped out of the top hundred when they were on page one previously? Any ideas what's going on?

Colorado Adventurer said...

I am getting the restricted by robots.txt for all of my site and not getting any information from webmaster. Should I not have labels at all, or is something else causing it?

Chuck said...

You have to look at the detail in the "restricted by robots.txt" message. If the reference is to "/label/search", this is the cause. If this is the cause, you have to either ignore the message, or stop using labels.

The latter "solution" will leave your blog without labels, and is completely unnecessary.

Noble Sounds said...

Thank you for sharing this!

Daniel said...

Thanks very much for clearing that up for me.(I think it's Chuck, yes)?

I had noticed that "Label"was contained in many of my restrictions.
At that time I had one of those "view screens in my header" the ones that show pictures with post snippets superimposed(It was a third party, Bloggerized WordPress theme.
What I also noticed at that time, was that the Label restrictions matched the posts in the viewer screen(I think five posts only).
My actual labeled posts(Many), were not showing as restricted.
Due to this(And a number of issues with that third party template) I changed templates.
Strangely, I then had no restrictions showing at all until changing over to a custom domain through blogger.
Now I have 66 restricted urls.

Though, thankfully due to your insightfulness, I am not pushing the panic buttons.

Now I just need to work out why my Google Analytics stopped working since I swapped over to the custom domain.

Thanks again,

Daniel

Picofly said...

Thank you for your elaboration on this issue. It has helped in clearing doubts. Thank you!

Dawn Conklin said...

Thank you for this post! I have been on Blogger since 2007 and never really thought too much about the robot restrictions until submitting a site map and it came up that no urls were in the index. I started to panic thinking I had a big problem!! I did a search and found your reply on the Google forum and came here to read your blog. Thank you again for clearing the air for us!

ET said...

Awesome. Now I know

Erika said...

This is the only site I've found with clear and concise and helpful info. You've helped me so much in the past few days. I didn't even realize I had to do all this stuff just to get my blog noticed. My site traffic has jumped a lot.
Thanks!!

Bloggered said...

Boy.. I had been panicking with google search results of my site https://www.theamazondealer.com..

Google search is still showing 100+ https://www.theamazondealer.com/search/label/... results, but thanks to you i have made change in robots.txt and google webmasters page..
you are the man!! :)

artikelnyadipo said...

So, that's why 'Label' tags are not getting index by google webmaster tools. Could you explain me what is the different between Label(in blogger) and tag (in wordpress)?

Chuck Croll said...

Arti,

Label searches have a different URL structure. This lead to indexing under two different URLs when labels were first released - and people saw their page rank drop because of duplicated content penalties.

If your blog is properly sitemapped, you don't need label search indexing.