As a publisher of a Blogger blog, besides publishing articles, you have to know how those articles are being read.
Just as knowing who is reading your blog, you need to know who is indexing your blog. The Google Search Engine, which feeds 2/3 of the known major search engines, provides Google Webmaster Tools, so we can monitor how well our blogs are being indexed.
If you're going to use Google Webmaster Tools / Search Console effectively, you need to know how to interpret the reports.
Specifically, you need to know what reports show problems needing attention, and what reports advise us of problems avoided.
Representing the latter, we have a message in the list of URLs restricted by robots.txt, which can be found from a link in "Overview", or from "Diagnostics" - "Web Crawl". Recently, we see email from the Search Console team.
Indexed, though blocked by robots.txt
Note the carefully worded advice.
This list, when run against a blog which has a lot of labels, will be rather long. This will cause panic in the hearts of many. Fortunately, this is unnecessary panic.
The keyword here is
This shows us a label search that is blocked, intentionally by Blogger, so the search engines won't follow it.
Were the search engines to index a label search, they would be indexing the same posts in the blog, that they find from the sitemap. They would then penalise both the sitemap indexed content - and the label search indexed content - in their indexing, for "duplicate content".
With "duplicate content" penalties applied, your posts appear lower in the search results - and your blog gets less new visitors.
So, when you see "... /search/label/ ... URL restricted by robots.txt" in your access report, don't panic. This restriction is to your advantage.
Just as knowing who is reading your blog, you need to know who is indexing your blog. The Google Search Engine, which feeds 2/3 of the known major search engines, provides Google Webmaster Tools, so we can monitor how well our blogs are being indexed.
If you're going to use Google Webmaster Tools / Search Console effectively, you need to know how to interpret the reports.
Specifically, you need to know what reports show problems needing attention, and what reports advise us of problems avoided.
Representing the latter, we have a message in the list of URLs restricted by robots.txt, which can be found from a link in "Overview", or from "Diagnostics" - "Web Crawl". Recently, we see email from the Search Console team.
Note the carefully worded advice.
This means that Index coverage may be negatively affected in Google search results.With a Blogger blog, "may be" should be "actually, is not".
http://blogging.nitecruzr.net/search/label/Search Engines URL restricted by robots.txt
This list, when run against a blog which has a lot of labels, will be rather long. This will cause panic in the hearts of many. Fortunately, this is unnecessary panic.
The keyword here is
/search/label/
This shows us a label search that is blocked, intentionally by Blogger, so the search engines won't follow it.
Were the search engines to index a label search, they would be indexing the same posts in the blog, that they find from the sitemap. They would then penalise both the sitemap indexed content - and the label search indexed content - in their indexing, for "duplicate content".
With "duplicate content" penalties applied, your posts appear lower in the search results - and your blog gets less new visitors.
So, when you see "... /search/label/ ... URL restricted by robots.txt" in your access report, don't panic. This restriction is to your advantage.
Comments
and this contained in robots.txt is normal?
User-agent: *
Disallow: /search
Noindex: /feedReaderJson
See What's In My "robots.txt" File?.
"Disallow: /search" line in default robot.txt is doing this trick.
http://ricocoffeeshop.blogspot.com
When you setup a sitemap for your blog, the posts are indexed as the sitemap is processed. Indexing the posts by tag / label would give the same posts, that are already indexed through the sitemap. The search engines would see this as duplicate content.
Knowing that you know ...nothing is the first step to appreciating when you learn something.
Your posts with nokia are restricted, intentionally.
If your posts with that label are not being indexed in Google search, it's because the posts aren't being publicised properly. Indexing your posts by the labels won't help you.
The label search restriction is applied by Blogger, universally. Leave it.
A page with 1 story (post?) will be indexed using the page URL, and using the URL for the 1 story (1 post?). There simply will be no indexing using any label searches.
The latter "solution" will leave your blog without labels, and is completely unnecessary.
I had noticed that "Label"was contained in many of my restrictions.
At that time I had one of those "view screens in my header" the ones that show pictures with post snippets superimposed(It was a third party, Bloggerized WordPress theme.
What I also noticed at that time, was that the Label restrictions matched the posts in the viewer screen(I think five posts only).
My actual labeled posts(Many), were not showing as restricted.
Due to this(And a number of issues with that third party template) I changed templates.
Strangely, I then had no restrictions showing at all until changing over to a custom domain through blogger.
Now I have 66 restricted urls.
Though, thankfully due to your insightfulness, I am not pushing the panic buttons.
Now I just need to work out why my Google Analytics stopped working since I swapped over to the custom domain.
Thanks again,
Daniel
Thanks!!
Google search is still showing 100+ https://www.theamazondealer.com/search/label/... results, but thanks to you i have made change in robots.txt and google webmasters page..
you are the man!! :)
Label searches have a different URL structure. This lead to indexing under two different URLs when labels were first released - and people saw their page rank drop because of duplicated content penalties.
If your blog is properly sitemapped, you don't need label search indexing.