Skip to main content

Blogger Magic - The "Robots.txt" File

Some new blog owners spend time examining the various settings provided by Blogger, in the dashboard.

Some dashboard settings inspire questions, in Blogger Help Forum: Learn More about Blogger. As helpers there, we know from experience that some questions, asked and answered there, will lead to later questions in Blogger Help Forum: Get Help with an Issue.

The "Robots.Txt" file is one feature which inspires these types of questions.
I was updating my settings on Blogger, and I discovered some settings that I didn't understand. What do I add for "Custom robot text" and "Custom robot header tag"?
When I was young, my mother used to provide advice "That is a well enough!" - as the proper answer, to this question, is to "Leave well enough alone!".

The "Robots.txt" file is a collection of various settings - and the dashboard "Search preferences" wizard provides useful options, for Blogger blog owners.

You may edit "robots.txt", if you wish - but be mindful of possible consequences.

When used properly, "Robots.Txt" provides us several possibilities. That said, we should heed the warning.
Warning! Use with caution. Incorrect use of these features can result in your blog being ignored by search engines.
The section "Crawlers and indexing" is a dashboard feature which should probably be left alone by 99% of all Blogger blog owners - except in specific, documented examples.

All "robots.txt" entries are carefully designed, by Blogger / Google Engineers.

Some portions of "Robots.Txt" are maintained by various well documented Blogger and Webmaster Tools features.
Other sections of "Search preferences" have similar value. Unless you understand "Robots.Txt" functionality, however, you should leave "Crawlers and indexing" alone.

If you are not familiar with the settings, it's best to not play with them.

Leave the magic spells, to the magicians and wizards. See Google Developers - Webmasters: Robots.txt Specifications, for details.

Work on publishing blog content - informative, interesting, and unique. Learn more, as you publish your blog.

Comments

Popular posts from this blog

Custom Domain Migration - Managing The Traffic

Your blog depends upon traffic for its success.

Anything that affects the traffic to your blog, such as any change in the URL, affects the success of your blog. Publishing the blog to a custom domain, like renaming the blog, will affect traffic to your blog. The effects of the change will vary from blog to blog, because of the different traffic to every different blog.Followers. People who find your blog because of recommendations by other people.Search engines. Robotic processes which methodically surf your blog, and provide dynamic indexing to people who search for information.Subscribers. People who read your content from their newsfeed reader, such as the dashboard Reading List.Viewers. People who read your content from their browser.No two blogs are the same - and no two blogs will have the same combinations of traffic sources.

Stats Components Are Significant, In Their Own Context

One popular Stats related accessory, which displays pageview information to the public, is the "Popular Posts" gadget.

Popular Posts identifies from 1 to 10 of the most popular posts in the blog, by comparing Stats pageview counts. Optional parts of the display of each post are a snippet of text, and an ever popular thumbnail photo.

Like many Stats features, blog owners have found imaginative uses for "Popular Posts" - and overlook the limitations of the gadget. Both the dynamic nature of Stats, and the timing of the various pageview count recalculations, create confusion, when Popular Posts is examined.