Skip to main content

Customise The "Robots.Txt" File, For Your Blog

The file "Robots.Txt", used to manage search engine access to your blog, is found in the blog root folder.

You can't edit "Robot.Txt" directly - but you can upload a new file. Long ago, you would use Webmaster Tools - which is now called Search Console.

Now, we can update "robots.txt" directly, using the Blogger dashboard "Custom robots.txt" wizard, in Settings - "Search preferences".

Here's the link to the "Robots.Txt" file, for this blog.

To view / copy the current contents, just click on the link. You can do the same with the file, for your blog.

If I want to change the file for this blog, I start with the current file. Excepting one small detail (my blog is published to "blogging.nitecruzr.net", yours is published to your chosen URL), your file will be identical - if you have not tweaked it.

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: http://blogging.nitecruzr.net/sitemap.xml

Let's look at a simple example - adding a sitemap.

Some people would like to index the static pages, in their blog. That's a simple enough task - just add the sitemap, for static pages, to your file.

"sitemap-pages.xml", like "sitemap.xml", is automatically generated for each blog.

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: http://blogging.nitecruzr.net/sitemap.xml Sitemap: http://blogging.nitecruzr.net/sitemap-pages.xml


Start from Settings - Search preferences.



Start with an empty ("[Disabled]") "Custom robots.txt".




Click on "Edit".




Click on "Yes".




Paste in your edited file content.

Note that you will overlay the current contents with what you paste - so paste the entire file, as you wish to have.





Click on "Save changes".




Select "Yes" for "Enable custom robots.txt content?", if necessary.





Right now, though, I don't want to change my file. So, I'll leave my custom file in the wizard, and leave "No" selected.




And the custom file is
"[Disabled]", again.




Even if disabled, I can get it back again, easily enough. Just click on "Yes", and there it is, again. For right now, though, I'll leave it
"[Disabled]".


Just be aware of the dangers of editing, without knowing what effect a given entry has, on search engine activity. Remember that you can get the default file back, with a couple mouse clicks.

Comments

BW Christopher said…
Thank-you! Brilliant instructions, even I understand! Much appreciated.

Bryan
Chem2tech said…
Hello Sir,

I tried to do this in my blog but it is showing only 151 submitted link and 126 indexed only knowing that i have 304 posts published.

http://chem2tech-en.blogspot.com

can you please check it, and help me to adjust this error.

Regards

Tamer
LexBlogger said…
I followed your excellent guidance, and now have my Robots.txt file customized with the sitemap-pages.xml line added. So far so good.

But a Custom Search does not find any words that are indexed on the four pages {I assume they are static) that I have. So obviously I have missed a step, or a concept somewhere. So I would appreciate a little guidance.

Harry

Popular posts from this blog

Custom Domain Migration - Managing The Traffic

Your blog depends upon traffic for its success.

Anything that affects the traffic to your blog, such as any change in the URL, affects the success of your blog. Publishing the blog to a custom domain, like renaming the blog, will affect traffic to your blog. The effects of the change will vary from blog to blog, because of the different traffic to every different blog.Followers. People who find your blog because of recommendations by other people.Search engines. Robotic processes which methodically surf your blog, and provide dynamic indexing to people who search for information.Subscribers. People who read your content from their newsfeed reader, such as the dashboard Reading List.Viewers. People who read your content from their browser.No two blogs are the same - and no two blogs will have the same combinations of traffic sources.

Stats Components Are Significant, In Their Own Context

One popular Stats related accessory, which displays pageview information to the public, is the "Popular Posts" gadget.

Popular Posts identifies from 1 to 10 of the most popular posts in the blog, by comparing Stats pageview counts. Optional parts of the display of each post are a snippet of text, and an ever popular thumbnail photo.

Like many Stats features, blog owners have found imaginative uses for "Popular Posts" - and overlook the limitations of the gadget. Both the dynamic nature of Stats, and the timing of the various pageview count recalculations, create confusion, when Popular Posts is examined.