Skip to main content

Customise The "Robots.Txt" File, For Your Blog

The file "Robots.Txt", used to manage search engine access to your blog, is found in the blog root folder.

You can't edit "Robot.Txt" directly - but you can upload a new file. Long ago, you would use Webmaster Tools - which is now called Search Console.

Now, we can update "robots.txt" directly, using the Blogger dashboard "Custom robots.txt" wizard, in Settings - "Search preferences".

Here's the link to the "Robots.Txt" file, for this blog.

To view / copy the current contents, just click on the link. You can do the same with the file, for your blog.

If I want to change the file for this blog, I start with the current file. Excepting one small detail (my blog is published to "", yours is published to your chosen URL), your file will be identical - if you have not tweaked it.

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap:

Let's look at a simple example - adding a sitemap.

Some people would like to index the static pages, in their blog. That's a simple enough task - just add the sitemap, for static pages, to your file.

"sitemap-pages.xml", like "sitemap.xml", is automatically generated for each blog.

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: Sitemap:

Start from Settings - Search preferences.

Start with an empty ("[Disabled]") "Custom robots.txt".

Click on "Edit".

Click on "Yes".

Paste in your edited file content. Note that you will overlay the current contents with what you paste - so paste the entire file, as you wish to have.

Click on "Save changes".

Select "Yes" for "Enable custom robots.txt content?", if necessary.

Right now, though, I don't want to change my file. So, I'll leave my custom file in the wizard, and leave "No" selected.

And the custom file is
"[Disabled]", again.

Even if disabled, I can get it back again, easily enough. Just click on "Yes", and there it is, again. For right now, though, I'll leave it "[Disabled]".

Just be aware of the dangers of editing, without knowing what effect a given entry has, on search engine activity. Remember that you can get the default file back, with a couple mouse clicks.


BW Christopher said…
Thank-you! Brilliant instructions, even I understand! Much appreciated.

Unknown said…
Hello Sir,

I tried to do this in my blog but it is showing only 151 submitted link and 126 indexed only knowing that i have 304 posts published.

can you please check it, and help me to adjust this error.


LexBlogger said…
I followed your excellent guidance, and now have my Robots.txt file customized with the sitemap-pages.xml line added. So far so good.

But a Custom Search does not find any words that are indexed on the four pages {I assume they are static) that I have. So obviously I have missed a step, or a concept somewhere. So I would appreciate a little guidance.


Popular posts from this blog

What's The URL Of My Blog?

We see the plea for help, periodicallyI need the URL of my blog, so I can give it to my friends. Help!Who's buried in Grant's Tomb, after all?No Chuck, be polite.OK, OK. The title of this blog is "The Real Blogger Status", and the title of this post is "What's The URL Of My Blog?".

Leave Comments Here

Like any blogger, I appreciate polite comments, when they are relevant to the blog, and posted to the relevant article in the right blog. If you want to ask me a question thats relevant to blogging, but you can't find the right post to start with (I haven't written about everything blogger related, yet, nor the way things are going I don't expect to either), ask your questions here, or leave an entry in my guestbook.

As noted above, please note my commenting policy. If you post a comment to this post, I will probably treat it as a "Contact Me" post. If you have an issue that's relevant to any technical issue in the blog, please leave a comment on the specific post, not here. This post is for general comments, and for non posted contact to me.

If the form below does not work for you, check your third party cookies setting!

For actual technical issues, note that peer support in Blogger Help Forum: Something Is Broken, or Nitecruzr Dot Net - Blogging is, almos…

What Is "" vs. ""?

With Google Domains registered custom domains becoming more normal, we are seeing one odd attention to detail, expressed as confusion in Blogger Help Forum: Learn More About Blogger.My website uses "" - am I supposed to use "", instead?It's good to be attentive to detail, particularly with custom domain publishing. This is one detail that may not require immediate attention, however.