Thursday, June 03, 2010

Blogger Magic - Search Engine Indexing

A lot of blog owners are confused about how their blogs get indexed by the search engines.
Why do many SERP hits lead to my main page, where there is no content?
is a normal question. Many blog owners think that the search engines, as they index a blog, are executing one big magic trick.

Some blog owners think that their blogs will be indexed, and will appear in Search Engine Results Pages (aka "SERPs"), automatically.

Each blog is indexed differently, from different starting points.
They are wrong, though - there is no magic here. Every blog is indexed differently, using different index points.
  • Main page indexing, from "Index my blog!" requests. The more often you request re indexing, the more often your blog will be re indexed.
  • Random indexing, from inlinks from other websites. The more inlinks you have, the more often your blog will be re indexed.
  • Scheduled indexing, using a sitemap. The more often you publish, the more often your sitemap gets updated, and the more likely that the search engines will see a need to re index in the future.
All of these factors affect the others. The more often your blog gets re indexed, the quicker any updates that you make will appear in SERPs, which will lead to more readers, and to more inlinks. The more inlinks your blog has, the more often the search engines will schedule the next re indexing. Eat, Sleep, Rave, Repeat.

Start with content, that people will actually read.
You start out by publishing a blog with interesting, useful, and unique content.

Once you have a blog, you spend time publicising your blog. One of the key steps in publicising involves submitting indexing requests to the search engines - either directly, or through one or more search engines submission services. Your blog is placed into a queue, and the main page is indexed. Until your blog has a reputation, the blog is indexed, using the main page - and that's it.

Indexing starts with the sitemap, provided for every Blogger blog.
With a Blogger blog, you provide a sitemap, which is provided automatically, for every blog. The search engines index the blog using the sitemap.

Please, always remember that your blog gets indexed faster, when the search engines can read the content. If your blog is published to a custom domain, it's in your best interest to use righteous DNS addressing.

There are many different search engines, each with different customers.
There are many search engines on the Internet. Each search engine indexes the websites on the Internet, so it can serve its customers - the people who want to read the websites. Each search engine wants to improve its own reputation for providing good results.

For an insightful reference into Google Search, you may enjoy Google: How Search Works. Remember that there are subtle differences between how Blogger blogs, and websites in general, are treated. Be selective, in accepting advice outside the Blogger environment, when asking about a Blogger blog.

Each search engine serves its own interests - and provides better visibility to blogs and websites with higher reputation, when people search for blogs and websites. Your blog gets a higher reputation, as more blogs and websites link to it. With better visibility, your blog gets more new readers, and more inlinks.

Links from other blogs and websites produce varying benefit, to your blog.
A link from a low reputation website - to your blog - gives your blog little reputation. A link from a high reputation website - to your blog - gives your blog more reputation. When that website is being indexed, and the search engine finds a link to your blog, your blog is indexed. This may be a main page link, or an individual post link.

As your blog gains reputation, it gets indexed regularly, by a search engine, on its own. This is where providing a sitemap may be a key ingredient to getting your blog indexed - and this is where your individual posts are indexed, possibly based on a sitemap.

Just remember that your blog needs a good reputation, to get more readers - but your good reputation won't be yours, forever.

Your posts will initially be indexed, using main page content.
Until your individual posts are indexed, and they get reputation, most SERPs that reference the posts will link to the main page. You'll see similar latency issues when you rename your blog without properly planning for the change - you'll have search hit entries referencing the old blog name (URL) long after the blog is renamed.

Neither issue is magic, it's all logic. None of this is magic, in reality.

Dude, hit me with a comment!

Alanna Kellogg said...

It also makes it even more important to have a search function available high on each page - so motivated searchers can find what they are looking for.

Antonia Noël said...

And until you get a significant amount of traffic, linking your blog on forums, twitter, facebook, myspace, technorati, etc. on a regular basis will increase your blog's reputation. I would do this anyway even if you already have a lot of readers. At least that's how I believe it works.

-Antonia Noel speaking on behalf of the earth