Getting Indexed in Google - The Googlebot, Freshbot & Deepbot

Written By: Trevor A. Winchell - Gooyah Search Founder

Pretty much everyone who is interested in search engine marketing has a vested interest in getting listed in Google. It is such an important search engine that we felt it worthwhile explaining how Google indexes sites and how you can tell if your site has been picked up in the latest Google trawl for inclusion in the latest search results.

There are two ways to get listed in Google. You can opt to submit your index page to Google directly here, or you can wait until Google spiders your site automatically, usually by finding your site via links on already listed sites. Once again, as we’ve discussed in other articles, you should actively seek to exchange links with other sites for a variety of reasons (link popularity being the key one). If you’ve just started out, links back to your new site will help the Google trawlers to find your site for the first time.

Google has two types of spiders – “Freshbot” and “Deepbot”. The Freshbot, as you’d expect from its name, is constantly crawling the web for updated information, so you may well get daily visits from it. The Deepbot goes out once a month and usually lasts a week – many search engine experts see the Deepbot as the more critical one, since it determines which pages get permanently added to the main index and looks at your link structure and therefore influences your page rank.

You can tell if “Googlebot” has visited your site by examining your own log files. You should see and entry such as this one, which indicates that either type of Googlebot has recently checked our your pages – it will appear as a “browser” in your records typically:

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

There are also a wide variety of IP addresses associated with the Googlebot – the main ones you’d expect to see will be something like this:

64.*.*.* (Freshbot should start with this numbers)
209.*.*.* (Deepbot should start with this number)

If you have a new site, or indeed have added new pages to an already listed site, there are no firm guarantees as to when they will appear in the main index. Typically, you should wait a month to see your main page URL appear, and then several weeks after this your supplementary pages should start showing up.

If you have had no success after the first month or so, there are reasons why Googlebot didn’t like your site –

First off, try reading up on Google’s own tips:

“Occasionally, websites are not reachable when we try to crawl them because of network or hosting problems. When this happens, we retry multiple times, but if the site cannot be crawled, it will not be listed in our current index. If it was a transient problem, the site will likely show up in the next index, which will be completed in a few weeks.

If we have not picked up your site and it has been several months, then it is likely that our spiders are not able to find your site. If you increase the links pointing to the page, Google will likely find your site in the future.”