Getting Indexed in Google - The Googlebot, Freshbot & Deepbot
Written By: Trevor A. Winchell - Gooyah Search Founder
Pretty much everyone who is interested in search engine marketing has a vested interest in getting listed in Google. It is such an important search engine that we felt it worthwhile explaining how Google indexes sites and how you can tell if your site has been picked up in the latest Google trawl for inclusion in the latest search results.
There are two ways to get listed in Google. You can opt to submit your index page to Google directly here, or you can wait until Google spiders your site automatically, usually by finding your site via links on already listed sites. Once again, as we’ve discussed in other articles, you should actively seek to exchange links with other sites for a variety of reasons (link popularity being the key one). If you’ve just started out, links back to your new site will help the Google trawlers to find your site for the first time.
Google has two types of spiders – “Freshbot” and “Deepbot”. The Freshbot, as you’d expect from its name, is constantly crawling the web for updated information, so you may well get daily visits from it. The Deepbot goes out once a month and usually lasts a week – many search engine experts see the Deepbot as the more critical one, since it determines which pages get permanently added to the main index and looks at your link structure and therefore influences your page rank.
You can tell if “Googlebot” has visited your site by examining your own log files. You should see and entry such as this one, which indicates that either type of Googlebot has recently checked our your pages – it will appear as a “browser” in your records typically:
There are also a wide variety of IP addresses associated with the Googlebot – the main ones you’d expect to see will be something like this:
64.*.*.* (Freshbot should start with this numbers) 209.*.*.* (Deepbot should start with this number)
If you have a new site, or indeed have added new pages to an already listed site, there are no firm guarantees as to when they will appear in the main index. Typically, you should wait a month to see your main page URL appear, and then several weeks after this your supplementary pages should start showing up.
If you have had no success after the first month or so, there are reasons why Googlebot didn’t like your site –
First off, try reading up on Google’s own tips:
“Occasionally, websites are not reachable when we try to crawl them because of network or hosting problems. When this happens, we retry multiple times, but if the site cannot be crawled, it will not be listed in our current index. If it was a transient problem, the site will likely show up in the next index, which will be completed in a few weeks.
If we have not picked up your site and it has been several months, then it is likely that our spiders are not able to find your site. If you increase the links pointing to the page, Google will likely find your site in the future.”
Written By: Trevor A. Winchell - Gooyah Search Founder
Here is something else for you to try...
Word has it that it's better to be found by Google than to directly submit your pages. Spidered pages apparently rank higher, so the Google ranking algorithm is set to pay more attention if you get a highly-ranked site to link to you.
But if you don't know anyone in that category, let AllTheWeb.com rescue you! Here's how...
Would you like to have the the KEY to Google's Back Door?
Written By: Trevor A. Winchell - Gooyah Search Founder
How Would you like to have the the KEY to Google's Back Door?
Now you CAN! Now you can submit ALL your webpages to Google through their Back Door with their full blessings!
It's easy, fast and very effective! A child could do it.
I couldn’t believe my own eyes! I submitted my website to Google through their back door on Friday evening. By Sunday morning, they had already spidered 74 web pages. On Monday they spidered another 110 web pages, and the next day even more!
This is truly going to revolutionize the way that people submit their web pages to Google.
In this report I will show you will show you exactly how YOU can use Google's back door.... with their blessings.
Have you been as frustrated as I have about how long it takes Google to “spider” your webpages and get them listed in the search engines? Do you find yourself checking each day or each week to see how many more of your pages Google spidered?
ARE YOU LOSING MONEY because Google doesn’t have your pages listed?
Well, you’re not alone. I have thousands of web pages, and it can be excruciating to wait for six months or more or NEVER to get Google to spider through the entire website.
Sound familiar? Yes, it CAN take a long time.... But don’t blame Google!
Even with their thousands upon thousands of servers, and a gazillions of megabytes fast Internet connection, when you consider the millions of pages that they try to “crawl” each day it’s no wonder that it takes so long.
Let me share a big secret with you....
Google WANTS to list your website and they've made it MUCH easier for you!
Why wouldn’t they want to list as much of your website as they could? After all, their whole business is geared around presenting the most complete data base on the Internet. But there’s no need to believe me… let me quote from Google’s own website.
"Google Sitemaps is intended for all web site owners, from those with a single web page to companies with millions of ever-changing pages. If any of the following are true, then you may be especially interested in Google Sitemaps:
You want Google to crawl more of your web pages. You want to be able to tell Google when content on your site changes. " A little farther down on their page they state:
"In alignment with Google's mission to organize the world's information and make it universally accessible, this collaborative crawling system will allow our crawlers to optimize the usefulness of Google's index for users by improving its coverage and freshness." Don't miss the significance of the above passage, quoted directly from Google's website.
It says that if you want Google to crawl more of your web pages, then the Google Site Map program is for you. Those are Google's words, NOT MINE!
Here’s how you can help Google by assisting them in the horrendous task of spidering the entire Internet.
But before I get into that.... let me stop for just a moment. I will not make ANY claims about what Google will do. It's their search engine, and they can change their algorithms any time they like... They can spider and list your site any time, and any way they like, and unfortunately they didn't ask me when they set up their algorithms.
This is a program that gives GOOGLE exactly what it wants!
Not what I THINK Google wants.... But what Google SAYS it wants.....
With that said, let me show you the easiest way you can get ALL of your webpage submitted to Google in one quick easy to follow process.
Forget about Blogging and Pinging. Oh I'm not saying don't do it, but for getting listed in Google, you'll get more pages listed faster this way! (Read Google's quote again).
Do NOT list your pages through their submission form. It will take you forever on even a moderately large site, and in addition it ties up more of their resources. That's not the way Google wants you to do it.
Never again pay a search engine submission service to do what you can do in just a few minutes.
Submit ALL your pages at once, not just a few at a time.
Again, Google WANTS you to use this system...
What is this Secret Google Website Submission Strategy?
It's no secret. This is what Google wants you to do, and you can read about it on their website. This new program is called Google’s Sitemap program.
The way their program works is like this. You create a data base file that contains information about ALL of your web pages. You just load that file onto your website and then let Google know where that file is.
By doing this you are cutting down on their overhead by a huge amount… and in the process you are enabling them to visit many MORE of your web pages quickly and easy. This is exactly what Google is begging you to do…. It’s a win win situation.
Google’s sitemap program is one of the most exciting programs to hit the Internet since web browsers. This program literally WILL revolutionize the way search engines list and spider websites. You can look at Googles sitemap from this URL which is on Google's website. http://www.google.com/webmasters/sitemap
But here's the problem.....
Google is very specific about how they want the data base file to be formatted. It must be done in an xml format.
XML format? Are you asking yourself what's an XML format? Don't worry about it. You don't need to worry about it any more, we take care of those programming details for you.
That's why most people couldn't use Google's back door. It was just too confusing.
Up until now, it was quite a daunting task to prepare this data base, especially if your website consisted of 20 or more pages.
But my revolutionary new software makes it easy for ANY webmaster or web designer to submit all their pages to Google quickly and easily.
The Secret Spider Generator
This amazing new software automates the entire task for you!
It will search through your entire website, and create the xml data feed for you so that Google can then visit all the pages on your website. (don't worry about what xml is, you don't need to worry about that)
The Secret Spider Generator makes it so simple that even a caveman could do it! The Secret Spider Generator is a very simple to use program consisting of only a few fields.
Are you beginning to see how simple this can really be?
You type in your URL, and select how often you think content on your site will change from the drop down menu. You name the file (usually sitemap.xml) and then select the folder that you want to save the file in. After you've done that, you select the start button.
The program will then spider (visit) your ENTIRE website, building a database of ALL your web pages.
When it is done, it will save the completed file to your computer. Then you copy that file across to your hosting account, and your data feed is ready to go. It takes longer to read the above description than to actually do it.
After you copy the file over to your hosting account, you log in to your Google sitemap account, which you can get for FREE, and enter the URL that leads to the sitemap.xml file you just loaded onto your webhosting account.
That’s it! It’s that simple!
Forget Blog and Ping. at least as far as Google is concerned!
Forget about using Google’s own submission form. That’s the slow way to get listed. Either you have to type in each URL manually, or you have to wait and wait and wait until Google finds and spiders all of the websites.
Google would RATHER you used this system.
Using this new backdoor to Google will get more pages listed faster and quicker than any other method you’ve ever seen!
How much is your time worth? How much would it cost you if you have even an average website to submit each of your pages to Google using the traditional methods?
Why do that when you have a pushbutton system through Google's back door?
OK... so the question comes up... how much is it going to cost?
How can you place a value on the Key to Google's Back Door?
I've deliberated long and hard about how much to charge for this program. In the past I know I've spent HUNDREDS of hours submitting pages the old fashioned way, and they were never as effective as I wanted them to be.
Google does not like the old fashioned, outdated, automated submission programs, we all know that. It's a well known fact that you can get banned if you use the old fashioned automated site submission software, most of them costing $300 or MORE. Yet Google is asking you to submit your site to them in this fashion. You can't get into trouble if you give Google what they want, and that's exactly what we're doing!
However, since this is a new product, and you're one of the very first to try it, I've decided to offer a discount to the first 500 people who order this product, you can. Honestly at $97 or $197... compared to what this program can do for you, it's a drop in the bucket... but you can get it today for the introductory price of only $97.00. That includes unlimited technical support, free upgrades, and a free subscription to coolwebtips, valued itself at $97.00.
Use it on as many sites as you'd like. You can submit hundreds of thousands of web pages this way... for only $97.00
This price WILL be going up in the future... but it's not a matter of whether the price goes up or not.
Every day you wait, is one more day your pages just sit there waiting for Google to come around and visit them. That's FAR more costly than an extra fifty bucks that I'll be raising the price to.... so don't wait another moment.
If you can't see the value of this program, then it's probably not for you.
If you do see the INCREDIBLE value, then you're already chomping at the bit to get started. You can download the program immediately, and have your entire website cataloged and submitted to Google within just a very short time.
The KEY to Google's Back Door will make your submissions to Google the quickest and easiest and most complete that you've ever done, or you don't pay a dime. You can return it at any time within 90 days no questions asked, and we'll part as friends. What could be more fair than that?
As with all of my products, if you don't like it you can have your money back. You take no risks. I take all the risks.
We've seen sites get spidered in just a few HOURS, but it could take longer depending on Google's schedule.
The only way you can lose in this deal is by NOT ordering. Don't let your competition get the jump on you. Order right now, while you're still thinking about it.
Remember, every day you hesitate is one more day that your site DOESN'T get listed with Google.
