Source: http://www.adsensebits.com/node/24
Up until now the bot (or "crawler") that Google used to index sites was based on a lynx browser. Which is to say it "saw" text. Previously the only way to tell if information was important, was if we told Google it was by using various forms of markup and a particularly labeled comment that Google would recognize as outlining important information. To get a better idea of how Google used to see your site, load up FireFox and disable images, scripts and style sheets. Start browsing.
Enter: Googlebot 2.0
But that was so 2005. 2006 sees the introduction of a newer, smarter and pickier Googlebot. One that cares about CSS and JavaScript, position and layout. Utilizing what shows up in logs as
Mozilla, Googlebot
Googlebot 2.0 is looking websites up and down in ways it couldn't have possibly done before by requesting style sheets (.css) and JavaScript's (.js).
I suppose the most obvious reason for this is that Mozilla Googlebot is taking snapshots of our websites in order to integrate the ever-so-popular but sometimes irritating thumbnail feature so many smaller search engines have implemented, or they're doing something much more Google; Searching our websites like a human being.
It seems to me that in order to maintain the ability to produce relevant ads to display, Google needs to be able to tell what information is important by finding out where it is, what it is and how it's displayed. The only way to do that is to start looking at our websites and seeing them like we do.
Whatever the reason for this new crawler, It'll be interesting to see what they do with the new information they're collecting. As the web evolves, so must Google.
Up until now the bot (or "crawler") that Google used to index sites was based on a lynx browser. Which is to say it "saw" text. Previously the only way to tell if information was important, was if we told Google it was by using various forms of markup and a particularly labeled comment that Google would recognize as outlining important information. To get a better idea of how Google used to see your site, load up FireFox and disable images, scripts and style sheets. Start browsing.
Enter: Googlebot 2.0
But that was so 2005. 2006 sees the introduction of a newer, smarter and pickier Googlebot. One that cares about CSS and JavaScript, position and layout. Utilizing what shows up in logs as
Mozilla, Googlebot
Googlebot 2.0 is looking websites up and down in ways it couldn't have possibly done before by requesting style sheets (.css) and JavaScript's (.js).
I suppose the most obvious reason for this is that Mozilla Googlebot is taking snapshots of our websites in order to integrate the ever-so-popular but sometimes irritating thumbnail feature so many smaller search engines have implemented, or they're doing something much more Google; Searching our websites like a human being.
It seems to me that in order to maintain the ability to produce relevant ads to display, Google needs to be able to tell what information is important by finding out where it is, what it is and how it's displayed. The only way to do that is to start looking at our websites and seeing them like we do.
Whatever the reason for this new crawler, It'll be interesting to see what they do with the new information they're collecting. As the web evolves, so must Google.
