Is this normal activity for spiders?

[Gex]

Hi

My site got listed within a few days, and all the content pages were soon indexed (like 20 so far?). I was actually surprised that it was this easy in the end. Now the spiders regularly visit - Googlebot for instance visits perhaps once every 24 hours. That said, they don't seem to get very far at all, and several new pages have been added since which haven't been indexed. I would say that for the last two weeks or so, Googlebot comes to the index page, and either:

- sees the index page and/or robots.txt, and that's it: 80%
- sees the above, plus 1-2 old pages (seemingly at random): 20%

Just wondering how normal/abnormal this sort of behavior is? Does this imply there's some problem with the site that can be rectified? I thought that getting the whole site indexed to start with would be the hard part, not actually getting new pages to be indexed!

 

[hostingfun]

What is the site built in? PHP? HTML? ASP? CFM? or hybrid of some? Too, do you have other domains linking to it already? Do you have Google adsense on the site? Lots of factors would affect Googlebot.

 

[Gex]

Thanks for the reply.

The website is PHP/MySQL and HTML, and yes there are domains linking to it. No, there is no Adsense. I just find it weird how initially it was so easy, but since then it's sort of stalled a bit. Maybe this is just the nature of spiders - if so, please let me know.

Oh, on a related question, which may be easier to answer: MSNBot stopped by about 7 hours ago. Went to a couple of old pages, then to robots.txt, then to another few pages, including all the new ones. However, they still don't appear in MSN search. Does it commonly take long for indexed pages to "appear" in the search listings or should it really happen instantly? Or maybe the spider visits the page and elects not to index on the majority of occasions?

Again, thanks in advance for any assistance!

 

[hostingfun]

Well, the spiders show that the robots are putting information in the queque. It may be some time before those pages visited are put through a series of test (the algorithms) and assessed position or relevancy. The bot visits don't mean that pages will show up in the search results right away.

Bots will follow links from other sites that are already indexed. If your other sites (or other individual sites) are linking to the newer site and they have already been indexed, the bots would find your newer site a bit faster.

PHP/MySQL and HTML is a good structure to use - so give the engines a bit of time and they'll be more regular - just remember content is king

 

[Gex]

Thanks for answering a lot of questions. Rep added!