Google not crawling

[Trojanfan]

Google last crawled my website on the 31st of may. I update it a few times a day and have a good number of backlinks. Does google just not visit new sites very often? My site has only been up for about three weeks, although the domain is a couple years old and used to be indexed. Any ideas?

 

[Keral_Patel]

Well can't say properly without looking into the site name.

Because as you mentioned everything looks all right. But yet.

 

[Trojanfan]

The site is Hykra.com. The domain is a couple years old and used to have a site on it which was indexed by google. I didn't maintain it and it went dead for a while. I relaunched it a few weeks ago and resubmitted it to google. Google since crawled it that one time when I submitted it to them and not since. Their bot has crawled certain pages which are linked to, but according to google webmaster tools they have not gone back to my homepage or indexed my pages despite the site being updated daily

There are no conflictions with the robots.txt file or anything like that either. I do have a sitemap as well, which is uploaded in the google webmaster tools

 

[Keral_Patel]

Yes the cached page is of 31st May

Your site is indexed and all.

Maybe google doesn't knows yet that your site gets fresh content faster then it used to get.

So I think today is 7th June so maybe you should wait a bit for somedays and let the robot update its frequency.

I also think nothing is wrong in there.

Thanks.

 

[shockie]

the individual posts are getting crawled and indexed though. :tu:

i looked up "these can generally tend to carry higher interest rates as well, which can get you in trouble in the long run" (from the post you made today) in google and it shows your post having been indexed "4 hours ago."

 

[Trojanfan]

Shockie: Yes, the individual posts are getting indexed, which is good. I noticed that as well. And I notice the googlebot on my site every hour. I have developed pretty good backlinks, so the bot is there and thankfully indexing certain pages. I'm just hoping the bot grabs the update from the main page and therefore the site as a whole soon.

Nick, that's good to hear. Hopefully once the bot does return it will have its settings fixed to come back more frequently.

 

[shockie]

another thought: you can set changing/update frequency in your sitemap, so make sure your homepage is set to "daily" i guess.

 

[Trojanfan]

I went ahead and updated the sitemap to hourly. It was set to daily

 

[Keral_Patel]

Nick, that's good to hear. Hopefully once the bot does return it will have its settings fixed to come back more frequently.

Yes mostly the bots are smart enough but yet you did took a good action and updated the frequency.

 

[Trojanfan]

yeah, hopefully I'll see an improvement after the next time it crawls...which hopefully is sooner rather than later!

 

[Adulu]

I saw similar question on others forum.
it might be slow down

 

[Trojanfan]

Alright, google finally returned. Hopefully now it will come back on a regular basis since it sees I update often. We'll see

 

[popopo]

as i know, ask.com will never cawl new sites, is google?

 

[MrCat]

I suggest you build more backlinks on dofollow sites/blogs/forums/etc. If the backlinks are from frequently crawled sites - the bots will find your backlinks and crawl your site as well.

Also set the robots.txt files accordingly.

 

[Trojanfan]

The robots.txt file is set. I have built up the backlinks pretty well. As mentioned, the bot itself does show up daily. But it doesn't visit and index the homepage for quite a while in between

 

[MrCat]

The robots.txt file is set. I have built up the backlinks pretty well. As mentioned, the bot itself does show up daily. But it doesn't visit and index the homepage for quite a while in between

Are your backlinks from dofollow sites? The bots don't index a page from a nofollow link.

Its really a bit weird: http://www.google.com/search?sourceid=gmail&q=www.hykra.com

It does not show there that your site is indexed with the www.

 

[shockie]

Are your backlinks from dofollow sites? The bots don't index a page from a nofollow link.

wrong. yes they do.

 

[weblord]

http://en.wikipedia.org/wiki/Nofollow

wrong. yes they do.

 

[Trojanfan]

Well, glad to see that's been cleared up

 

[shockie]

http://en.wikipedia.org/wiki/Nofollow

While some take it literally and do not follow the link to the page being linked to, others still "follow" the link to find new web pages for indexing.

in particular, googlebot does index pages from nofollow links.