Google Googlebot can't access your site

[SirDrago]

I received this E-mail yesterday, Can anyone shed some light on this ???
Is this a legitimate email from Google????

"""Over the last 24 hours, Googlebot encountered 2 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%""""

""""
Recommended action

If the site error rate is 100%:

• Using a web browser, attempt to access http://www.Example site.com/robots.txt. If you are able to access it from your browser, then your site may be configured to deny access to googlebot. Check the configuration of your firewall and site to ensure that you are not denying access to googlebot.
• If your robots.txt is a static page, verify that your web service has proper permissions to access the file.
• If your robots.txt is dynamically generated, verify that the scripts that generate the robots.txt are properly configured and have permission to run. Check the logs for your website to see if your scripts are failing, and if so attempt to diagnose the cause of the failure.

If the site error rate is less than 100%:

• Using Webmaster Tools, find a day with a high error rate and examine the logs for your web server for that day. Look for errors accessing robots.txt in the logs for that day and fix the causes of those errors.
• The most likely explanation is that your site is overloaded. Contact your hosting provider and discuss reconfiguring your web server or adding more resources to your website.
• If your site redirects to another hostname, another possible explanation is that a URL on your site is redirecting to a hostname whose serving of its robots.txt file is exhibiting one or more of these issues.

After you've fixed the problem, use Fetch as Google to fetchhttp://www.Example site.com/robots.txt to verify that Googlebot can properly access your site.

Learn more in our Help Center.

 

[iowadawg]

Never heard of google sending out emails to webmasters saying they could not access their site.
May be this is a new thing?

 

[SirDrago]

Never heard of google sending out emails to webmasters saying they could not access their site.
May be this is a new thing?

Neither have I which is why I'm skeptical about clicking any of the links in the Email.
Might be a new Scam/Spam... The Email came from [email protected]

 

[Messiah]

Strange. For 2 crawler error Google is sending out email. Robots.txt file is to deny access to specific content of your site usually folder paths.
As mentioned in the last line, try to fetch the url example.com/robots.txt and verify if Googlebot is able to access.

 

[enlytend]

If you're using Google Webmaster Tools, depending on how your alerts are set up you can get an email like this. It's a valid message.

Look in GWT to see if there's more information. You can use Fetch as Google and check robots.txt from there.

Usually it's just a "glitch" - network issues, nameserver outages or hosting downtime that keeps them from accessing the site temporarily. If you didn't make any changes, that's probably the case.

PS [email protected] IS the email that sends webmaster tool alerts.

 

[SirDrago]

After checking GWT I have a greeen check on DNS, Green check on Server connectivity and a RED exclamation mark on robots.txt fetch.
I examined the files for that Domain on my server and I don't have a Robot.txt file in there whatsoever. Could that be the issue? Do i need to generate one?

 

[Messiah]

Not sure if you are using Wordpress or other. Usually every wordpress installation will create a virtual robots.txt file by default. If your site is not on Wordpress, then create a robots.txt file and upload it. Try fetching the url. It should work.

 

[SirDrago]

It's PHP script and I thank you all for your guidance and will definitely try that. I did find THIS.

 

[D. Wells]

Add the following line at the end of your robots.txt file:

User-agent: Google
Disallow:

After you do this, fetch as GoogleBot to test.

 

[Messiah]

That's right. However I use * to give access to all the bots from other search engines and not only to Google.

User-agent: *
Disallow:

 

[D. Wells]

That's right. However I use * to give access to all the bots from other search engines and not only to Google.

User-agent: *
Disallow:

True, I just used Google, because I was unsure of what rules you had in your robots.txt.

 

[enlytend]

If you HAVE no robots.txt, it isn't blocking anything and therefore isn't the reason you got that email.

Try a couple of pages with 'fetch as google" - it was probably a temporary error. I get them all the time on one of the crap hosting sites a couple of my sites are on.

 

[D. Wells]

@SirDrago

OP can you provide the URL to you site?

 

[SirDrago]

I want to Thank each and every one of you for the knowledge and quick responses. This is exactly why i love it here a NamePros!!!
Taking the above advise I am now Google friendly and passed their Robots.txt test.

A simple "NotePad" file with the above suggestion placed in the root directory of my script consisting of

User-agent: *
Disallow:

Got me up and running!!! THANK YOU ALL VERY MUCH!!!
@IceDude URL

 

[enlytend]

Just to reiterate though
- NOT having a robots.txt file is not the reason you got the initial email. You can get that same message regardless of weather you have a robots.txt file or not - it's usually an availability issue. You should have one, which is why GWT flagged it, but its not a showstopper if you don't.

 

[SirDrago]

Just to reiterate though
- NOT having a robots.txt file is not the reason you got the initial email. You can get that same message regardless of weather you have a robots.txt file or not - it's usually an availability issue. You should have one, which is why GWT flagged it, but its not a showstopper if you don't.

Is there more I should check into?
If so what in particular should I look into?

 

[enlytend]

Unless it persists I wouldn't worry. These are usually a transient error when there's a brief outage or the server is busy/overloaded when Google visits. You can try the Fetch as Google function in webmaster tools ...

 

[SirDrago]

The fetch worked perfectly, thank you for all your help. I really appreciate everyone taking time and helping need out in this.