Is robots.txt file necessary

[esnc]

Hi
Is it necessary to put robots.txt file in the root directory of website, i know it is helpful to control crawlers, but if i not put this file in the website, does this cause some problem, please advice
Thanks for your support

 

[whitebark]

Not having one is never an issue.

 

[HappyDay]

http://www.robotstxt.org/faq.html


God bots like the Google bot are always looking for the robots.txt while the bad bots ignore it.

Its just better to have one in your root I believe.

 

[shockie]

you should have one if you need one, but the average new minisite won't need one.

 

[Peter]

As stated by a few people. If you have no content that needs to be blocked from search engines then no you do not need it (also remember if you do block search engines using it search engines do not always take notice also if you have something private in a specific location humans will still be able to find it and robot's will make it easier).

Some people do have an empty robots.txt file as it can make it easier to check which pages show 404 errors (makes life difficult if looking through raw logs and you keep seeing a 404 for a robots.txt request).

 

[andreyknure]

Robots.txt file is not always necessary for a website, nevertheless it is better to use in the certain cases.