What is robots.txt?

[Jammer]

I saw this in another post and was wondering what it means.
I've also seen it in a few of my error logs,but never knew what it was.
Thanks!

 

[RJ]

Search engines like Google look for this file on your website when indexing your site. You can create a robots.txt file to give the search engine instructions for indexing your site, for example excluding certain pages or directories.

Here's a whole site about it
http://www.robotstxt.org/

 

[Jammer]

Thanks RJ!

 

[TheWatcher]

Nice link RJ.

Thanks,
TW