Robot.txt How do I?

[Billy!]

Stupid question maybe? But what do I have to put into a robot.txt file thingy for my VB Forums to keep the spiders happy and to get more visitors to my sites?

 

[Nathan]

Not exactly necessary, vB does an ok job with this, but I still recommend adding one, as it takes like 4 seconds to do :P

Here is a guide on robots.txt:
http://rsforums.org/forums/showthread.php?t=2544

It includes the text that you want to put in a robots.txt file for vB as well, so you can just copy and paste that :P

 

[Billy!]

Thanks so much Matt, that has help me greatly ... I did the last one!

 

[Nathan]

Cheers! :tu:

Glad I could help

 

[ag9]

You can point out sitemap path in the robots.txt.

Sitemap: http://www.example.com/sitemap.xml

This will help your site to be more indexed by search engines.

 

[Keral_Patel]

I have always made a blank robots.txt and used NoFollow tag to hide the duplicate content from the search engines.

Just yesterday I figured out that one of the sites of my friend had a problem in his robots.txt and he had accidentally disallowed google bot.

 

[sufi]

I use this tool to generate robots.txt files for my websites. I mainly do it to block images directory to save bandwidth.