Robots.txt

[xtremeoff-road]

Hello everyone, I have a site www.bignutsracing.com

Could someone be so generous as to instruct me on the proper use of a robots.txt file.

Spiders are allowed to crawl everything on the site, therefore, I have placed the file in www.bignutsracing.com and www.bignutsracing.com/forums

All I put in the file was:

User-agent: *
Disallow:

Is this correct?

Any help is greatly appreciated!

 

[dewd]

yes

 

[apexad]

That will work. I reccommend the site www.robotstxt.org

 

[AbsoluteKC]

Actually you do not have to include robots.txt if you want bots to index all your pages

 

[apexad]

Actually you do not have to include robots.txt if you want bots to index all your pages

I think robots like your site better if you do stuff to help them though, like create a 'robots.txt' file, even if it tells them they can index everything.

 

[xtremeoff-road]

Thank you everyone for taking the time!

@ apexad, thanks for the link, I will check it out!

@AbsoluteKC, I think that is the case, however, I have been reading and they say some crawlers immediately look for a robots.txt file...I figure anything I can do to help them crawl the site would be beneficial

Thanks again!

 

[907Group]

yes that is correct.
I have mine setup as

# Robots.txt file from http://www.yourdomainhere.com
# All robots will spider the domain

User-agent: *
Disallow:


upload it to your server..

 

[Amnezia]

I think robots like your site better if you do stuff to help them though, like create a 'robots.txt' file, even if it tells them they can index everything.

lol robots dont have feelings, if you want your site fully accessible then there is no need for robots.txt