robots.txt

asgsoft · 05-04-2006, 12:57 PM

What do I need to have in robots.txt to make my site search engine friendly?

Any advice?

It's about webmasters tools, seo,domains etc..

Any help would be appreciated

Thank you in advnace

ASGSOFT

webfreak · 05-04-2006, 01:05 PM

Code:

User-agent: *
Disallow:

Create a file named robots.txt and place it in your root directory and everything will be allowed with the code above.

Robots.txt has nothing to do with keywords or anything, just a standard way of controlling robots.

Richieboy67 · 05-04-2006, 01:12 PM

The robot.txt doesn't really make your site friendly or anything but it allows you to control what the robots can see. The code above allows access to everything which is fine for some sites. On other sites you might want to block some pages from being viewed.

More important than the robots.txt is your metatags. Make sure you have them and take your time and do them right. I would be glad to help you seo your site. I am totally into it and I like to think I am good at it too but I won't say it here because everyone will pounce on me and try to prove me wrong....lol
????: NamePros.com http://www.namepros.com/programming/193938-robots-txt.html

Anyhow, I can help you with free advice if you like.

shag · 05-08-2006, 12:04 PM

Well.. if you can put many robots name like below a example then included robots will surf your fast mostly. I may be wrong though.

Code:

User-agent: googlebot
Disallow:


User-agent: Gigabot
Disallow:

User-agent: Gulper
Disallow:

User-agent: MantraAgent
Disallow:

I am using the robot file this way http://xtra-host.com/robots.txt

asgsoft · 05-08-2006, 12:23 PM

I will try that and see what happens, Thank you

05-04-2006, 12:57 PM	THREAD STARTER #1 (permalink)
asgsoft NamePros Regular Join Date: Sep 2005 Location: At Home Posts: 881	robots.txt What do I need to have in robots.txt to make my site search engine friendly? Any advice? It's about webmasters tools, seo,domains etc.. Any help would be appreciated Thank you in advnace ASGSOFT __________________ Free SEO and Domain tools \|HTML ENT\| Your Source to expired domains with PageRank & tips

05-04-2006, 01:05 PM	#2 (permalink)
webfreak Senior Member Join Date: Dec 2005 Posts: 1,604	Code: User-agent: * Disallow: Create a file named robots.txt and place it in your root directory and everything will be allowed with the code above. Robots.txt has nothing to do with keywords or anything, just a standard way of controlling robots.

05-08-2006, 12:04 PM	#4 (permalink)
shag Account Closed Join Date: Apr 2006 Location: United States Posts: 133	Well.. if you can put many robots name like below a example then included robots will surf your fast mostly. I may be wrong though. Code: User-agent: googlebot Disallow: User-agent: Gigabot Disallow: User-agent: Gulper Disallow: User-agent: MantraAgent Disallow: I am using the robot file this way http://xtra-host.com/robots.txt

05-08-2006, 12:23 PM	THREAD STARTER #5 (permalink)
asgsoft NamePros Regular Join Date: Sep 2005 Location: At Home Posts: 881	I will try that and see what happens, Thank you __________________ Free SEO and Domain tools \|HTML ENT\| Your Source to expired domains with PageRank & tips

05-04-2006, 01:12 PM	#3 (permalink)
Richieboy67 Account Closed Join Date: Mar 2005 Location: East Coast USA Posts: 840	The robot.txt doesn't really make your site friendly or anything but it allows you to control what the robots can see. The code above allows access to everything which is fine for some sites. On other sites you might want to block some pages from being viewed. More important than the robots.txt is your metatags. Make sure you have them and take your time and do them right. I would be glad to help you seo your site. I am totally into it and I like to think I am good at it too but I won't say it here because everyone will pounce on me and try to prove me wrong....lol ????: NamePros.com http://www.namepros.com/programming/193938-robots-txt.html Anyhow, I can help you with free advice if you like.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)