Is this robots.txt not allowing bots to crawl?

dan_Vt · 05-03-2007, 07:04 AM

Found this on a site, is this preventing the bots from crawling?

Code:

User-Agent: OmniExplorer_Bot
Disallow: /

Sorry, answered my own question.

As you were.

jabba_29 · 05-05-2007, 02:01 AM

Any the answer was???

Yes? It should do if the bot respects robots.txt

jerometan · 05-05-2007, 07:24 AM

Nasty bots don't follow robots.txt.

robots.txt is something like a "Hey, I want/don't want you to come here, but if you insist, there is nothing I can do

"

dan_Vt · 05-05-2007, 11:14 AM

It tells OmniExplorer_Bot not to follow, other bots are fine.

05-03-2007, 07:04 AM	#1 (permalink)
dan_Vt Senior Member Join Date: Dec 2004 Location: Burlington, Vermont Posts: 1,043 1,086.90 NP$ (Donate)	Is this robots.txt not allowing bots to crawl? Found this on a site, is this preventing the bots from crawling? Code: User-Agent: OmniExplorer_Bot Disallow: / Sorry, answered my own question. As you were. __________________ Vermont SEO Automotive SEO For Sale: Co-Sign.com \| Skiny.net \| CSIFAQ.com \| FuelBank.net \| Jibbar.com \|

05-05-2007, 02:01 AM	#2 (permalink)
jabba_29 NamePros Member Join Date: Mar 2007 Location: Finland Posts: 29 5.00 NP$ (Donate)	Any the answer was??? Yes? It should do if the bot respects robots.txt __________________ Regards Jamie Let the might of your compassion arise to bring a quick end to the flowing stream of the blood and tears ..... Please hear my anguished words of truth.

05-05-2007, 07:24 AM	#3 (permalink)
jerometan NamePros Regular Join Date: Feb 2006 Posts: 588 1,620.95 NP$ (Donate)	Nasty bots don't follow robots.txt. robots.txt is something like a "Hey, I want/don't want you to come here, but if you insist, there is nothing I can do " __________________ Budget NP$ Hosting - Affordable CPanel Hosting! [ Click Here For Sales/Support ] 50NP Proxy Hosting \| URLToUs \| Fast Reliable Budget Host! [ Click Here For Sales/Support ]

05-05-2007, 11:14 AM	#4 (permalink)
dan_Vt Senior Member Join Date: Dec 2004 Location: Burlington, Vermont Posts: 1,043 1,086.90 NP$ (Donate)	It tells OmniExplorer_Bot not to follow, other bots are fine. __________________ Vermont SEO Automotive SEO For Sale: Co-Sign.com \| Skiny.net \| CSIFAQ.com \| FuelBank.net \| Jibbar.com \|

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)