Need help from experts

[earn4sure]

Hi,

I hope anybody help my friend out of this problem.

My friend's domain has 700 MB/per month bandwidth Limit. Now when he checked 2 days back there is mostly bandwidth used his search engines.Please see below given status.

11 different robots* Hits Bandwidth Last visit
MSNBot 3191 66.75 MB 30 Nov 2005 - 02:39
Unknown robot (identified by 'crawl') 1008 380.99 MB 27 Nov 2005 - 07:54
Googlebot 296 106.83 MB 30 Nov 2005 - 01:04
Inktomi Slurp 167 2.33 MB 29 Nov 2005 - 23:22
Unknown robot (identified by 'spider') 154 3.16 MB 28 Nov 2005 - 19:47
AskJeeves 111 3.61 MB 30 Nov 2005 - 01:58
Alexa (IA Archiver) 60 1.42 MB 27 Nov 2005 - 22:25
LinkWalker 19 252.01 KB 25 Nov 2005 - 07:45
WISENutbot 17 368.72 KB 26 Nov 2005 - 07:11
Walhello appie 8 148.45 KB 23 Nov 2005 - 20:32
Unknown robot (identified by 'robot') 1 24.19 KB 19 Nov 2005 - 00:45

So please tell me how he prevent/save his bandwidth from unwanted search engines.

Regards.

 

[Palyriot]

in the robots.txt he can disable robots from going to specific pages. It will dramatically drop his bandwidth. If the bots can only view his index, then he will be in good shape.

 

[earn4sure]

How to make robots.txt ? He use google adsense (adsense publisher). So is there any problem after stop spiders.

 

[dongan]

http://www.robotstxt.org/wc/robots.html

 

[JFS]

be adviced that blocking some spiders can mean less visitors from search engines

 

[Palyriot]

not if its like an admin section or something. Also config.php files don't need to be searched.

 

[JFS]

so what if they are searched?
they don't output any html code
it only apears a blank page

 

[Palyriot]

what??

 

[vslevin]

You do not want to block search engine bots, thats how you get indexed and ranked. It looks like Google and MSN are doing a full index of your site, which is a good thing, and is definitely worth the bandwidth. It won't always be as heavy after the initial indexing.

 

[Genialnames]

I had a similar problem sometime ago, especially googleboot seemed liked my site a lot.

I found in the webmasters section of google site a reference to that problem.
You just write to them and tell them you have such problem.
They will reply in a week or so asking you to provide details,(so if you provide these data first you may reduce the time for answer)
They are able in some way to reduce the robot visits, at least it worked for me.
Hope it help

 

[earn4sure]

Thank u all. I think thats my wrong decision to block search engine to visit my website. Thanks to all members for your proper help.