(sorry for posting this here, but I'm getting an error when trying to post in the *proper* sub-forum, so I don't know what else to do... mods, feel free to move this)
Hello people.
I've got a minor situation... 2 possibilities here:
1: It seems like the Yahoo Slurp bot has issues... serious ones, I think.
Take a look at this:
These are my site's top visitors (by bandwidth usage). The address that ends with 208 is me. All the others until you reach the one ending in 183 are the Yahoo Slurp bot (according to IP traces
[here]).
The bot has gobbled up about 20 gigs of bandwidth in just the last 20 days. I don't get it. I have about 50 pages, a sleepy forum, and get about 5,000 visits per month... There really isn't much going on here.
________
2: I'm wondering if it isn't perhaps a sneaky site stalker using fake IP's. First, does this make sense to anyone? I'm not jumping to any conclusions yet.
If it is, there's a certain person that I used to be in contact with that I suspect (would make a lot of sense), but I haven't been able to tell for certain... IF this were the case, I would think they're using some program like UpdatePatrol or NeoDownloader (or both) to keep track of changed pages/dirs, and download whatever's new or modified.
The activity coming from these IP's has increased gradually in the past months... starting from around 2 gigs per month, and growing quickly in the past couple months to 8 gigs, 12 gigs, and now 20 (and counting, because this month isnt over)...
Any ideas? Is there any way to lure a visitor into a trap that bots can't use, that would prove that it's a human and not a bot?
Otherwise, if it IS a bot, what on Earth is Yahoo thinking? The MSN and Google bots don't use even NEAR this amount of bandwidth.
Btw, I just added a robots.txt to my site, to block the Yahoo bot entirely. I assume if this is a human, I'll figure it out next month, when Webalyzer starts showing a new month... and then I can see if the robots.txt hasn't driven away those IP's entirely.
My robots.txt says this:
Code:
User-Agent: Yahoo! Slurp
Disallow: /
