Protecting ourselves from malicious spiders

[gattoplano]

Hi,
I'm building a directory containing images. Now, I want to prevent this directory from being lurked from malicious spiders, sucking all of its contents.

Then, I want search engines' spiders to navigate it deeply, and to find all of the images and contents.

I was thinking about limiting the number of images/day views for common users and to set no limitations for search engines. But it sounds so bad :\

Should I use some other cloaking technique? Is there any good article around about this issue?

 

[puzzlebox]

try searching for "robots.txt". It's some sort of spider-limiter, although if it's a malicious spider I doubt if it will stop at any limitations, whether cloaking or not.

 

[stscac]

You can forbid direct folder access by changing chmod to something less than 755.

-Steve

 

[gattoplano]

I can't as my directory is structured so that common users must have access to any point in any moment.

I just want to limit the amount of information that someone can view. I would like to assume that it's impossible for a human to open and read carefully more than 10 directories in a minute, or to see more than 25 images, and limit access for content stealers while leaving it open for search engines.

 

[Spider Ninja]

try mod rewrite:

http://httpd.apache.org/docs/2.0/misc/rewriteguide.html

and search the web for "spider user agent" ...that's a way to start.