Protecting ourselves from malicious spiders

gattoplano · 11-02-2005, 04:44 AM

Hi,
I'm building a directory containing images. Now, I want to prevent this directory from being lurked from malicious spiders, sucking all of its contents.

Then, I want search engines' spiders to navigate it deeply, and to find all of the images and contents.

I was thinking about limiting the number of images/day views for common users and to set no limitations for search engines. But it sounds so bad :\

Should I use some other cloaking technique? Is there any good article around about this issue?

puzzlebox · 11-02-2005, 10:17 AM

try searching for "robots.txt". It's some sort of spider-limiter, although if it's a malicious spider I doubt if it will stop at any limitations, whether cloaking or not.

stscac · 11-02-2005, 11:10 AM

You can forbid direct folder access by changing chmod to something less than 755.

-Steve

gattoplano · 11-02-2005, 01:28 PM

I can't as my directory is structured so that common users must have access to any point in any moment.

I just want to limit the amount of information that someone can view. I would like to assume that it's impossible for a human to open and read carefully more than 10 directories in a minute, or to see more than 25 images, and limit access for content stealers while leaving it open for search engines.

Spider Ninja · 11-07-2005, 06:01 AM

try mod rewrite:

http://httpd.apache.org/docs/2.0/misc/rewriteguide.html

and search the web for "spider user agent" ...that's a way to start.

11-02-2005, 04:44 AM	THREAD STARTER #1 (permalink)
gattoplano NamePros Regular Join Date: Sep 2005 Location: Roma Posts: 591	Protecting ourselves from malicious spiders Hi, I'm building a directory containing images. Now, I want to prevent this directory from being lurked from malicious spiders, sucking all of its contents. Then, I want search engines' spiders to navigate it deeply, and to find all of the images and contents. I was thinking about limiting the number of images/day views for common users and to set no limitations for search engines. But it sounds so bad :\ Should I use some other cloaking technique? Is there any good article around about this issue?

11-07-2005, 06:01 AM	#5 (permalink)
Spider Ninja NamePros Member Join Date: Nov 2005 Posts: 41	try mod rewrite: http://httpd.apache.org/docs/2.0/misc/rewriteguide.html and search the web for "spider user agent" ...that's a way to start. __________________ webmaster tools

11-02-2005, 10:17 AM	#2 (permalink)
puzzlebox New Member Join Date: Oct 2005 Posts: 2	try searching for "robots.txt". It's some sort of spider-limiter, although if it's a malicious spider I doubt if it will stop at any limitations, whether cloaking or not.

11-02-2005, 11:10 AM	#3 (permalink)
stscac A Wealth of Knowledge Join Date: Aug 2004 Posts: 3,809	You can forbid direct folder access by changing chmod to something less than 755. -Steve

11-02-2005, 01:28 PM	THREAD STARTER #4 (permalink)
gattoplano NamePros Regular Join Date: Sep 2005 Location: Roma Posts: 591	I can't as my directory is structured so that common users must have access to any point in any moment. I just want to limit the amount of information that someone can view. I would like to assume that it's impossible for a human to open and read carefully more than 10 directories in a minute, or to see more than 25 images, and limit access for content stealers while leaving it open for search engines.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)