RUPERT

Could anyone please explain how I do this?

According to their site, I need to place a simple robots.txt file on my Web server, but I'm not exactly sure what they mean by this, or how this is done.


Thanks

nielsencl

All you have to do is create a simple text file and name it robots.txt. Then you have to add this information to the file:

User-agent: ia_archiver
Disallow: /

Upload the file to the folder where your site files are.

This info is from their page about this, but they don't exactly spell it out as clearly as they could.

If you are already in Archive.org I don't think this will remove any information they have for your site.

__________________
NameAuction.es - Sell Gold - Homecare Software - Asbestos - This Is Funny - NR Auction

weblord

The Internet Archive is not interested in offering access to Web sites or other Internet documents whose authors do not want their materials in the collection. To remove your site from the Wayback Machine, place a robots.txt file at the top level of your site (e.g. www.yourdomain.com/robots.txt) and then submit your site below.

The robots.txt file will do two things:

1. It will remove all documents from your domain from the Wayback Machine.
2. It will tell us not to crawl your site in the future.

To exclude the Internet Archive’s crawler (and remove documents from the Wayback Machine) while allowing all other robots to crawl your site, your robots.txt file should say:

User-agent: ia_archiver
Disallow: /

Robots.txt is the most widely used method for controlling the behavior of automated robots on your site (all major robots, including those of Google, Alta Vista, etc. respect these exclusions). It can be used to block access to the whole domain, or any file or directory within. There are a large number of resources for webmasters and site owners describing this method and how to use it. Here are some:

* http://www.robotstxt.org/
* http://pageresource.com/zine/robotstxt.htm

Once you have put a robots.txt file up, submit your site (www.yourdomain.com) on the form on http://pages.alexa.com/help/webmaste...tml#crawl_site.

The robots.txt file must be placed at the root of your domain (www.yourdomain.com/robots.txt). If you cannot put a robots.txt file up, read our exclusion policy. If you think it applies to you, send a request to us at info@archive.org.
http://www.archive.org/about/exclude.php