robots.txt question

Ringr · Apr 28, 2006

Hey guys,

I've always known that robots.txt is what is responcible for letting robots and spiders access or not access certain parts of your website and whatnot. But I was wondering, how would I go about making a robots.txt that didn't allow any bot whatsoever to search my website and index it?

Thank you!
Andy

weblord · Apr 28, 2006

your robots.txt will contain this

User-Agent: *
Disallow: /

if you want to allow just google
it will contain this;

Code:

    User-Agent: *
    Disallow: /

    User-Agent: Googlebot
    Allow: /

otherwise the first robots.txt is what you need. hth

Eric · Apr 28, 2006

weblord said:
your robots.txt will contain this

if you want to allow just google
it will contain this;

Code:

User-Agent: * Disallow: / User-Agent: Googlebot Allow: /

otherwise the first robots.txt is what you need. hth

Actually your second one is the incorrect syntax, it should be:

Code:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

www.robotstxt.org/wc/norobots.html#format
www.robotstxt.org/wc/norobots.html#examples

stscac · Apr 28, 2006

Well, I'm hoping you don't disallow Gooogle from your root. :o

Unless its a completely private site.

-Steve

Kate · Apr 28, 2006

Also don't assume this will block all robots/spiders.
robots.txt works on a voluntary basis so some spiders decide to ignore it.

Masahiro · Apr 28, 2006

As sdsinc has just mentioned, not all bots follow robots.txt, particularly spam harvesting bots.

However, a bot cannot get around .htaccess - but you'll need to be hosted on Apache for that [if you've got cPanel hosting, you've got Apache.]

Richieboy67 · Apr 28, 2006

This is what I use.... Can anyone see anything wrong with this??? Grab it if you link but it is for vb...

Code:

User-agent: *
Disallow: /admincp/
Disallow: /arcade.php
Disallow: /announcement.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /faq.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /subscriptions.php
Disallow: /threadrate.php
Disallow: /usercp.php

Ringr · Apr 28, 2006

The reason why I need the code is because it is for a blog, which is completely private on my server. Odd thing is, I go onto http://blogsearch.google.com only about 8 hours after I had posted things in it - and I found that Google BlogSearch had already indexed it. How, I don't know - the blog address is not linked to by any website or anything, I am the only person that knows its existance. There are also no ads, etc. on it.

Not sure if the code weblord provided me with will keep BlogSearch away too (since sdsinc said it's more voluntary.) But I guess it's worth a try.

Thanks you guys,
Andy

Richieboy67 · Apr 28, 2006

You want to keep blog search away?? That is your intention??? You can use htaccess also...... I am not familiar with blogsearch to be honest but if you can find out the name of the bot and add it to your robots.txt you should be ok....

Kate · Apr 28, 2006

Masahiro said:
As sdsinc has just mentioned, not all bots follow robots.txt, particularly spam harvesting bots.

However, a bot cannot get around .htaccess - but you'll need to be hosted on Apache for that [if you've got cPanel hosting, you've got Apache.]

As Masahiro suggests .htaccess could do the work for you.

Ringr · May 2, 2006

UGH! WTF is going on? This stupid "Google Blogsearch" keeps publicizing my whole blog to the damn world.

FIRST I stick the robots.txt file in my directory as you guys said, it had the coding in it as it was supposed to. Yet for some reason fricken "BlogSearch" STILL keeps indexing it.

SO, then I stuck I stuck this code under my <head> tag:

Code:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

And despite those efforts, GOOGLE BLOGSEARCH STILL KEEPS INDEXING IT. Ugh, I don't know wtf to do. Talk about invading my damned privacy, it just keeps indexing and indexing and indexing when I've gone more than out of my damn way to make sure it doesn't.

What do I do?

Ringr · May 4, 2006

So can anyone offer me any guidance. I mean, am I doing something wrong here?
I have that robots.txt EXACTLY how weblord said to, and I have that <meta> tag (as described above) in there too.

Is there something I'm missing? :s

eagle12 · May 4, 2006

Why don't you password protect it.

robots.txt question

VIP Member

Top Member

VIP Member

A Wealth of KnowledgeVIP Member

Domainosaurus RexTop Member

Established Member

Account Suspended

VIP Member

Account Suspended

Domainosaurus RexTop Member

VIP Member

VIP Member

Established Member

Similar threads

We're social

tacumbu.com

Spotcher.Com

gy.vc

Leggings.gr

IoTjourney.com

iGotADeal.com

ApoCoin.com

ONUSD.com

ILoveSolar.com

AgriArea.com

Pinned

Appreciation

Agreement

Answers

Relevance

Reaction

Status

Feeling