Robot.txt Help

gobbly2100 · Jan 3, 2007

If I want to allow all indexing of my site from search engines what to I need to put in a robot.txt file, I currently have no robot.txt files on my site but my adsense is saying it is getting blocked somehow and apparently I need to make a robot .txt

Any help is much appreciated.

nmr86 · Jan 3, 2007

Put this in your meta description (between your <head> and </head>

<meta name="robots" content="index, follow"><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

gobbly2100 · Jan 3, 2007

I was hoping to make an actual robot.txt

Thanks for the help anyway

enlytend · Jan 3, 2007

Meta's aren't the same as having a robots.txt - certain spiders expect/want to find a robots.txt file in your web root.

Here's a very basic robots.txt which allows everybody access to everything:

User-agent: *
Disallow:

To specifically grant access to the Adsense bot, you could also have this in your file:

User-agent: Mediapartners-google*
Disallow:

majorkontrapelo · Jan 7, 2007

enlytend said:
Meta's aren't the same as having a robots.txt - certain spiders expect/want to find a robots.txt file in your web root.

Here's a very basic robots.txt which allows everybody access to everything:

User-agent: *
Disallow:

To specifically grant access to the Adsense bot, you could also have this in your file:

User-agent: Mediapartners-google*
Disallow:

Yeah this one is right, as I've seen in the first post, be reminded that it is robots.txt and not robot.txt - be sure to have the "s"

Robot.txt Help

Established Member

Established Member

Established Member

Top Member

VIP Member

kheg.com

RoboDigest.com

Madagascan.com

kourser.com

affluenceexchange.com

Soccermena.com

breathe.free

AccCoin.com

Marwil.com

P-C-I.com

Similar threads

We're social

kheg.com

RoboDigest.com

Madagascan.com

kourser.com

affluenceexchange.com

Soccermena.com

breathe.free

AccCoin.com

Marwil.com

P-C-I.com

Pinned

Appreciation

Agreement

Answers

Relevance

Reaction

Status

Feeling