Robot.txt Help

gobbly2100 · Jan 3, 2007

If I want to allow all indexing of my site from search engines what to I need to put in a robot.txt file, I currently have no robot.txt files on my site but my adsense is saying it is getting blocked somehow and apparently I need to make a robot .txt

Any help is much appreciated.

nmr86 · Jan 3, 2007

Put this in your meta description (between your <head> and </head>

<meta name="robots" content="index, follow"><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

gobbly2100 · Jan 3, 2007

I was hoping to make an actual robot.txt

Thanks for the help anyway

enlytend · Jan 3, 2007

Meta's aren't the same as having a robots.txt - certain spiders expect/want to find a robots.txt file in your web root.

Here's a very basic robots.txt which allows everybody access to everything:

User-agent: *
Disallow:

To specifically grant access to the Adsense bot, you could also have this in your file:

User-agent: Mediapartners-google*
Disallow:

majorkontrapelo · Jan 7, 2007

enlytend said:
Meta's aren't the same as having a robots.txt - certain spiders expect/want to find a robots.txt file in your web root.

Here's a very basic robots.txt which allows everybody access to everything:

User-agent: *
Disallow:

To specifically grant access to the Adsense bot, you could also have this in your file:

User-agent: Mediapartners-google*
Disallow:

Yeah this one is right, as I've seen in the first post, be reminded that it is robots.txt and not robot.txt - be sure to have the "s"

bidder.de

jtv8811.pro

lafleurgalante.fr

maviaferrata.fr

besnier-audrey-sage-femme.fr

deskonlinevest.org

bollyflix.taxi

mellkrku.life

smartmart.watch

Robot.txt Help

Established Member

Established Member

Established Member

Top Member

VIP Member

hondabigwing.com

Gesture.com

SweetWise.com

DropCash.com

X.tools

betbinary.com

TasksDaily.com

EcoCat.com

MommaBear.com

gemado.com

Similar threads

We're social

hondabigwing.com

Gesture.com

SweetWise.com

DropCash.com

X.tools

betbinary.com

TasksDaily.com

EcoCat.com

MommaBear.com

gemado.com

Pinned

Appreciation

Preference

Appreciation

Upvoting

Agreement

Answers

Decisions

Relevance

Reaction

Facial expressions

Vocal expressions

Status

Workflow

Clarification

Feeling

Sympathy