How to make your site Spider-Friendly in 5 steps.

[Barrucadu]

These techniques will help search engine spiders index your site better:

1) Include a sitemap that has a link to every page on the site that you want to index (basicly all non-admin, non-members pages)

2) Include at least one static text link to all your main pages on the index page.

3) Never use relative urls, use the full url: eg http://www.yourdomain.com/page.php instead of page.php

4) Goto this page and download the freeware spider Phenix Spider (first file) and spider your page.

5) If you did steps 1 - 3 correctly Phenix Spider should find all of the pages you want to index.

 

[stscac]

Thanks for the Tips

Not sure that I completely agree with #3 (It might come across as a problem within the Spidering program, beause they are not problems in the search engines).

I haven't tried the software yet, but might give it a look

Thanks for the contribution

-Steve

 

[ignat]

Nice tips. Do you mind elaborating more on the sitemap thing. I created and submitted Google Sitemaps. Do you also meen creating a static page that is designed to be a sitemap?

 

[Barrucadu]

Nice tips. Do you mind elaborating more on the sitemap thing. I created and submitted Google Sitemaps. Do you also meen creating a static page that is designed to be a sitemap?

hmm I havent tried google sitemaps yet, but I did mean a static page, heres mine

http://www.myfileminder.com/sitemap.html

 

[rehash]

I also disagree with #3

 

[ignat]

hmm I havent tried google sitemaps yet, but I did mean a static page, heres mine

http://www.myfileminder.com/sitemap.html

Gotcha. I think you should try using Google Sitemaps. You can tell directly to Google which pages to crawl and how often they are updated. I did some research on this at my SEO Blog . Here's the article URL: http://www.hawaiistreets.com/seoblog/seoblog.php?itemid=92

The sweet thing about Google Sitemaps is that it allows Google to crawl your page sooner. Especially if you're just starting out. Results do vary. Let me know how this works out.

 

[Barrucadu]

Gotcha. I think you should try using Google Sitemaps. You can tell directly to Google which pages to crawl and how often they are updated. I did some research on this at my SEO Blog . Here's the article URL: http://www.hawaiistreets.com/seoblog/seoblog.php?itemid=92

The sweet thing about Google Sitemaps is that it allows Google to crawl your page sooner. Especially if you're just starting out. Results do vary. Let me know how this works out.

whats the URL? I cant find a link, also google crawled my page this morning.

 

[bentong]

IMO, just submit your sitemap on search engines..

 

[ignat]

this is the URL for some information on Google Sitemap: http://www.google.com/webmasters/sitemaps/docs/en/about.html

A sitemap is very easy to create. Just look at examples Google gives. Once you write it up, using XML file, submit it to Google using your Gmail account id and password.

 

[netzilla]

I just wanted to add for those who may not know...while the steps above may make your site spider friendly it does not necessairly mean that what is spidered and indexed is going to be exactly what you want your visitors to see. For more information on some whitehat simple SEO techniques: http://blog.cancerdaily.com/?p=6

 

[dre]

if your site is dynamic with url such as file.php?id=2&type=jpg, try mod_rewrite using .htaccess. When you have done mod_rewrite your urlwill be something like www.domain.com/file/id/2/type/jpg this will be easier for search engines to pick up!

PM if u want mod_rewriting done

check out www.tutorialselect.com, great use of mod rewrite there! :D

 

[ignat]

You can't do mod rewrite on Wondows based servers though. Not for free. You can do it for free on Apache servers. I have a Windows server

 

[simpson733]

thanks ill try out the script after i finish coding my design site

 

[dre]

You can't do mod rewrite on Wondows based servers though. Not for free. You can do it for free on Apache servers. I have a Windows server

Oh i didnt know that! I on a apache server
:kickass:

 

[tolkein]

I haven't had much success with Google Sitemaps yet - 3 out of 4 errors. I trying to take the shortcut of using .txt and I'm getting a "parsing error". Will update here once I succeed.