Building a web site crawler

Personaltrainer · 05-10-2007, 09:11 AM

Hi,
We are in the process of building a cutomised site crawlers. We are quiet successful in building one. But I have a question for the expert coders. Is it possible to fetch last modified data of a page from anywhere if so how is it done?

-NC- · 05-10-2007, 02:12 PM

Found this:

14.29 Last-Modified

The Last-Modified entity-header field indicates the date and time at which the origin server believes the variant was last modified.

Last-Modified = "Last-Modified" ":" HTTP-date

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

Might only work for static pages where the server can tell when the content has changed though. (e.g. web server has no idea if the database content has changed)

Barts · 05-12-2007, 02:50 PM

The http last-modified header only works if your webserver supports it. If you are crawling sites on servers that are not yours, you shouldn't rely on that header. But besides that I have the feeling that you would like to only fetch the differences since the last update. Right? If so, than the answer is that you should code it yourself. Fetch a page, save that copy in a database (or file if you like) and compare the saved copy with the online version. Thazzz how you should do it.

Personaltrainer · 05-28-2007, 06:44 AM

Thanks buddies, But I could not find a practical solution for this. May be its yet to be resolved for dynamic sites

DylanButler · 05-30-2007, 11:20 PM

05-10-2007, 09:11 AM	#1 (permalink)
Personaltrainer New Member Join Date: May 2007 Posts: 4 0.00 NP$ (Donate)	Building a web site crawler Hi, We are in the process of building a cutomised site crawlers. We are quiet successful in building one. But I have a question for the expert coders. Is it possible to fetch last modified data of a page from anywhere if so how is it done? __________________ Personal Trainer\| Fitness Blog\| Fitness Trainer\| Bed Sale Templates with CMS at 50$ http://www.affordablewebsolutions.com/ready-templates.php

05-10-2007, 02:12 PM	#2 (permalink)
-NC- Traveller Join Date: Mar 2007 Location: Yet another city Posts: 1,392 614.57 NP$ (Donate)	Found this: 14.29 Last-Modified The Last-Modified entity-header field indicates the date and time at which the origin server believes the variant was last modified. Last-Modified = "Last-Modified" ":" HTTP-date http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html Might only work for static pages where the server can tell when the content has changed though. (e.g. web server has no idea if the database content has changed) __________________ Internet.geek.nz NameCooler.com Unlimited Domain Name Web Hosting Travel Money Rates

05-12-2007, 02:50 PM	#3 (permalink)
Barts NamePros Member Join Date: Feb 2006 Location: Online Posts: 111 100.80 NP$ (Donate)	The http last-modified header only works if your webserver supports it. If you are crawling sites on servers that are not yours, you shouldn't rely on that header. But besides that I have the feeling that you would like to only fetch the differences since the last update. Right? If so, than the answer is that you should code it yourself. Fetch a page, save that copy in a database (or file if you like) and compare the saved copy with the online version. Thazzz how you should do it. __________________ A soul?... I've got not use for such frivolities.

05-28-2007, 06:44 AM	#4 (permalink)
Personaltrainer New Member Join Date: May 2007 Posts: 4 0.00 NP$ (Donate)	Thanks buddies, But I could not find a practical solution for this. May be its yet to be resolved for dynamic sites __________________ Personal Trainer\| Fitness Blog\| Fitness Trainer\| Bed Sale Templates with CMS at 50$ http://www.affordablewebsolutions.com/ready-templates.php

05-30-2007, 11:20 PM	#5 (permalink)
DylanButler NamePros Regular Join Date: Jan 2006 Location: San Diego, CA Posts: 704 0.00 NP$ (Donate)	more posts from you guys please __________________ :great: -Dylan Butler EXAMP - San Diego Web Design

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)