[advanced search]
11 members in the live chat room. Join Chat!
Register Rules & FAQ NP$ Store Active Threads Mark Forums Read
Go Back   NamePros.Com > Design and Development > Programming > CODE
User Name
Password

Old 09-29-2006, 02:05 PM   · #1
Mikor
Resident Linux Geek
 
Mikor's Avatar
 
Name: Michael Walker
Location: East Yorkshire, England
Trader Rating: (7)
Join Date: Aug 2005
Posts: 2,413
NP$: 300.25 (Donate)
Mikor is a splendid one to beholdMikor is a splendid one to beholdMikor is a splendid one to beholdMikor is a splendid one to beholdMikor is a splendid one to beholdMikor is a splendid one to behold
Simple Search Engine

Demo

What this does is search a pages contents for a keyword, if the keyword is found, the uri is added to an array, and every page on the array is scanned for links, which are then scanned for the keyword, etc.
The array returned is a user-definable size, and it includes options to limit the spider to only crawl specific domains.

It's a bit slow because it indexes all the pages on the fly, if you made it store them in a database, and then search through the database it would be so much faster, but this is just a simple bit of code for people. Also: because it can be slow, I wouldnt recommend setting the max number of pages to much more than 100, thats what the demo is set to and you can see how slow it is (over a minute and a half, lol)

ToDo:
  • robots.txt support
  • <meta> robots support
  • speed it up a bit


Please register or log-in into NamePros to hide ads
Attached Files
File Type: zip Spider.zip (1.5 KB, 13 views)
__________________
Me | Blog | Last.fm | F@h

archlinux User
Mikor is offline   Reply With Quote
Old 10-04-2006, 09:51 PM   · #2
kleszcz
 
kleszcz's Avatar
 
Trader Rating: (229)
Join Date: Jul 2006
Posts: 3,257
NP$: 535.45 (Donate)
kleszcz has a reputation beyond reputekleszcz has a reputation beyond reputekleszcz has a reputation beyond reputekleszcz has a reputation beyond reputekleszcz has a reputation beyond reputekleszcz has a reputation beyond reputekleszcz has a reputation beyond reputekleszcz has a reputation beyond reputekleszcz has a reputation beyond reputekleszcz has a reputation beyond reputekleszcz has a reputation beyond repute
Marrow Donor Program Multiple Sclerosis
Yeah, I think additional caching function would make it awesome search engine script. Looking forward to stable version.
__________________
BQB.com auction: UJMH.com
LLLL.coms: ALL 2 Premium Letters + U + K
LLLL.com - *going cheap*
kleszcz is offline   Reply With Quote
Old 10-20-2006, 01:16 PM   · #3
boebi
NamePros Regular
 
Location: Inside your firmware.......
Trader Rating: (7)
Join Date: Oct 2006
Posts: 216
NP$: 0.00 (Donate)
boebi will become famous soon enoughboebi will become famous soon enough
Yeah, if would have some more speed, and some better stuff (like better design,caching),
i would run it on one of my site
boebi is offline   Reply With Quote
Old 11-01-2006, 08:01 AM   · #4
wolvenraider
NamePros Member
 
Location: Belgium
Trader Rating: (3)
Join Date: Mar 2006
Posts: 77
NP$: 289.00 (Donate)
wolvenraider is an unknown quantity at this point
Originally Posted by boebi
Yeah, if would have some more speed, and some better stuff (like better design,caching),
i would run it on one of my site



Actually I don't see the problem with the layout, it can be easily edited in the demo.php cpage, a simple fieldset would make it good. But that's probably my point of view as a website creator
wolvenraider is offline   Reply With Quote
Closed Thread

NamePros is a revenue sharing forum.

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump


Site Sponsors
Custom Logo Design Build your NameBrand Get Me Visits
Advertise your business at NamePros
All times are GMT -7. The time now is 11:54 PM.


Powered by: vBulletin Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Friendly URLs by vBSEO 2.4.0