Get all links from a page

Barrucadu · 07-29-2007, 01:17 PM

This code will get all links from a page, example. I developed it as part of a simple spider i'm working on.

This is what i'm using it for, obviously it's not finished, but I think its a pretty good (if strange) idea. Needs JavaScript. Only tested in Opera.

PHP Code:

 
<pre><?php

 
$url = $_GET['url'];

$html = file_get_contents($url);

$preg = array();

$base = array();

$links = array();

$parsed = parse_url($url);

 
preg_match_all("/\<a(\s*)href(\s*)=(\s*)\"(.*?)\"(.*?)\>(.*?)\<\/a\>/i", $html, $preg[0]);

preg_match_all("/\<a(\s*)href(\s*)=(\s*)'(.*?)'(.*?)\>(.*?)\<\/a\>/i", $html, $preg[1]);

preg_match("/\<base(\s*)href(\s*)=(\s*)\"(.*?)\"(\s*)\/\>/i", $html, $base);

 
$title = array_merge($preg[0][6], $preg[1][6]);

$href = array_merge($preg[0][4], $preg[1][4]);

????: NamePros.com http://www.namepros.com/code/355541-get-all-links-from-a-page.html
$base = $base[4];

 
if(empty($base))

    $base = (!empty($parsed['user'])) ? "{$parsed['scheme']}://{$parsed['user']}:{$parsed['pass']}@{$parsed['host']}" : "{$parsed['scheme']}://{$parsed['host']}";

 
for($i = 0; $i < count($href); $i ++){

    if(substr($href[$i], 0, 1) == '/')

        $href[$i] = "{$base}{$href[$i]}";

    if(substr($href[$i], 0, 1) == '?' || substr($href[$i], 0, 1) == '#')

        $href[$i] = "{$url}{$href[$i]}";

????: NamePros.com http://www.namepros.com/showthread.php?t=355541
    $links[$i] = array("title" => htmlentities($title[$i]), "url" => htmlentities($href[$i]));

}

 
print_r($links);

 
?></pre>

Daniel · 08-04-2007, 11:13 AM

I like it!

Very good potential on this script, thanks for posting.

You don't suppose you could post/zip the other files, .css, .js (although I think it's inline), and .php - We could of course source them, but it's polite to ask.

Thanks,

Dan

Joseph · 08-05-2007, 04:45 PM

Thanks

Repped

Barrucadu · 08-06-2007, 12:07 AM

Here you go, make sure to set the url it will be hosted at in class.php
http://mikor.clearlyhosted.org/spider/zipped.rar

Daniel · 08-06-2007, 12:54 PM

Is this Open Source, unrestricted code? I have a commercial use of this, I can send you a finished script with Resell Rights in exchange for full reseller rights usage of the code.

Thanks,

Dan.

(P.S. Please don't say no

)

Barrucadu · 08-06-2007, 03:22 PM

Originally Posted by Danltn

Is this Open Source, unrestricted code? I have a commercial use of this, I can send you a finished script with Resell Rights in exchange for full reseller rights usage of the code.
????: NamePros.com http://www.namepros.com/showthread.php?t=355541

Thanks,

Dan.

(P.S. Please don't say no

)

Of course.

07-29-2007, 01:17 PM	THREAD STARTER #1 (permalink)
Barrucadu Senior Member Join Date: Aug 2005 Location: East Yorkshire, England Posts: 2,689 Follow @barrucadu	Get all links from a page This code will get all links from a page, example. I developed it as part of a simple spider i'm working on. This is what i'm using it for, obviously it's not finished, but I think its a pretty good (if strange) idea. Needs JavaScript. Only tested in Opera. PHP Code: <pre><?php $url = $_GET['url']; $html = file_get_contents($url); $preg = array(); $base = array(); $links = array(); $parsed = parse_url($url); preg_match_all("/\<a(\s)href(\s)=(\s)\"(.?)\"(.?)\>(.?)\<\/a\>/i", $html, $preg[0]); preg_match_all("/\<a(\s)href(\s)=(\s)'(.?)'(.?)\>(.?)\<\/a\>/i", $html, $preg[1]); preg_match("/\<base(\s)href(\s)=(\s)\"(.?)\"(\s)\/\>/i", $html, $base); $title = array_merge($preg[0][6], $preg[1][6]); $href = array_merge($preg[0][4], $preg[1][4]); ????: NamePros.com http://www.namepros.com/code/355541-get-all-links-from-a-page.html $base = $base[4]; if(empty($base)) $base = (!empty($parsed['user'])) ? "{$parsed['scheme']}://{$parsed['user']}:{$parsed['pass']}@{$parsed['host']}" : "{$parsed['scheme']}://{$parsed['host']}"; for($i = 0; $i < count($href); $i ++){ if(substr($href[$i], 0, 1) == '/') $href[$i] = "{$base}{$href[$i]}"; if(substr($href[$i], 0, 1) == '?' \|\| substr($href[$i], 0, 1) == '#') $href[$i] = "{$url}{$href[$i]}"; ????: NamePros.com http://www.namepros.com/showthread.php?t=355541 $links[$i] = array("title" => htmlentities($title[$i]), "url" => htmlentities($href[$i])); } print_r($links); ?></pre> __________________ Website \| Blog* Arch Hurd developer; GNU Webmaster; FSF member #8385 Last edited by Mikor; 07-29-2007 at 03:23 PM.

08-04-2007, 11:13 AM	#2 (permalink)
Daniel Danltn.com Join Date: May 2007 Location: Danltn.com / Nottingham, UK Posts: 1,201	I like it! Very good potential on this script, thanks for posting. You don't suppose you could post/zip the other files, .css, .js (although I think it's inline), and .php - We could of course source them, but it's polite to ask. Thanks, Dan __________________ :lala: *VERY CHEAP ADVANCED* BULK DOMAIN-STATISTICS LOOKUP** :kickass: Danltn's Blog - Daniel Neville's Site Spoof your links! Submit your blog NOW! Putting bloggers on the map!

08-05-2007, 04:45 PM	#3 (permalink)
Joseph Senior Member Join Date: Dec 2004 Location: Ireland Posts: 2,684	Thanks Repped __________________ - ForumCures.com Quality Forum Posting Service

08-06-2007, 12:07 AM	THREAD STARTER #4 (permalink)
Barrucadu Senior Member Join Date: Aug 2005 Location: East Yorkshire, England Posts: 2,689 Follow @barrucadu	Here you go, make sure to set the url it will be hosted at in class.php http://mikor.clearlyhosted.org/spider/zipped.rar __________________ Website \| Blog Arch Hurd developer; GNU Webmaster; FSF member #8385

08-06-2007, 12:54 PM	#5 (permalink)
Daniel Danltn.com Join Date: May 2007 Location: Danltn.com / Nottingham, UK Posts: 1,201	Is this Open Source, unrestricted code? I have a commercial use of this, I can send you a finished script with Resell Rights in exchange for full reseller rights usage of the code. Thanks, Dan. (P.S. Please don't say no ) __________________ :lala: *VERY CHEAP ADVANCED* BULK DOMAIN-STATISTICS LOOKUP** :kickass: Danltn's Blog - Daniel Neville's Site Spoof your links! Submit your blog NOW! Putting bloggers on the map!

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)