Gathering stats from Google

col · 05-11-2006, 06:36 AM

I've done a little php script that is fetching some link stats from google. The script is working fine, but Google doesn't seem to like it... I'm getting the stats by parsing the page I get by file_get_contents() (PHP). After a while the script just didn't want to work and when I checked up on it I found that my automated searches returned a page looking like this: <link>
Anyone had the same problem? Any suggestions on alternative solutions?

BillyConnite · 05-11-2006, 06:54 AM

I'd say that they have blocked your sites IP most likely, as it was probably against their TOS for you to be doing that

.

They block a lot of PR predictors too, but most use various proxies to get the data from google now to spread the work, and lessen any risk of being banned.

Noobie · 05-11-2006, 07:21 AM

Hi,

I got that error yesterday too out of the blue.
Had nothing to do with google stats, just visiting google to search for something.

I typed in the required characters and I haven't been asked since.

col · 05-11-2006, 10:05 AM

Well, I guess I just have to skip their stats at the moment

mikesherov · 05-11-2006, 01:58 PM

Well, the way to do it is to build or use a proxy sniffer to get a list of valid proxies and just build a PHP function that uses proxies to fetch the information.

Well, I leave finding valid proxies a task left to the reader, but what kind of guy would I be if I didn't provide the function that works exactly like file_get_contents, but uses proxies?

PHP Code:

  function proxy_url($server,$port,$proxy_url)

{

   $proxy_name = $server;

   $proxy_port = $port;

   $proxy_cont = '';

 
   $proxy_fp = fsockopen($proxy_name, $proxy_port);

   if (!$proxy_fp)    {return false;}

   fputs($proxy_fp, "GET $proxy_url HTTP/1.0\r\nHost: $proxy_name\r\n\r\n");

   while(!feof($proxy_fp)) {$proxy_cont .= fread($proxy_fp,4096);}

   fclose($proxy_fp);

   $proxy_cont = substr($proxy_cont, strpos($proxy_cont,"\r\n\r\n")+4);

   return $proxy_cont;

}

all you do is provide the proxy server name or ip address, the proxy port, and the the page you want to get.

col · 05-11-2006, 03:31 PM

Quote:

Originally Posted by mikesherov

Well, the way to do it is to build or use a proxy sniffer to get a list of valid proxies and just build a PHP function that uses proxies to fetch the information.

Well, I leave finding valid proxies a task left to the reader, but what kind of guy would I be if I didn't provide the function that works exactly like file_get_contents, but uses proxies?

PHP Code:

  function proxy_url($server,$port,$proxy_url)

{

   $proxy_name = $server;

   $proxy_port = $port;

   $proxy_cont = '';

 
   $proxy_fp = fsockopen($proxy_name, $proxy_port);

   if (!$proxy_fp)    {return false;}

   fputs($proxy_fp, "GET $proxy_url HTTP/1.0\r\nHost: $proxy_name\r\n\r\n");

   while(!feof($proxy_fp)) {$proxy_cont .= fread($proxy_fp,4096);}

   fclose($proxy_fp);

   $proxy_cont = substr($proxy_cont, strpos($proxy_cont,"\r\n\r\n")+4);

   return $proxy_cont;

}

all you do is provide the proxy server name or ip address, the proxy port, and the the page you want to get.

Thanks alot! Reputation added

sacx13 · 05-12-2006, 12:45 PM

Use google api (registration and blah) if you don't have more than 1000 per day.

Regards

Peter · 05-12-2006, 03:52 PM

yes as sacx13 states use the api if you do not have too many queries. The problem with using proxies is that they will soon enough be blocked as well if too many requests get put in.

I have come across the page you get myself many times when searching manually on google.

05-11-2006, 06:36 AM	#1 (permalink)
col NamePros Regular Join Date: Jan 2005 Location: Land of the m00 Posts: 723 140.10 NP$ (Donate)	Gathering stats from Google I've done a little php script that is fetching some link stats from google. The script is working fine, but Google doesn't seem to like it... I'm getting the stats by parsing the page I get by file_get_contents() (PHP). After a while the script just didn't want to work and when I checked up on it I found that my automated searches returned a page looking like this: <link> Anyone had the same problem? Any suggestions on alternative solutions? __________________ The more I think the more confused I get...

05-11-2006, 06:54 AM	#2 (permalink)
BillyConnite Join Date: Jul 2005 Location: Coffs H, Australia Posts: 3,107 47.00 NP$ (Donate)	I'd say that they have blocked your sites IP most likely, as it was probably against their TOS for you to be doing that . They block a lot of PR predictors too, but most use various proxies to get the data from google now to spread the work, and lessen any risk of being banned. __________________ <?php if(1===1){ $computer="fine."; }else{ $computer="broken."; } echo "Your computer is ".$computer; ?>

05-11-2006, 07:21 AM	#3 (permalink)
Noobie NamePros Regular Join Date: Feb 2006 Location: Montreal, Quebec, Canada Posts: 324 66.75 NP$ (Donate)	Hi, I got that error yesterday too out of the blue. Had nothing to do with google stats, just visiting google to search for something. I typed in the required characters and I haven't been asked since. __________________ Goldkey.com is a scam What's your BMI? \| Timestamp Generator

05-11-2006, 10:05 AM	#4 (permalink)
col NamePros Regular Join Date: Jan 2005 Location: Land of the m00 Posts: 723 140.10 NP$ (Donate)	Well, I guess I just have to skip their stats at the moment __________________ The more I think the more confused I get...

05-11-2006, 01:58 PM	#5 (permalink)
mikesherov NamePros Member Join Date: Apr 2005 Posts: 116 134.00 NP$ (Donate)	Well, the way to do it is to build or use a proxy sniffer to get a list of valid proxies and just build a PHP function that uses proxies to fetch the information. Well, I leave finding valid proxies a task left to the reader, but what kind of guy would I be if I didn't provide the function that works exactly like file_get_contents, but uses proxies? PHP Code: function proxy_url($server,$port,$proxy_url) { $proxy_name = $server; $proxy_port = $port; $proxy_cont = ''; $proxy_fp = fsockopen($proxy_name, $proxy_port); if (!$proxy_fp) {return false;} fputs($proxy_fp, "GET $proxy_url HTTP/1.0\r\nHost: $proxy_name\r\n\r\n"); while(!feof($proxy_fp)) {$proxy_cont .= fread($proxy_fp,4096);} fclose($proxy_fp); $proxy_cont = substr($proxy_cont, strpos($proxy_cont,"\r\n\r\n")+4); return $proxy_cont; } all you do is provide the proxy server name or ip address, the proxy port, and the the page you want to get.

05-12-2006, 12:45 PM	#7 (permalink)
sacx13 NamePros Regular Join Date: Mar 2006 Posts: 394 211.38 NP$ (Donate)	Use google api (registration and blah) if you don't have more than 1000 per day. Regards __________________ █ Free Pending&Expired Domains Tracker and more ... █ Webmaster's Talks Revenue Forum █ Webmaster Web Tools █ Hostmaster Web Tools █ PhotoGraphy Portal

05-12-2006, 03:52 PM	#8 (permalink)
Peter Senior Member Join Date: Nov 2003 Location: Scotland Posts: 4,900 0.60 NP$ (Donate)	yes as sacx13 states use the api if you do not have too many queries. The problem with using proxies is that they will soon enough be blocked as well if too many requests get put in. I have come across the page you get myself many times when searching manually on google.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)