Gathering stats from Google

[col]

I've done a little php script that is fetching some link stats from google. The script is working fine, but Google doesn't seem to like it... I'm getting the stats by parsing the page I get by file_get_contents() (PHP). After a while the script just didn't want to work and when I checked up on it I found that my automated searches returned a page looking like this: <link>
Anyone had the same problem? Any suggestions on alternative solutions?

 

[yeppers]

I'd say that they have blocked your sites IP most likely, as it was probably against their TOS for you to be doing that .

They block a lot of PR predictors too, but most use various proxies to get the data from google now to spread the work, and lessen any risk of being banned.

 

[Noobie]

Hi,

I got that error yesterday too out of the blue.
Had nothing to do with google stats, just visiting google to search for something.

I typed in the required characters and I haven't been asked since.

 

[col]

Well, I guess I just have to skip their stats at the moment

 

[mikesherov]

Well, the way to do it is to build or use a proxy sniffer to get a list of valid proxies and just build a PHP function that uses proxies to fetch the information.

Well, I leave finding valid proxies a task left to the reader, but what kind of guy would I be if I didn't provide the function that works exactly like file_get_contents, but uses proxies?

function proxy_url($server,$port,$proxy_url)
{
   $proxy_name = $server;
   $proxy_port = $port;
   $proxy_cont = '';

   $proxy_fp = fsockopen($proxy_name, $proxy_port);
   if (!$proxy_fp)    {return false;}
   fputs($proxy_fp, "GET $proxy_url HTTP/1.0\r\nHost: $proxy_name\r\n\r\n");
   while(!feof($proxy_fp)) {$proxy_cont .= fread($proxy_fp,4096);}
   fclose($proxy_fp);
   $proxy_cont = substr($proxy_cont, strpos($proxy_cont,"\r\n\r\n")+4);
   return $proxy_cont;
}

all you do is provide the proxy server name or ip address, the proxy port, and the the page you want to get.

 

[col]

Well, the way to do it is to build or use a proxy sniffer to get a list of valid proxies and just build a PHP function that uses proxies to fetch the information.

Well, I leave finding valid proxies a task left to the reader, but what kind of guy would I be if I didn't provide the function that works exactly like file_get_contents, but uses proxies?

function proxy_url($server,$port,$proxy_url)
{
   $proxy_name = $server;
   $proxy_port = $port;
   $proxy_cont = '';

   $proxy_fp = fsockopen($proxy_name, $proxy_port);
   if (!$proxy_fp)    {return false;}
   fputs($proxy_fp, "GET $proxy_url HTTP/1.0\r\nHost: $proxy_name\r\n\r\n");
   while(!feof($proxy_fp)) {$proxy_cont .= fread($proxy_fp,4096);}
   fclose($proxy_fp);
   $proxy_cont = substr($proxy_cont, strpos($proxy_cont,"\r\n\r\n")+4);
   return $proxy_cont;
}

all you do is provide the proxy server name or ip address, the proxy port, and the the page you want to get.

Thanks alot! Reputation added

 

[sacx13]

Use google api (registration and blah) if you don't have more than 1000 per day.

Regards

 

[Peter]

yes as sacx13 states use the api if you do not have too many queries. The problem with using proxies is that they will soon enough be blocked as well if too many requests get put in.

I have come across the page you get myself many times when searching manually on google.