Question on Whois info extraction

[STDomains]

Hi All,

Anyone have any ideas on programmable whois info extraction given a specific domain name?

What we are trying to accomplish, is to be able to extract whois info for a number of domains that we own and get it into a database.

We found a couple of good threads at NP (http://www.namepros.com/code/280961-php-bulk-domain-availability-check.html and one or 2 other threads linked from that one).. but not sure if the threads are doing exactly what we need.

Is there a "format" that all automated whois servers are supposed to conform to on their responses?

Any thoughts/direction would be much appreciated!

Thanks,

ST Domains

 

[Peter]

No there is no set format returned from whois. Different registries return data slightly different.

What precise information are you after?

There should be code within the code section that does what you want as I know several people have written classes and functions for whois.

 

[monaco]

Certain items are pretty constant, like the Administrative Contact and Technical / Tech contact fields. Maybe some simplistic regex parsing could get the most vital information out of the whois data.

 

[STDomains]

Thanks for your responses Peter and Monaco.

Its unfortunate that there is no standard. Seems that this would be a great place for standardization!

Do most whois severs permit automated queries though ? Pretty sure the response to this will be that there is no standard (which seems to be the standard with whois servers ), but just asking for your thoughts based on your past experience ..

We are looking to extract reg date, expiry, contacts, name servers, and anything else we can "automate".

 

[Peter]

whois servers would find it hard to know if a request is an automated 1 or not. Unless of course you hammer the server. To combat this they only allow x amount of requests per hour or so.

 

[STDomains]

Makes sense.. Thx Peter.

 

[Keral_Patel]

I think cURL would do the trick in PHP.

For a nice automation I would put the script as a function on some site of mine where there is little bit of traffic

So whenever someone visits my site the code part executes and it checks the availability.

If it is available then it goes to available.txt file and if not available then it dumps the stuff into the database

After all the data is extracted like this. You just need to find and replace junk stuff within the database like: Name etc.

 

[monaco]

So whenever someone visits my site the code part executes and it checks the availability.

I wonder if you could screw with the x-forwarded-for header to confuse the site and buy you more time before you got banned...maybe they'd think you were the endpoint for an entire org...

 

[Peter]

i doubt it. I should imagine they are actively looking for proxies as well.

 

[Keral_Patel]

I am not doing it that it would get detected. I am more smarter then that. :D

Just trying to share some ideas.