Question on Whois info extraction

STDomains · 03-09-2008, 02:28 PM

Hi All,

Anyone have any ideas on programmable whois info extraction given a specific domain name?

What we are trying to accomplish, is to be able to extract whois info for a number of domains that we own and get it into a database.

We found a couple of good threads at NP (http://www.namepros.com/code/280961-...ity-check.html and one or 2 other threads linked from that one).. but not sure if the threads are doing exactly what we need.

Is there a "format" that all automated whois servers are supposed to conform to on their responses?

Any thoughts/direction would be much appreciated!

Thanks,

ST Domains

Peter · 03-09-2008, 04:37 PM

No there is no set format returned from whois. Different registries return data slightly different.

What precise information are you after?

There should be code within the code section that does what you want as I know several people have written classes and functions for whois.

monaco · 03-09-2008, 08:32 PM

Certain items are pretty constant, like the Administrative Contact and Technical / Tech contact fields. Maybe some simplistic regex parsing could get the most vital information out of the whois data.

STDomains · 03-12-2008, 06:45 AM

Thanks for your responses Peter and Monaco.

Its unfortunate that there is no standard. Seems that this would be a great place for standardization!

Do most whois severs permit automated queries though ? Pretty sure the response to this will be that there is no standard (which seems to be the standard with whois servers

), but just asking for your thoughts based on your past experience ..

We are looking to extract reg date, expiry, contacts, name servers, and anything else we can "automate".

Peter · 03-12-2008, 01:14 PM

whois servers would find it hard to know if a request is an automated 1 or not. Unless of course you hammer the server. To combat this they only allow x amount of requests per hour or so.

STDomains · 03-15-2008, 08:44 PM

Makes sense.. Thx Peter.

**-Nick-** · 03-15-2008, 08:48 PM

I think cURL would do the trick in PHP.

For a nice automation I would put the script as a function on some site of mine where there is little bit of traffic

So whenever someone visits my site the code part executes and it checks the availability.

If it is available then it goes to available.txt file and if not available then it dumps the stuff into the database

After all the data is extracted like this. You just need to find and replace junk stuff within the database like: Name etc.

monaco · 03-16-2008, 01:16 AM

Quote:

Originally Posted by -Nick-

So whenever someone visits my site the code part executes and it checks the availability.

I wonder if you could screw with the x-forwarded-for header to confuse the site and buy you more time before you got banned...maybe they'd think you were the endpoint for an entire org...

Peter · 03-16-2008, 05:08 AM

i doubt it. I should imagine they are actively looking for proxies as well.

**-Nick-** · 03-16-2008, 06:02 AM

I am not doing it that it would get detected.

I am more smarter then that.

Just trying to share some ideas.

03-09-2008, 02:28 PM	#1 (permalink)
STDomains NamePros Member Join Date: Mar 2007 Posts: 130 3,268.00 NP$ (Donate)	Question on Whois info extraction Hi All, Anyone have any ideas on programmable whois info extraction given a specific domain name? What we are trying to accomplish, is to be able to extract whois info for a number of domains that we own and get it into a database. We found a couple of good threads at NP (http://www.namepros.com/code/280961-...ity-check.html and one or 2 other threads linked from that one).. but not sure if the threads are doing exactly what we need. Is there a "format" that all automated whois servers are supposed to conform to on their responses? Any thoughts/direction would be much appreciated! Thanks, ST Domains __________________ Domains for Sale @ Sedo

03-09-2008, 04:37 PM	#2 (permalink)
Peter Senior Member Join Date: Nov 2003 Location: Scotland Posts: 4,900 0.60 NP$ (Donate)	No there is no set format returned from whois. Different registries return data slightly different. What precise information are you after? There should be code within the code section that does what you want as I know several people have written classes and functions for whois. __________________ Manage your portfolio using my new Domain Portfolio Management script. Securing Your Domain Name From Theft

03-09-2008, 08:32 PM	#3 (permalink)
monaco NamePros Regular Join Date: Jul 2005 Location: Tucson, AZ Posts: 695 314.80 NP$ (Donate)	Certain items are pretty constant, like the Administrative Contact and Technical / Tech contact fields. Maybe some simplistic regex parsing could get the most vital information out of the whois data. __________________ My Website \| My Blog

03-12-2008, 06:45 AM	#4 (permalink)
STDomains NamePros Member Join Date: Mar 2007 Posts: 130 3,268.00 NP$ (Donate)	Thanks for your responses Peter and Monaco. Its unfortunate that there is no standard. Seems that this would be a great place for standardization! Do most whois severs permit automated queries though ? Pretty sure the response to this will be that there is no standard (which seems to be the standard with whois servers ), but just asking for your thoughts based on your past experience .. We are looking to extract reg date, expiry, contacts, name servers, and anything else we can "automate". __________________ Domains for Sale @ Sedo

03-12-2008, 01:14 PM	#5 (permalink)
Peter Senior Member Join Date: Nov 2003 Location: Scotland Posts: 4,900 0.60 NP$ (Donate)	whois servers would find it hard to know if a request is an automated 1 or not. Unless of course you hammer the server. To combat this they only allow x amount of requests per hour or so. __________________ Manage your portfolio using my new Domain Portfolio Management script. Securing Your Domain Name From Theft

Technical Services

Technical Services

03-15-2008, 08:44 PM	#6 (permalink)
STDomains NamePros Member Join Date: Mar 2007 Posts: 130 3,268.00 NP$ (Donate)	Makes sense.. Thx Peter. __________________ Domains for Sale @ Sedo

03-15-2008, 08:48 PM	#7 (permalink)
-Nick- I'll do it Technical Services Join Date: Dec 2005 Location: India Posts: 6,434 5,169.80 NP$ (Donate)	I think cURL would do the trick in PHP. For a nice automation I would put the script as a function on some site of mine where there is little bit of traffic So whenever someone visits my site the code part executes and it checks the availability. If it is available then it goes to available.txt file and if not available then it dumps the stuff into the database After all the data is extracted like this. You just need to find and replace junk stuff within the database like: Name etc. __________________ Vhuv\|Delq\|Fhur\|Twut\|Tluz\|Kegh\|Vhir\|Juhy\|Ruuz\|Jyos\|Jupt\|Vhek Webmaster Blog \| Software Downloads Sponsor Ads at SponAds.com

03-16-2008, 05:08 AM	#9 (permalink)
Peter Senior Member Join Date: Nov 2003 Location: Scotland Posts: 4,900 0.60 NP$ (Donate)	i doubt it. I should imagine they are actively looking for proxies as well. __________________ Manage your portfolio using my new Domain Portfolio Management script. Securing Your Domain Name From Theft

03-16-2008, 06:02 AM	#10 (permalink)
-Nick- I'll do it Technical Services Join Date: Dec 2005 Location: India Posts: 6,434 5,169.80 NP$ (Donate)	I am not doing it that it would get detected. I am more smarter then that. Just trying to share some ideas. __________________ Vhuv\|Delq\|Fhur\|Twut\|Tluz\|Kegh\|Vhir\|Juhy\|Ruuz\|Jyos\|Jupt\|Vhek Webmaster Blog \| Software Downloads Sponsor Ads at SponAds.com

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)