Internic Whois

[Midano]

Hi guys.

While writing a custom whois script, I hit the wall with some strange behavior of Internic's whois.
After successfully retrieving 3 or 4 com/net/org/edu queries (from my site), it throws an error at me:

Rate Error
Sorry, too many requests. Try again later

Same happens at Internic, same happens at GoDaddy (sporadically) when it fetches info from Internic's whois DB.

I don't see a clear pattern of this error, so maybe you guys know better. Does Internic limits a number of queries from particular IP within x seconds? Do they limit a total collective number of requests?

 

[nRnF]

Does Internic limits a number of queries from particular IP within x seconds? Do they limit a total collective number of requests?

Yes, they do limit the number of queries from particular IP within x seconds.
Problem here is, that if you do this several times, and they may block you IP forever.
PS: How many seconds "x" I don't know.

 

[Midano]

Thanks nRnF.
That's what I thought. I'm going to research this further, will post what I find out.

 

[primacomputer]

"TERMS OF USE: You are not authorized to access or query our Whois
database through the use of electronic processes that are high-volume and
automated except as reasonably necessary to register domain names or
modify existing registrations...VeriSign may restrict or terminate your access to the
Whois database for failure to abide by these terms of use"

Not only is raping the internic whois server slow and inefficient but it's also against the rules.

 

[Midano]

Good point, Prima.

Being a programmer, I treat the "high-volume and automated" passage as an "AND" clause, meaning you have to be both high-volume AND automated to violate the rules.
So, high-volume manual and low-volume automated should be fine?

 

[primacomputer]

Technically any software, even a whois client, is a form of automation. In reality they don't care whether it's automated, they care about the volume of requests. High volume manual won't work. They will block you IP all the same.

You really need to work out what it is that you really want to accomplish and see if there isn't some other way to do it, or worst case distribute it.

 

[Midano]

You really need to work out what it is that you really want to accomplish and see if there isn't some other way to do it, or worst case distribute it.

There's not really much to it. I'm not going to harvest info from whois, just want to allow people to do comprehensive whois search without leaving a site. And create a custom script along the way - just because I can.
There's no "other way to do it", the only way to whois is to query registrars' databases.

 

[primacomputer]

Like I said, work out “what you really want to accomplish”. Break your job down into the most fundamental basic elements. Basic elements will rarely have the word “comprehensive” in their description.

So you want to do a “comprehensive whois”. What exactly is that anyway? Seems to me you'd do a whois once and display the results. How much more comprehensive can you get than that? First step will be to properly define your goal before breaking it down.

Let me just take a stab in the dark here. Are you, per chance, wanting to feed a list of names into a script and get a list of which ones are registered and which ones aren't? If so then yes. There is another way to do it.

 

[armstrong]

There should be a way of arranging to be allowed to perform high-volume whois queries. Otherwise sites like whois.sc won't work at all. Have you tried talking to internic, or maybe one of the accredited registrars? If you foresee driving registrations their way, I don't see why they will object.

 

[Midano]

Like I said, work out “what you really want to accomplish”. Break your job down into the most fundamental basic elements. Basic elements will rarely have the word “comprehensive” in their description.

So you want to do a “comprehensive whois”. What exactly is that anyway? Seems to me you'd do a whois once and display the results. How much more comprehensive can you get than that? First step will be to properly define your goal before breaking it down.

Let me just take a stab in the dark here. Are you, per chance, wanting to feed a list of names into a script and get a list of which ones are registered and which ones aren't? If so then yes. There is another way to do it.

"Comprehensive" is what whois.sc have. So I'm duplicating their functionality to some extent (different approach though). There's an easy option - partner with whois.sc and use their XML feed to display on other site, but it comes with a price. Thus I'm doing my own.

 

[primacomputer]

"Comprehensive" is what whois.sc have.

Well there you go, it's all about definitions. Whois.sc “have” a lot of things, but I assume you're referring to the historical whois data. Despite rumors to the contrary, this is not comprehensive. They have data on a fraction of registered domains with nothing before 2001.

You're never going to get all the historical data, but you can start collecting data now. You can start the same way whois.sc did. Stick a whois box on your web site and cache the results. No black magic, just a whois box.

It's actually not a bad idea. You end up checking the relatively few names that people are interested in, not wasting resources on brute forcing everything. Of course there are other ways to narrow your focus. For example I have historical whois data for every MAIL FROM and RCPT TO that has come through any of my mail servers since mid 1997.

And if you must brute force it then there are still other ways to accomplish this. Your main problem will be time. Assuming you can do a query a second it will take you a year to pull down all the whois data for .com alone. Of course by this time it will be out of date.

If you want something that is truly comprehensive you are going to have to build a distributed collaborative system get 365 people involved in the project and pull down the whois data for every .com in one day.