A few questions about zone files and Whois info

[dsiomtw]

Say I wanted to track certain whois info for a large number of domains (say 100,000) by analyzing certain info in the whois record every few months ...

I know I can get ahold of zone files in one way or another, but zone files don't contain any registration related data - they only contain the domain and DNS info right?

So basically I would need to get ahold of the zone files, make a list of the names I wanted to track, and then go through the entire list and do a whois lookup on each domain to get the info I want?

I don't imagine you can buy whois info, or can you?

If not I assume I need a good tool that uses proxy servers etc. in order to do bulk whois lookups - is there a general consensus as to what is the best product on the market for doing this? Obviously proxy support is key.

I could develop a script to do this I guess, but if there is already a solid product on the market that I could just buy I'd rather go that route and save the time.

And no, I'm not trying to do anything devious. It's just a little expirement, for research purposes ...

 

[primacomputer]

I know I can get ahold of zone files in one way or another, but zone files don't contain any registration related data - they only contain the domain and DNS info right?

Correct

So basically I would need to get ahold of the zone files, make a list of the names I wanted to track, and then go through the entire list and do a whois lookup on each domain to get the info I want?

I don't imagine you can buy whois info, or can you?

You can buy some whois information but you would have to do a deal with every registrar in the world for this to work. Not really an option so you need to query whois servers.

If not I assume I need a good tool that uses proxy servers etc. in order to do bulk whois lookups - is there a general consensus as to what is the best product on the market for doing this? Obviously proxy support is key.

I could develop a script to do this I guess, but if there is already a solid product on the market that I could just buy I'd rather go that route and save the time.

And no, I'm not trying to do anything devious. It's just a little expirement, for research purposes ...

It all depends on what you want to do. Is there a tool that does the analysis you want or do you need something that will do the whois and store it in a format you can use?

Proxy support is not necessarily the key. Personally I use a distributed system instead. It's much faster than proxies. When querying 100k names the difference between several queries a second and one query every few seconds is significant.

Odds are you'll have to roll you own. All my whois tools are home made and although they are very powerful they probably don't total more than 1,000 lines of code.

 

[dsiomtw]

I assume that the good tools out there will break up the whois info and save the fields separately into a database, etc. so I can just access what I need later. Or maybe not?

I only need to do the lookups say once every few months. So relatively speaking it's not TONS like I assume some people are doing LOL. If I only did 1 query every 3 seconds it's no big thing - that would only take a little over 3 days or so and that would be fine. But I just assumed that if I hit any whois service, even once every 3 seconds, for 3 days straight I will get blocked.

When you say distributed do you mean that you are rotating your queries through a bunch of different whois lookup services, etc. or do you mean you're running your scripts from a bunch of different machines?

Any details you can provide would be much appreciated. If you prefer, please send me a PM instead. I'll owe you big time!

Thanks!

 

[xtremest]

Just do whois requests to whois servers which are normally on 43 port. It you'll make lookup on .com zone then each request will be forwarded from internic.net (who does not care about your volume of requests as it serves majority of requests) to whois server of registrar.
So, if you'll check the list there is a big (over 80%) possibility that your names will be checked on different servers, so you'll not be banned.

But if you'll check some cctld - yes, you should do the pause. Use sleep() if you will use PHP.

 

[marisolblue]

If I recall a company called exody use to sell these lists.

 

[RJ]

So, if you'll check the list there is a big (over 80%) possibility that your names will be checked on different servers, so you'll not be banned.

Right, good post. I have been on the receiving end of temporary IP bans from a couple of the registrars (particularly NetSol) when doing bulk WHOIS lookups in my research, however I'm not sure how many lookups it takes to earn that.

 

[primacomputer]

I assume that the good tools out there will break up the whois info and save the fields separately into a database, etc. so I can just access what I need later. Or maybe not?

I'm not sure about this. The problem is that there are hundreds of registrars and each has their own format for returning whois information. This can change from time to time, so you either have to have an intelligent algorithm, or you stand the chance of missing something.

I only need to do the lookups say once every few months. So relatively speaking it's not TONS like I assume some people are doing LOL. If I only did 1 query every 3 seconds it's no big thing - that would only take a little over 3 days or so and that would be fine. But I just assumed that if I hit any whois service, even once every 3 seconds, for 3 days straight I will get blocked.

Absolutely correct. I've had a unique opportunity to view the source code of a number of whois scripts used by various registrars. Some of them keep track of your IP for as long as a month. Some also have pattern monitoring which can really screw up in an low volume script.

When you say distributed do you mean that you are rotating your queries through a bunch of different whois lookup services, etc. or do you mean you're running your scripts from a bunch of different machines?

I run through a bunch of different machines, which run through many different whois servers (there is no central whois server anyway). These will fall back to a number of different proxies if necessary, but this has only happened a few times. I haven't found anything I want to do with the system that I can't. I have in the past, but it's been easy enough to add what I needed.

Any details you can provide would be much appreciated. If you prefer, please send me a PM instead. I'll owe you big time!

I use jwhois as the client and htdig for the full text index. Perhaps if you can describe what you want to do in more detail I can give you some pointers on what tools might be appropriate for you.

Just do whois requests to whois servers which are normally on 43 port. It you'll make lookup on .com zone then each request will be forwarded from internic.net (who does not care about your volume of requests as it serves majority of requests) to whois server of registrar.

This isn't quite true. They do care, and they will block your IP. But you need to do something on the order of 1,000 queries in a short period of time (24 hours?) This becomes a problem when the names you are whoising are .coms and .nets because you need to query the NSI whois server to get the registrar information to know where to look it up.
Also, I'd say that at least half to whois servers aren't available via rwhois but only http.

 

[dsiomtw]

So it sounds like you are hitting web-based whois lookup services, and scraping the results? If not, what ar eyou using htdig for??

 

[primacomputer]

I use htdig to build a full text index of my text files, it never touches the network. I use web based whois only out of necessity. Many registrars simply don't have rwhois servers. Where TW hois is available I use it.