advice What's the best way to bulk-check domain age?

[umpire]

Say you have 100k domains you want to check when they were first registered, according to the current whois data (or potentially other methods). What's the best way to do this?

I have considered using Ubuntu and running "whois example.com" repeatedly - apparently parsing the data will be difficult and I will be rate limited.

Is there a better way than using an API like WhoisXML, which is about $0.004 per domain?

 

[SixFigures.ltd]

Do you need the exactly time and date of each one? At Above.com after you've added your domains you can check their 'rough' age in years and then check the exact time by clicking the one you need

Spoiler: screen

 

[Hypersot]

@SixFigures.ltd ,
the OP probably wants to check domains in bulk for research. ie. domains not necessarily currently registered.

Also, I'm not sure how easy it is for Above's panel to handle 100k domains,.. at least not easy for the user.

 

[umpire]

@SixFigures.ltd ,
the OP probably wants to check domains in bulk for research. ie. domains not necessarily currently registered.

Also, I'm not sure how easy it is for Above's panel to handle 100k domains,.. at least not easy for the user.

I am more looking for a programmatic/API way of doing this, using a GUI isn't really the best.

These are nearly all going to be registered domains. I am looking for the age in years, it doesn't need to be exact, I just want to know how long it's been around for.

Note I'm not necessarily looking for the create date. I would be open to a solution like scraping the Wayback machine for example, but it's not very accurate and would be resource-intensive.

 

[johnn]

100k names?
When I am done checking for 100k names I will be in a nursing home.

 

[Hypersot]

@umpire,
the only tool I can think of is DomainTools but, afaik, it's on the expensive side.

Also, afaik, a Whois check won't do the job as when a domain gets deleted or re-registered the previous creation date is gone. So, the results will vary wildly.
Just imagine a domain that it has been around for many years and gets domain tasted after it expires, you'll only see that last 4/5-day long registration.

A possible solution could be found with http://www.hosterstats.com . No API that I know of however. Maybe if you contact them you'll sort something out for yourself.

Wayback, imo, is a good solution. Regarding it being 'resource intensive: with such a high volume of calls, anything you try will be resource intensive the way I see it.

 

[IrfanHabib]

https://www.namelooka.com/bulk-whois
this might help you. bulk whois with domain age. download the list.

 

[dukar]

If you need to check a large number of domains' registration dates, using a Whois API like WhoisXML could be a more efficient and cost-effective option. Running whois commands manually on Ubuntu might be slow,

Say you have 100k domains you want to check when they were first registered, according to the current whois data (or potentially other methods). What's the best way to do this?

I have considered using Ubuntu and running "whois example.com" repeatedly - apparently parsing the data will be difficult and I will be rate limited.

Is there a better way than using an API like WhoisXML, which is about $0.004 per domain?

cumbersome, and subject to rate limitations. The API can handle bulk queries and provide structured data, saving you time and effort.

 

[YammerSMTH]

If you want to check the age of lots of domains at once, the best way is to use a domain age checker tool. You can find plenty of these tools online, both free and paid, like BulkSEOTools or SmallSEOTools. All you need to do is type in the list of domains you want to check, and the tool will tell you how old they are.

 

[Timofey]

Use Foxwhois works with most of extensions

 

[umpire]

Use Foxwhois works with most of extensions

I had a look at your website and have a few questions:

1. Do you have documentation to get this up and running on a Ubuntu server without a GUI?
2. How many domains can it analyse per minute?
3. To confirm this doesn't work with .co.uk, .com.au? Referring to the premium version.

 

[Timofey]

I had a look at your website and have a few questions:

1. Do you have documentation to get this up and running on a Ubuntu server without a GUI?
2. How many domains can it analyse per minute?
3. To confirm this doesn't work with .co.uk, .com.au? Referring to the premium version.

1. No way now
2. Depends on many factors like as a domain extension, number of socks proxies etc.
3. Foxwhois works with .co.uk

 

[evil]

Register means that had at least two nameservers.
You can use https://completedns.com/dns-history/ if you are interested in the history. However is not cheap and tld limited.
Otherwise https://member.expireddomains.net/tools/bulk-domain-search/ sorting by ABY (the first archive.org date)

 

[umpire]

1. No way now
2. Depends on many factors like as a domain extension, number of socks proxies etc.
3. Foxwhois works with .co.uk

The tool looks like it could be helpful, but we would need to have it run on a server unfortunately - the number of domains we're analysing isn't really appropriate to run on a local computer.