Drop domain name list

[MisterD]

I am wondering where does the domains name drop list comes from?
I mean is this a public database that everybody could gather the data from?

 

[stscac]

I am wondering where does the domains name drop list comes from?
I mean is this a public database that everybody could gather the data from?

Droplists can be done in two ways.

1. Compile the list based on domain name searches on a name-by-name basis
2. Download and Compile the Zone files that registrars make available and then determine via whois data which names are expired, ok, active, pending renewal, pending delete, etc. This is what most of the expired name services do.

Was that the answer you were looking for?

-Steve

 

[MisterD]

Droplists can be done in two ways.

1. 
   Was that the answer you were looking for?
   -Steve

1. 
   Hi Steve and thanks for your reply, yes that's what I wanted to know, So
   1. Compile the list based on domain name searches on a name-by-name basis : What do you mean by name-by-name basis?
   
   2. Download and Compile the Zone files that registrars make available : Each registrar has a zone files that they make available? And where can i get those zone files? ( I guess by asking to the registrar ? ) If you have some links to point me out that would be great.
   
   and then determine via whois data which names are expired, ok, active, pending renewal, pending delete, etc. This is what most of the expired name services do.
   What is the best whois services that offers developpers to use they're database?
   
   I guess I'll be doing some googling on any API offer for developpers. Would love to get some new ideas on what I might be able to do on this area.

 

[stscac]

Droplists can be done in two ways.

1. 
   Was that the answer you were looking for?
   -Steve

1. 
   Hi Steve and thanks for your reply, yes that's what I wanted to know, So
   1. Compile the list based on domain name searches on a name-by-name basis : What do you mean by name-by-name basis?

1. 
   Some people compile their own lists based on interests, certain keywords they are in contact with on a daily basis, or current events seen on the news. Searching on a name-by-name basis means it is done by hand and not an automated system.
   

   2. Download and Compile the Zone files that registrars make available : Each registrar has a zone files that they make available? And where can i get those zone files? ( I guess by asking to the registrar ? ) If you have some links to point me out that would be great.

   Here is an example of the .mobi zone file agreement. It is something that is fairly regulated, it will run you several gigabytes per day (depending on what extensions you want to download, and how many)
   
   http://mtld.mobi/domain/zonefile
   
   

   and then determine via whois data which names are expired, ok, active, pending renewal, pending delete, etc. This is what most of the expired name services do.
   What is the best whois services that offers developpers to use they're database?

   DomainTools does a good job at figuring out whois information and displaying it in a readable and consistent format across domain extensions.
   
   If you're just looking to see the status of a domain name, you should just query to see if the domain name is available or not, and the status, not a request for the whois information.
   
   -Steve

 

[MisterD]

Thanks for those usefull informations. So as I understand it, there is a always an official global registry for a specific extension, right? Can there be more than one official global registry for the same extension? ( Or maybe am just really out track here.. starting to have alot in the head and things are mixing up!! )

So I would need to download the agreement file from those registry , send it back to them and will receive my logins informations in order to gain access to the zone files. That would be the first step.

Next step is confusing me a bit.. like I know domaintools and it's pretty much complete, but where do themselves takes they're data of? This is what I would be after

am I slightly on the right path ?

 

[stscac]

There is only one global registry for each extension.

I am sure they get their information from a variety of sources, but I would imagine that they download all of the zone files each day an analyze them.

Here is a linkto the .com and .net zone request form:
http://www.verisign.com/information-services/naming-services/com-net-registry/page_001052.html

-Steve

 

[MisterD]

Thanks Steve.
Currently reading about the zone files. I guess I'll have for a while before being confortable with a "zone file".. So for the moment you've gived me good information actually. Will come back on this thread later on when I'll have moved forward ( or dropped some plans!! ) with that.

 

[stscac]

Thanks Steve.
Currently reading about the zone files. I guess I'll have for a while before being confortable with a "zone file".. So for the moment you've gived me good information actually. Will come back on this thread later on when I'll have moved forward ( or dropped some plans!! ) with that.

Just let me know if you need anything else.

Best of luck....

-Steve

 

[Kate]

MisterD: you can find lists for download at pool.com and snapnames.com.
If you are comfortable with databases and a scripting language like PHP it should be easy for you to extract the names of interest to you, based on certain keywords/extensions for example.
I think it would be a good start to use these lists.

The drop lists from pool or snap can be confusing at first glance because they are different. Certains registrars have agreements with drop catchers (snapnames in particular) so they will send their expired names for auctions. These names will not drop but be assigned to someone else instead. These are the so-called pre-release names.
The drop lists may contain both dropping (deleting) domain names, and expired (pre-release) domains.
One thing you could do for good measure is, download the lists from pool, snapnames, namejet and merge them.


You can also compile your own lists using the zone files. The theory is simple: you compare the current release with the previous one and make a 'diff' to find the changes (new registrations/deletions). In practice it's more complicated than that.
These files are huge (several Gb for .com, unzipped) and may contain tens of millions of domains.
Also the zone files do not contain all domains: the domains with no name servers are missing from the zone files etc.

 

[stscac]

Also the zone files do not contain all domains: the domains with no name servers are missing from the zone files etc.

Excellent point. For the community sake, where can those be found?

-Steve

 

[MisterD]

Wow, am not sure if it's my enthousiasmis or something else but I have the feelling that this thread contains quality information on it.
@sdsinc , thanks for those infos.

MisterD: you can find lists for download at pool.com and snapnames.com.
If you are comfortable with databases and a scripting language like PHP it should be easy for you to extract the names of interest to you, based on certain keywords/extensions for example.
I think it would be a good start to use these lists.

Not a bad idea at all. Would give me the opportunity to get confortable with the format of the lists.

The drop lists from pool or snap can be confusing at first glance because they are different. Certains registrars have agreements with drop catchers (snapnames in particular) so they will send their expired names for auctions. These names will not drop but be assigned to someone else instead. These are the so-called pre-release names.
The drop lists may contain both dropping (deleting) domain names, and expired (pre-release) domains.
One thing you could do for good measure is, download the lists from pool, snapnames, namejet and merge them.

merging is a nice option

You can also compile your own lists using the zone files. The theory is simple: you compare the current release with the previous one and make a 'diff' to find the changes (new registrations/deletions). In practice it's more complicated than that.

Well if am applying the theory into a concrete parser that I could develop and will then do it for me the complicated part will be in the building process only. Then would come the fun of it!

These files are huge (several Gb for .com, unzipped) and may contain tens of millions of domains.

Speaking of making a diff, I wonder why are'nt they offering a simple dayly diff from they're yesterday file and so on? I mean instead of me or us developping tools in order to perform the diff on those list dayly, why is this services not already available? They could for instance supply the whole list on monday and the rest of the week supply a diff file of today's file versus the previous ones and it would be up to you or us to apply the diff to our local files. [/QUOTE]

Also the zone files do not contain all domains: the domains with no name servers are missing from the zone files etc.

I did'nt knew that a domain name could be without a nameserver, how can this be possible according that each domain name must be registered somewhere and that this somewhere as it's own nameservers applied to the domain if it's only parked

Thanks again for those inputs

 

[Michael]

Wow, great thread, thanks for all the info Steve. +repped you

 

[stscac]

Wow, great thread, thanks for all the info Steve. +repped you

Don't forget Kate as well.

-Steve

 

[BinderGang]

Wow, great thread, thanks for all the info Steve. +repped you

Great thread, indeed.

 

[MisterD]

@sdsinc, you say:

If you are comfortable with databases and a scripting language like PHP it should be easy for you to extract the names of interest to you, based on certain keywords/extensions for example.

Is there a desktop tool ( or online tools ) that does that sort of data manipulation based on a large domain name list files?

In other words, what are the software or website you guy's are using if you have a big list like that and want to filter it and output a result and so on.

 

[Kate]

Is there a desktop tool ( or online tools ) that does that sort of data manipulation based on a large domain name list files?

Some people use DRT, I don't know if it has the features you would be looking for.
I use my own scripts but if you don't have the programming skills one thing you could do is to grab the available lists which are plain text files. Then you could import them into an Access DB (choose open as CSV). It should then be pretty easy to run queries based on keywords of interest.
PS: add an index on the domain name field or you queries will be slow.
PPS: snapnames has a search feature on their site which is pretty decent IMO.

 

[MisterD]

I use my own scripts but if you don't have the programming skills

Well am a developper since 12+ years.. and in fact my intention here is to see what can be done.. off course nobody could re-invent the wheel but maybe there is something that can be done and would be usefull?