Get your catchy domain at it.com

Tool bulk nameserver deleted domains

Dynadot Dynadot

Jonh Borin

Established Member
Impact
0
hello guys Are there a tool that gives bulk nameserver to previously ended domains
 
-1 0
•••
The views expressed on this page by users and staff are their own, not those of NamePros.

twiki

Top Contributor
Impact
22,442
hello guys Are there a tool that gives bulk nameserver to previously ended domains
Might be one soon, once Dotible is fully up (see my sig). But it won't have the ns of domains deleted BEFORE its launch.

What do you want to accomplish? Just asking so if I see how I can help out.
 

Jonh Borin

Established Member
Impact
0
Might be one soon, once Dotible is fully up (see my sig). But it won't have the ns of domains deleted BEFORE its launch.

What do you want to accomplish? Just asking so if I see how I can help out.
I want to know the domains that were on brandbucket and squadhelp, And her companions didn't renew her, and she fell. Now it is available again.
 

twiki

Top Contributor
Impact
22,442
I want to know the domains that were on brandbucket and squadhelp, And her companions didn't renew her, and she fell. Now it is available again.
Good one, thumbs up for the idea.

Yes I could help with that. I'm just working right now on the data mining portion of the site, an initial version will be live soon and nameservers are already in.

Side note I dropped quite a lot of SH .COM's as well. They might still be in the dropping phase or dropped. I'm not even keeping track of them.

Let's keep in touch if you agree and you can check it out once done. Send me a DM in case I forget.

This is a valid use case for my data, although it's a small niche but there are other types of users wanting the NS data for other reasons, so it makes sense. Within a couple weeks or so the tool should be live, and then it will "remember" all this data.

Note, this will probably be not free anywhere so just as a paid option. Although not a high priced one (like those looking for enterprise grade data would be expected to pay) but more of everyday user's one.

For the sole reason it requires quite a bit of storage and processing. From the outside is hard to figure. To go back in history you need tons of snapshots of all domains in the world, well one such snapshot takes a ton of data space and filtering through it also has its incurring costs. Even if someone else pays for the data, there's still running cost to be covered. This is why certain tools like DotDB are not free and there's not many of them available.

Just saying where I expect it goes. Some limited searches will be free on the site though.

Edit: Check this tool: https://dotible.com/domain-tools/bulk-registered-tld-checker . Might give you a hint of how big the data is and power behind it. The tool gives you all registered TLD's for a list of 100 domains for free instantly, but it can also do that for 1M domains if a client needs that.

In a similar fashion will work your needed NS search (but better, this is just an unfinished /unpolished initial version out there).
 
Last edited:

okaydomains

Upgraded Member
Impact
963
I want to know the domains that were on brandbucket and squadhelp, And her companions didn't renew her, and she fell. Now it is available again.

I see this happen all the time with Afternic listed domains but rarely SH (although I have caught a few). Never checked for BB domains.

I suspect there's just a lot less SH/BB names in general, as compared to Afternic.

Would definitely like to see this as a tool, @twiki!
 

twiki

Top Contributor
Impact
22,442
I see this happen all the time with Afternic listed domains but rarely SH (although I have caught a few). Never checked for BB domains.

I suspect there's just a lot less SH/BB names in general, as compared to Afternic.

Would definitely like to see this as a tool, @twiki!
Sure thing! Please hit me with a DM in a couple weeks if I don't say anything.

( reason is, I might lose track - too much stuff wobbling around my head, not just this new startup, I'm running 3 businesses at the same time, now I have to move the entire warehouse for physical/brick mortar one. You can imagine I lose track on contacts and messaging due to volume. )
 

twiki

Top Contributor
Impact
22,442
@Jonh Borin , @okaydomains , additional question.

What is the status of the domain you search? I guess you want a specific status. Expired, pending delete, or fully deleted only?

This raises quite some special requirements so I need to know.

It might require a massive Whois live poll to determine which are still available or not... some of them will be re-taken meantime... etc. That is quite a challenge so I will be looking at ways to optimize the tasks.

Edit: And I can offer live bulk Whois for certain TLDs only, .com and .net so far. Not so much for others. So it all depends on what data to pull exactly. Might have to mix in some other data sources for it.
 
Last edited:

PoMa

Established Member
Impact
256
Besides @twiki 's tool, you can query/parse hosterstats to find already dropped domains that were published in SH/BB, but the amount of queries is limited per IP address.

You can also contact @domainior or follow him on twitter. He/she used to publish expired SH and BB domains here.
 

okaydomains

Upgraded Member
Impact
963
@Jonh Borin , @okaydomains , additional question.

What is the status of the domain you search? I guess you want a specific status. Expired, pending delete, or fully deleted only?

This raises quite some special requirements so I need to know.

It might require a massive Whois live poll to determine which are still available or not... some of them will be re-taken meantime... etc. That is quite a challenge so I will be looking at ways to optimize the tasks.

Edit: And I can offer live bulk Whois for certain TLDs only, .com and .net so far. Not so much for others. So it all depends on what data to pull exactly. Might have to mix in some other data sources for it.

I'm interested in domains that are in redemption or pending delete. I guess I'm interested in fully deleted but I assume they're crap if still available for hand reg so I don't spend a lot of time on those domains myself.

I'm willing to buy from expired domain auctions.

Its a hard problem because you can't just check the nameservers. They're often replaced with the registrars' nameservers during the redemption period, and even if not plenty of people use SH's nameservers for their private label names (which I do not care about).

I have this worked out for Afternic. Its a decent signal of good domains but its not overwhelmingly good. There's a lot of crap on Afternic, even if its highly priced. I'd be more interested in SH/BB accepted domains because of the human element.
 
Last edited:

twiki

Top Contributor
Impact
22,442
I'm interested in domains that are in redemption or pending delete. I guess I'm interested in fully deleted but I assume they're crap if still available for hand reg so I don't spend a lot of time on those domains myself.

I'm willing to buy from expired domain auctions.

Its a hard problem because you can't just check the nameservers. They're often replaced with the registrars' nameservers during the redemption period, and even if not plenty of people use SH's nameservers for their private label names (which I do not care about).

I have this worked out for Afternic. Its a decent signal of good domains but its not overwhelmingly good. There's a lot of crap on Afternic, even if its highly priced. I'd be more interested in SH/BB accepted domains because of the human element.

Yes I pretty much figured that. Fully deleted would probably not help.

Hmm, have to think better over technicals.

What about TLD's? I can primarily do this for .com and .net at this stage. Not so much for other tld's.
 

okaydomains

Upgraded Member
Impact
963
What about TLD's? I can primarily do this for .com and .net at this stage. Not so much for other tld's.

In addition to .com, I am also interested in app, club, dev, finance, io, link, network, pro, shop, tech, tv, xyz, and zone.

I don't care about .net.

Edit: SH allows a list of TLDs thats bigger than what's listed in their submission guidelines, but its still < 20 TLDs. That would be a good starting place.
 
Last edited:

Jonh Borin

Established Member
Impact
0
@Jonh Borin , @okaydomains , additional question.

What is the status of the domain you search? I guess you want a specific status. Expired, pending delete, or fully deleted only?

This raises quite some special requirements so I need to know.

It might require a massive Whois live poll to determine which are still available or not... some of them will be re-taken meantime... etc. That is quite a challenge so I will be looking at ways to optimize the tasks.

Edit: And I can offer live bulk Whois for certain TLDs only, .com and .net so far. Not so much for others. So it all depends on what data to pull exactly. Might have to mix in some other data sources for it.
deleted only
 

twiki

Top Contributor
Impact
22,442
Might be able to do this only for .coms for now. It's half of the market anyway...
 

okaydomains

Upgraded Member
Impact
963
Might be able to do this only for .coms for now. It's half of the market anyway...

.com is probably > 90% of the domains listed on SquadHelp anyway. I don't have experience with BrandBucket but its probably similar.

.com only seems fine.
 

twiki

Top Contributor
Impact
22,442
The point is, I just realized, this whole thing will have to be a byproduct of another product - bulk Whois data. You cannot know what a domain status is without Whois bigdata. If this one ever gets a chance.

That is a BIG business.

Here's an example: https://www.whoisxmlapi.com/whois-database-download.php

They used to have some prices (in many hundreds or thousands and up). Now, it's just request a quote. No wonder.

So I need to build this first before doing the other (byproduct). Side note I know this well, been working on it for a while (few years) as well as other tech.

But once that is built, unsure if I (or anyone else) might be interested in doing that domain research tool anymore. It's simply... pennies vs the real money and our lifetime is limited.

Anyway, just thoughts.
 
Last edited:

twiki

Top Contributor
Impact
22,442
See about the stale aspect here:

https://whois.whoisxmlapi.com/database/specifications/datafeed-files

Quarterly.

And incremental, which means "fresh bits here and there"

This is not bad work on their part. It is how things are.

Edit: Funny, you can't even download a sample without signing up and then their reps pounding you with some offer.
 
Last edited:

twiki

Top Contributor
Impact
22,442
Actually I think the tool might be doable, and soon-ish.

I've just pulled SH domains. only 1000 names are non-coms... So there is no point really in actually working on the other tld's.

Running other tests now.
 

twiki

Top Contributor
Impact
22,442
I found one available domain so far, in case one of you wants it:

sun-nergy.com

Important though there is no guarantee a domain has been approved or just pointed via nameservers but never made it into SH.
 

okaydomains

Upgraded Member
Impact
963
Important though there is no guarantee a domain has been approved or just pointed via nameservers but never made it into SH.

This is what makes it a hard problem. I suspect most you'll find will just be pointed at the SH nameservers, but not approved.
 

twiki

Top Contributor
Impact
22,442
Okay so here is how it went. Nothing found.

All the domains I have in list pointing at SH nameservers (165k, quite a lot) are still active and most have plenty of validity ahead, even for 2 years. Except the dashed one I posted above.

It is possible the data I have is too fresh so there is no 'former SH' in, expired or expiring.

Will check again in a couple weeks or so and see what's going on.
 
Impact
2,615
The point is, I just realized, this whole thing will have to be a byproduct of another product - bulk Whois data. You cannot know what a domain status is without Whois bigdata. If this one ever gets a chance.

That is a BIG business.

Here's an example: https://www.whoisxmlapi.com/whois-database-download.php

They used to have some prices (in many hundreds or thousands and up). Now, it's just request a quote. No wonder.

So I need to build this first before doing the other (byproduct). Side note I know this well, been working on it for a while (few years) as well as other tech.

But once that is built, unsure if I (or anyone else) might be interested in doing that domain research tool anymore. It's simply... pennies vs the real money and our lifetime is limited.

Anyway, just thoughts.
A bit late to the thread but The whole bulk WHOIS model is in trouble and has been since the introduction of GDRP in May 2018. This means that a lot of gTLD registrars have now started to redact data for registrants in the EU and elsewhere. ICANN's approach to the matter has been less than optimal and it has caused serious damage to the security of the Web.

There's also an "aging data" problem with scraped WHOIS data. The .COM gets over 20 million new registrations a year and that's an awful lot of new domain names with partial data. Domain names drop and are renewed. That's a worse problem because the number of points that can be compared is reduced to country of registrant, nameservers, and registrar. Silent transfers (where a domain name is sold and transferred) within large registrars and countries can be almost invisible unless the data from web surveys is correlated.

As a category, For Sale domain names (based on Web Usage surveys) tend to have a high non-renewal rate outside the premiums.

Regards...jmcc
 

twiki

Top Contributor
Impact
22,442
A bit late to the thread but The whole bulk WHOIS model is in trouble and has been since the introduction of GDRP in May 2018. This means that a lot of gTLD registrars have now started to redact data for registrants in the EU and elsewhere. ICANN's approach to the matter has been less than optimal and it has caused serious damage to the security of the Web.

There's also an "aging data" problem with scraped WHOIS data. The .COM gets over 20 million new registrations a year and that's an awful lot of new domain names with partial data. Domain names drop and are renewed. That's a worse problem because the number of points that can be compared is reduced to country of registrant, nameservers, and registrar. Silent transfers (where a domain name is sold and transferred) within large registrars and countries can be almost invisible unless the data from web surveys is correlated.

As a category, For Sale domain names (based on Web Usage surveys) tend to have a high non-renewal rate outside the premiums.

Regards...jmcc
Well you took that a bit too literally, and I haven't been precise. I meant domain data, including Whois data, not only Whois data.

You are right about GDPR though, and most whois is years' stale anyway not because of GDPR, but because of due to rate limits it takes years for a large Whois provider to rescan the entire web whois - just to make an update. Also, nameservers are not PII so they don't fall under GDPR.

But here is how we do it. We aggregate not just Whois (which BTW is just starting right now) but also DNS records, website data and other sources. Let me give an example via resolution records.

Go to Dotible and use this pre-filled search (for a random SH domain) : https://dotible.com/bigdata-tools/find-domain-information?domain=whizza.com:

Untitled.png


For now there is limited information as this is just a few day's old but anyway. Click on "Find all domains on this NS" next to ns1.squadhelp.com and you get this:

Untitled.png


The free version is limited to 100 max domains and up to 10k count depth, but a pro user will have access to any amount based on their quota. Also access to search with many subfilters to narrow to a given set of domains as needed via their user panel.

This is based on name resolution only, whois isn't even in yet. (comes shortly) and we plan to have it fast refreshed, significantly above market average. There will be a large archive containing snapshots over time, so for example in October-November you could look for SH domains that haven't been renewed and see where they ended up, via our snapshot archive, a domain status search (expiring, expired or renewed+ auctioned or whatever) , condition = previously on SH nameservers etc.

Whois offers more indeed as it can also show delegated but not activated ns records, or domains in expired status. But we are able to do that too rather fast but only on given TLD's (com included, which is half of the world's stock anyway; Edit: also many others too, except particular TLDs don't offer a public whois interface or the rate limits are extreme). And for this particular application, results are not hindered by GDRP.

Within a month' time there will be a wealth of data and possibilities to use our platform for this and many other niches who need freshly squeezed domain/web data.

Edit: This is all public information btw.

Edit2: ( Beta status: Please note that this isn't a fully usable version yet, we're just like 1%...2% complete with what needs to be activated on site. So please don't throw bricks yet. However it gives a small glimpse of what comes next.

We probably have around 40% of the data needed but once our network will be fully active and all bells and whistles as well, it will be really useful for such research. Much more comprehensive and kept refreshed + historical data. )
 
Last edited:
Impact
2,615
Well you took that a bit too literally, and I haven't been precise. I meant domain data, including Whois data, not only Whois data.
It is based on years of dealing with ICANN and with WHOIS data. The WHOIS service for gTLDs is being replaced by RDAP. ( https://www.icann.org/rdap )

You are right about GDPR though, and most whois is years' stale anyway not because of GDPR, but because of due to rate limits it takes years for a large Whois provider to rescan the entire web whois - just to make an update. Also, nameservers are not PII so they don't fall under GDPR.
The WHOIS data for a significant percentage of .COM domains has changed since 2018. I think that some of the WHOIS providers just do checks on new and tracked domain names now and use the last modified date for full updates. All that will change when RDAP is fully implemented. Some of the new gTLDs are already using it.

The free version is limited to 100 max domains and up to 10k count depth, but a pro user will have access to any amount based on their quota. Also access to search with many subfilters to narrow to a given set of domains as needed via their user panel.
If a user requests data on a loadbalancer IP then the number of records returned could be in the millions. It might be a good thing to allow users to limit results with a keyword to stop them going over their quota.

Regards...jmcc
 

twiki

Top Contributor
Impact
22,442
Sure, can't wait for RDAP. I'm sick and tired working with Whois already as each TLD is a separate nightmare and they tend to change things, so overnight now your whois isn't working anymore.

Howeber, as ICANN says:

"In the short term, RDAP will not replace web-based or port 43 WHOIS. Based on current policies and agreements, contracted parties will be required to implement RDAP in addition to (port 43) WHOIS and web-based WHOIS."

So Whois is still here to stay, with its limitations of course.

But again, you don't necessarily need Whois in this case, perhaps just on a tiny subset (your end records).

A SH activated domain would have nameservers activated anyway, which means it's captured via resolution and cached in our archive. And once a domain has expired, it will either point to registrar parking page OR drop the resolution completely. Both will be detected by our bot network and flagged as a change to look at. Edit: A Whois call is needed for detecting neither.

Whois is just a bonus and actually more needed for other applications. (Edit: but useful to the end user via our live whois calls, to see when domain they want to snap in the end will drop; but they can check that on their registrar's whois page as well.)

Furthermore there will be check of status for live domains, via the web server for example.

Things like Cloudflare are a problem, but for this particular case, there is no problem actually.

These domains will have the common identifier = the SH or similar platform's nameservers. Same as everything put on sale via afternic will use nsX.afternic.com or nsY.dan.com or whatnot.

So again you're pointing at issues, but for other things really.
 
Last edited:
Impact
2,615
Sure, can't wait for RDAP. I'm sick and tired working with Whois already as each TLD is a separate nightmare and they tend to change things, so overnight now your whois isn't working anymore.
Better be careful what you wish for as RDAP is gated and more controlled than WHOIS. :) There is also talk of people being charged for access and tiered access.

So Whois is still here to stay, with its limitations of course.
It is going away and there is a trend to limit the amount of information available. ICANN has already announced the end of WHOIS. The registrars are transitioning to RDAP and some have already listed their RDAP services.

A SH activated domain would have nameservers activated anyway, which means it's captured via resolution and cached in our archive. And once a domain has expired, it will either point to registrar parking page OR drop the resolution completely. Both will be detected by our bot network and flagged as a change to look at. Edit: A Whois call is needed for detecting neither.
Some registrars have been transferring expired domain names to auction sites rather than moving them to a parking/expired page. (The WHOIS data can also change from the registrant's to that of the registrar.) This happens before the Pending Delete stage. Depending on the registrar, it may appear that the domain name has shifted from SH to an auction site. If a domain name does not sell then it will often go through the typical deletion process.

Whois is just a bonus and actually more needed for other applications. (Edit: but useful to the end user via our live whois calls, to see when domain they want to snap in the end will drop; but they can check that on their registrar's whois page as well.)
It is a valuable source of data.

Things like Cloudflare are a problem, but for this particular case, there is no problem actually.
For some sales/auction sites it will be a problem but maybe not for this particular case.

These domains will have the common identifier = the SH or similar platform's nameservers. Same as everything put on sale via afternic will use nsX.afternic.com or nsY.dan.com or whatnot.
Dan also has verification nameservers that may needed to be added. With Godaddy's takeover of Dan and Undeveloped, there may be some IP changes on the way.

So again you're pointing at issues, but for other things really.
Approximately 9% of .COM is on sale at the moment. It is a lot of domain names and it is best to get the scalability aspect of development built in at the start.

Regards...jmcc