Tool bulk nameserver deleted domains

[Jonh Borin]

hello guys Are there a tool that gives bulk nameserver to previously ended domains

 

[twiki]

hello guys Are there a tool that gives bulk nameserver to previously ended domains

Might be one soon, once Dotible is fully up (see my sig). But it won't have the ns of domains deleted BEFORE its launch.

What do you want to accomplish? Just asking so if I see how I can help out.

 

[twiki]

@Jonh Borin , @okaydomains , additional question.

What is the status of the domain you search? I guess you want a specific status. Expired, pending delete, or fully deleted only?

This raises quite some special requirements so I need to know.

It might require a massive Whois live poll to determine which are still available or not... some of them will be re-taken meantime... etc. That is quite a challenge so I will be looking at ways to optimize the tasks.

Edit: And I can offer live bulk Whois for certain TLDs only, .com and .net so far. Not so much for others. So it all depends on what data to pull exactly. Might have to mix in some other data sources for it.

 

[PoMa]

Besides @twiki 's tool, you can query/parse hosterstats to find already dropped domains that were published in SH/BB, but the amount of queries is limited per IP address.

You can also contact @domainior or follow him on twitter. He/she used to publish expired SH and BB domains here.

 

[jmcc]

Sure, can't wait for RDAP. I'm sick and tired working with Whois already as each TLD is a separate nightmare and they tend to change things, so overnight now your whois isn't working anymore.

Better be careful what you wish for as RDAP is gated and more controlled than WHOIS. There is also talk of people being charged for access and tiered access.

So Whois is still here to stay, with its limitations of course.

It is going away and there is a trend to limit the amount of information available. ICANN has already announced the end of WHOIS. The registrars are transitioning to RDAP and some have already listed their RDAP services.

A SH activated domain would have nameservers activated anyway, which means it's captured via resolution and cached in our archive. And once a domain has expired, it will either point to registrar parking page OR drop the resolution completely. Both will be detected by our bot network and flagged as a change to look at. Edit: A Whois call is needed for detecting neither.

Some registrars have been transferring expired domain names to auction sites rather than moving them to a parking/expired page. (The WHOIS data can also change from the registrant's to that of the registrar.) This happens before the Pending Delete stage. Depending on the registrar, it may appear that the domain name has shifted from SH to an auction site. If a domain name does not sell then it will often go through the typical deletion process.

Whois is just a bonus and actually more needed for other applications. (Edit: but useful to the end user via our live whois calls, to see when domain they want to snap in the end will drop; but they can check that on their registrar's whois page as well.)

It is a valuable source of data.

Things like Cloudflare are a problem, but for this particular case, there is no problem actually.

For some sales/auction sites it will be a problem but maybe not for this particular case.

These domains will have the common identifier = the SH or similar platform's nameservers. Same as everything put on sale via afternic will use nsX.afternic.com or nsY.dan.com or whatnot.

Dan also has verification nameservers that may needed to be added. With Godaddy's takeover of Dan and Undeveloped, there may be some IP changes on the way.

So again you're pointing at issues, but for other things really.

Approximately 9% of .COM is on sale at the moment. It is a lot of domain names and it is best to get the scalability aspect of development built in at the start.

Regards...jmcc

 

[twiki]

I see this happen all the time with Afternic listed domains but rarely SH (although I have caught a few). Never checked for BB domains.

I suspect there's just a lot less SH/BB names in general, as compared to Afternic.

Would definitely like to see this as a tool, @twiki!

Sure thing! Please hit me with a DM in a couple weeks if I don't say anything.

( reason is, I might lose track - too much stuff wobbling around my head, not just this new startup, I'm running 3 businesses at the same time, now I have to move the entire warehouse for physical/brick mortar one. You can imagine I lose track on contacts and messaging due to volume. )

 

[okaydomains]

@Jonh Borin , @okaydomains , additional question.

What is the status of the domain you search? I guess you want a specific status. Expired, pending delete, or fully deleted only?

This raises quite some special requirements so I need to know.

It might require a massive Whois live poll to determine which are still available or not... some of them will be re-taken meantime... etc. That is quite a challenge so I will be looking at ways to optimize the tasks.

Edit: And I can offer live bulk Whois for certain TLDs only, .com and .net so far. Not so much for others. So it all depends on what data to pull exactly. Might have to mix in some other data sources for it.

I'm interested in domains that are in redemption or pending delete. I guess I'm interested in fully deleted but I assume they're crap if still available for hand reg so I don't spend a lot of time on those domains myself.

I'm willing to buy from expired domain auctions.

Its a hard problem because you can't just check the nameservers. They're often replaced with the registrars' nameservers during the redemption period, and even if not plenty of people use SH's nameservers for their private label names (which I do not care about).

I have this worked out for Afternic. Its a decent signal of good domains but its not overwhelmingly good. There's a lot of crap on Afternic, even if its highly priced. I'd be more interested in SH/BB accepted domains because of the human element.

 

[okaydomains]

Might be able to do this only for .coms for now. It's half of the market anyway...

.com is probably > 90% of the domains listed on SquadHelp anyway. I don't have experience with BrandBucket but its probably similar.

.com only seems fine.

 

[twiki]

Okay so here is how it went. Nothing found.

All the domains I have in list pointing at SH nameservers (165k, quite a lot) are still active and most have plenty of validity ahead, even for 2 years. Except the dashed one I posted above.

It is possible the data I have is too fresh so there is no 'former SH' in, expired or expiring.

Will check again in a couple weeks or so and see what's going on.

 

[twiki]

I want to know the domains that were on brandbucket and squadhelp, And her companions didn't renew her, and she fell. Now it is available again.

Good one, thumbs up for the idea.

Yes I could help with that. I'm just working right now on the data mining portion of the site, an initial version will be live soon and nameservers are already in.

Side note I dropped quite a lot of SH .COM's as well. They might still be in the dropping phase or dropped. I'm not even keeping track of them.

Let's keep in touch if you agree and you can check it out once done. Send me a DM in case I forget.

This is a valid use case for my data, although it's a small niche but there are other types of users wanting the NS data for other reasons, so it makes sense. Within a couple weeks or so the tool should be live, and then it will "remember" all this data.

Note, this will probably be not free anywhere so just as a paid option. Although not a high priced one (like those looking for enterprise grade data would be expected to pay) but more of everyday user's one.

For the sole reason it requires quite a bit of storage and processing. From the outside is hard to figure. To go back in history you need tons of snapshots of all domains in the world, well one such snapshot takes a ton of data space and filtering through it also has its incurring costs. Even if someone else pays for the data, there's still running cost to be covered. This is why certain tools like DotDB are not free and there's not many of them available.

Just saying where I expect it goes. Some limited searches will be free on the site though.

Edit: Check this tool: https://dotible.com/domain-tools/bulk-registered-tld-checker . Might give you a hint of how big the data is and power behind it. The tool gives you all registered TLD's for a list of 100 domains for free instantly, but it can also do that for 1M domains if a client needs that.

In a similar fashion will work your needed NS search (but better, this is just an unfinished /unpolished initial version out there).

 

[okaydomains]

I want to know the domains that were on brandbucket and squadhelp, And her companions didn't renew her, and she fell. Now it is available again.

I see this happen all the time with Afternic listed domains but rarely SH (although I have caught a few). Never checked for BB domains.

I suspect there's just a lot less SH/BB names in general, as compared to Afternic.

Would definitely like to see this as a tool, @twiki!

 

[twiki]

I'm interested in domains that are in redemption or pending delete. I guess I'm interested in fully deleted but I assume they're crap if still available for hand reg so I don't spend a lot of time on those domains myself.

I'm willing to buy from expired domain auctions.

Its a hard problem because you can't just check the nameservers. They're often replaced with the registrars' nameservers during the redemption period, and even if not plenty of people use SH's nameservers for their private label names (which I do not care about).

I have this worked out for Afternic. Its a decent signal of good domains but its not overwhelmingly good. There's a lot of crap on Afternic, even if its highly priced. I'd be more interested in SH/BB accepted domains because of the human element.

Yes I pretty much figured that. Fully deleted would probably not help.

Hmm, have to think better over technicals.

What about TLD's? I can primarily do this for .com and .net at this stage. Not so much for other tld's.

 

[okaydomains]

What about TLD's? I can primarily do this for .com and .net at this stage. Not so much for other tld's.

In addition to .com, I am also interested in app, club, dev, finance, io, link, network, pro, shop, tech, tv, xyz, and zone.

I don't care about .net.

Edit: SH allows a list of TLDs thats bigger than what's listed in their submission guidelines, but its still < 20 TLDs. That would be a good starting place.

 

[twiki]

Might be able to do this only for .coms for now. It's half of the market anyway...

 

[twiki]

See about the stale aspect here:

https://whois.whoisxmlapi.com/database/specifications/datafeed-files

Quarterly.

And incremental, which means "fresh bits here and there"

This is not bad work on their part. It is how things are.

Edit: Funny, you can't even download a sample without signing up and then their reps pounding you with some offer.

 

[twiki]

Actually I think the tool might be doable, and soon-ish.

I've just pulled SH domains. only 1000 names are non-coms... So there is no point really in actually working on the other tld's.

Running other tests now.

 

[twiki]

I found one available domain so far, in case one of you wants it:

sun-nergy.com

Important though there is no guarantee a domain has been approved or just pointed via nameservers but never made it into SH.

 

[jmcc]

The point is, I just realized, this whole thing will have to be a byproduct of another product - bulk Whois data. You cannot know what a domain status is without Whois bigdata. If this one ever gets a chance.

That is a BIG business.

Here's an example: https://www.whoisxmlapi.com/whois-database-download.php

They used to have some prices (in many hundreds or thousands and up). Now, it's just request a quote. No wonder.

So I need to build this first before doing the other (byproduct). Side note I know this well, been working on it for a while (few years) as well as other tech.

But once that is built, unsure if I (or anyone else) might be interested in doing that domain research tool anymore. It's simply... pennies vs the real money and our lifetime is limited.

Anyway, just thoughts.

A bit late to the thread but The whole bulk WHOIS model is in trouble and has been since the introduction of GDRP in May 2018. This means that a lot of gTLD registrars have now started to redact data for registrants in the EU and elsewhere. ICANN's approach to the matter has been less than optimal and it has caused serious damage to the security of the Web.

There's also an "aging data" problem with scraped WHOIS data. The .COM gets over 20 million new registrations a year and that's an awful lot of new domain names with partial data. Domain names drop and are renewed. That's a worse problem because the number of points that can be compared is reduced to country of registrant, nameservers, and registrar. Silent transfers (where a domain name is sold and transferred) within large registrars and countries can be almost invisible unless the data from web surveys is correlated.

As a category, For Sale domain names (based on Web Usage surveys) tend to have a high non-renewal rate outside the premiums.

Regards...jmcc

 

[twiki]

A bit late to the thread but The whole bulk WHOIS model is in trouble and has been since the introduction of GDRP in May 2018. This means that a lot of gTLD registrars have now started to redact data for registrants in the EU and elsewhere. ICANN's approach to the matter has been less than optimal and it has caused serious damage to the security of the Web.

There's also an "aging data" problem with scraped WHOIS data. The .COM gets over 20 million new registrations a year and that's an awful lot of new domain names with partial data. Domain names drop and are renewed. That's a worse problem because the number of points that can be compared is reduced to country of registrant, nameservers, and registrar. Silent transfers (where a domain name is sold and transferred) within large registrars and countries can be almost invisible unless the data from web surveys is correlated.

As a category, For Sale domain names (based on Web Usage surveys) tend to have a high non-renewal rate outside the premiums.

Regards...jmcc

Well you took that a bit too literally, and I haven't been precise. I meant domain data, including Whois data, not only Whois data.

You are right about GDPR though, and most whois is years' stale anyway not because of GDPR, but because of due to rate limits it takes years for a large Whois provider to rescan the entire web whois - just to make an update. Also, nameservers are not PII so they don't fall under GDPR.

But here is how we do it. We aggregate not just Whois (which BTW is just starting right now) but also DNS records, website data and other sources. Let me give an example via resolution records.

Go to Dotible and use this pre-filled search (for a random SH domain) : https://dotible.com/bigdata-tools/find-domain-information?domain=whizza.com:

For now there is limited information as this is just a few day's old but anyway. Click on "Find all domains on this NS" next to ns1.squadhelp.com and you get this:

The free version is limited to 100 max domains and up to 10k count depth, but a pro user will have access to any amount based on their quota. Also access to search with many subfilters to narrow to a given set of domains as needed via their user panel.

This is based on name resolution only, whois isn't even in yet. (comes shortly) and we plan to have it fast refreshed, significantly above market average. There will be a large archive containing snapshots over time, so for example in October-November you could look for SH domains that haven't been renewed and see where they ended up, via our snapshot archive, a domain status search (expiring, expired or renewed+ auctioned or whatever) , condition = previously on SH nameservers etc.

Whois offers more indeed as it can also show delegated but not activated ns records, or domains in expired status. But we are able to do that too rather fast but only on given TLD's (com included, which is half of the world's stock anyway; Edit: also many others too, except particular TLDs don't offer a public whois interface or the rate limits are extreme). And for this particular application, results are not hindered by GDRP.

Within a month' time there will be a wealth of data and possibilities to use our platform for this and many other niches who need freshly squeezed domain/web data.

Edit: This is all public information btw.

Edit2: ( Beta status: Please note that this isn't a fully usable version yet, we're just like 1%...2% complete with what needs to be activated on site. So please don't throw bricks yet. However it gives a small glimpse of what comes next.

We probably have around 40% of the data needed but once our network will be fully active and all bells and whistles as well, it will be really useful for such research. Much more comprehensive and kept refreshed + historical data. )

 

[jmcc]

Well you took that a bit too literally, and I haven't been precise. I meant domain data, including Whois data, not only Whois data.

It is based on years of dealing with ICANN and with WHOIS data. The WHOIS service for gTLDs is being replaced by RDAP. ( https://www.icann.org/rdap )

You are right about GDPR though, and most whois is years' stale anyway not because of GDPR, but because of due to rate limits it takes years for a large Whois provider to rescan the entire web whois - just to make an update. Also, nameservers are not PII so they don't fall under GDPR.

The WHOIS data for a significant percentage of .COM domains has changed since 2018. I think that some of the WHOIS providers just do checks on new and tracked domain names now and use the last modified date for full updates. All that will change when RDAP is fully implemented. Some of the new gTLDs are already using it.

The free version is limited to 100 max domains and up to 10k count depth, but a pro user will have access to any amount based on their quota. Also access to search with many subfilters to narrow to a given set of domains as needed via their user panel.

If a user requests data on a loadbalancer IP then the number of records returned could be in the millions. It might be a good thing to allow users to limit results with a keyword to stop them going over their quota.

Regards...jmcc

 

[twiki]

@jmcc

Some valid points here.

Note: Just realized you're behind HosterStats.... I have limited time so I was skimming through it. My apologies.

To be honest I've heard about your site multiple times but never used. What is your monetization model, may I ask?

RDAP: I know there will be gated and tiered paid access to data. The future is going there anyway; do you need to access your car's seat heating? But that is already factored in. For the moment we have what is needed. For the future we will obviously pay for data. In fact we are paying for data even before starting.

The scalability aspect:

Ehe... you've just hit the nail in the head. That's the key here.

I'd say that not every startup out there is going to get that key. By looking at your site, can tell you know well about this.

Fortunately, this is right down my alley.
Started in 1997 or so with my first search engine... at the end of an 33Kbps leased line (copper wire) internet connection which was costing me an arm an leg back then, it was that expensive.

Building things that scale is my specialty. And it does require a lot of experience and a lot of work. Especially with optimizing large databases and stuff. But it goes far beyond that. Syncing a swarm of bots and making all working smooth without anything crashing each 10 minutes isn't an easy job.

TBH I have been considering some of the worst applications, where the above is just... piece of cake.

What I always wanted to crack is the area which Ahrefs and Majestic are in, and nowadays Semrush has cracked. But that ain't for the faint hearted. You need not only some serious power, petabytes of disk space and network, and a LOT of funding. Great engineers, large team etc. So this is still nice to look at but probably out of my league.

Downsizing a bit, I've been running my network(s) over a few years doing lots of different things and preparing for it. Still doing that. It is a challenge. But it is already up to that. 80% of the tech needed is already done more or less; with codebase in. What is not yet done is putting together everything, and aspects like sales, case use, niches etc. Plus the user interface / layer. But all these are the fun part, right?

I have zero doubts in the project at this time.

In fact I have been running a couple startups over the past doing various things, for example I had an url shortener that did a few things, such as an A/B rotator test, tracking, blocking some unwanted traffic and providing some stats. It was profitable from the first month; but it was scaling slow so I decided to cut it loose after a bit. There's experience. That's what matters.

Anyway, once you have a vision things start unfolding. And it's an amazing experience that opens so many avenues.

I've already opened to the world a few things which are unique, from my backyard. Such as my formerly private appraisal tool. There are lots of other things which are NOT live yet but there are gaps in various industries where they fit nicely. I prepare a lot, and launch only a very little = when I am 100% sure of stuff. Note, I already have paid user requests for it (got my first 2 days after launch?) but I'm postponing them for now. Got very good feedback + a bit of acid ones which only stirred me to get even better.

There's more to come and fortunately it's based on work previously done over a couple decades, just adapted to today's needs and user pains.

Edit: I am the unconventional ( tech ) guy.

Where everyone tells me it's not possible, I wanna try for myself.

Where everyone tells me you have to do that, I want to do the other - the one they tell me don't do that, ever. Finding ways to do that, always.

Same is with domaining.

I've carved myself a niche in repurposing dropped names. If you look at the sales reports, there's no direct domainer competitor that does it. A few do but... not on the same scale. I feel kinda lonely in my niche.

In tech, it pays off to have unconventional, lateral thinking. I often have to duck cause bricks are coming at ya, but hey, it's part of the fun. Being able to finish things, provide stuff, at least differently if not unique.

 

[twiki]

@jmcc

My favorite personal saying about the above is:

Say if I'm on a beach and everyone is yelling and running to the left, I would go to the right to see what they are running from, and how I can tackle that thing my way. Perhaps even by going into water, who knows.

I always trust my instinct. And while I listen (I do listen to your advice above) I still don't trust so much what others have to say to me, especially about limits and limitations. Those are there just to be broken - somehow.

Most of my businesses have always been deemed impossible and are still deemed impossible, including one I've built 15 years ago and is still running and I'm rebuilding right now (totally unrelated). Fighting directly with some major local corporations and again with no direct competitor in the niche. 15 years later, yes - still no direct competitor whatsoever.

It helps to try to be different. It doesn't get you well understood though.

Some folks are really willing to not understand anything and say you're a fraud and throw bricks at ya. I just do my work and that's my proof, nothing else. Whatever is running you, mate. Peace.

Liked your feedback, thanks.

 

[jmcc]

@jmcc

Some valid points here.

Note: Just realized you're behind HosterStats.... I have limited time so I was skimming through it. My apologies.

To be honest I've heard about your site multiple times but never used. What is your monetization model, may I ask?

Research, reports and some site monetisation. I had been planning to add the registrar and hoster statistics side back to the site but have been a bit busy.

RDAP: I know there will be gated and tiered paid access to data. The future is going there anyway; do you need to access your car's seat heating? But that is already factored in. For the moment we have what is needed. For the future we will obviously pay for data. In fact we are paying for data even before starting.

It is some of the proposals that are worrying. Some in the ICANN Intellectual Property Constituency (lawyers/brand protection etc) want those accessing the data to be pre-approved and effectively be lawyers investigating intellectual property rights issues. Others want limitations on the data provided. The is some discussion about a field to differentiate natural persons (people) from legal persons (companies) as GDPR does not generally apply to companies.

The scalability aspect:

Ehe... you've just hit the nail in the head. That's the key here.

I'd say that not every startup out there is going to get that key. By looking at your site, can tell you know well about this.

It is a nightmare to get right. Most sites never do the necessary design work at the start and end up trying to retrofit scalability.

Fortunately, this is right down my alley.
Started in 1997 or so with my first search engine... at the end of an 33Kbps leased line (copper wire) internet connection which was costing me an arm an leg back then, it was that expensive.

Four wire analogue modems used to be popular for those lines. The problem then was trying to monetise the content.

Building things that scale is my specialty. And it does require a lot of experience and a lot of work. Especially with optimizing large databases and stuff. But it goes far beyond that. Syncing a swarm of bots and making all working smooth without anything crashing each 10 minutes isn't an easy job.

It sounds like a set of sites rather than one single site.

TBH I have been considering some of the worst applications, where the above is just... piece of cake.

What I always wanted to crack is the area which Ahrefs and Majestic are in, and nowadays Semrush has cracked. But that ain't for the faint hearted. You need not only some serious power, petabytes of disk space and network, and a LOT of funding. Great engineers, large team etc. So this is still nice to look at but probably out of my league.

The guy who started Majestic used to post back on Webmasterworld back in the day. Markus Frind, the guy who started Plenty Of Fish, also posted there and his description of how he set up the site is definitely worth reading even now. He took a completely different approach to design than the large dating sites at the time and beat them. The guy who built ZFbot used to post here on Namepros. Many of the largest websites start with one developer with an idea rather than well resourced teams.

Downsizing a bit, I've been running my network(s) over a few years doing lots of different things and preparing for it. Still doing that. It is a challenge. But it is already up to that. 80% of the tech needed is already done more or less; with codebase in. What is not yet done is putting together everything, and aspects like sales, case use, niches etc. Plus the user interface / layer. But all these are the fun part, right?

The simultaneous users number is always a concern. The key to this is a bit counter-intuitive but it has to do with limiting the user's options.My HTML has been know to make grown web developers cry.

I have zero doubts in the project at this time.

In fact I have been running a couple startups over the past doing various things, for example I had an url shortener that did a few things, such as an A/B rotator test, tracking, blocking some unwanted traffic and providing some stats. It was profitable from the first month; but it was scaling slow so I decided to cut it loose after a bit. There's experience. That's what matters.

There's only one way to get experience and that's by doing it.

There is suppose to be another round of new gTLDs in the next few years and some potential gTLDs are being mentioned already. (coin, wallet etc) and these will grow the numbers of domain names. If they use the heavy discounting model that some of the new gTLDs like .XYZ and others used then the number of deleted domain names will start building up after the second year.

During the Domain Tasting problem in the mid-2000s some of the domain tracking sites crashed because of the millions of domain names being registered and deleted in the five day Add Grace Period. ICANN changed the regulations on this and most of the taster registrars stopped when a restocking fee was added. A few of the main tasting registrars were sued. The only sites that survived were the ones who had built-in scalability. To put it in perspective, approximately 1,000,000,000 .COM domain names were registered and deleted over a few years.

Regards...jmcc

 

[jmcc]

This is the Webmasteworld on running a one person high traffic site:
https://www.webmasterworld.com/website_technology/4496.htm

It is from 2006 and is effectively a Q&A with Markus Frind. What makes this impressive is that he developed and coded it himself.

Regards...jmcc

 

[Jonh Borin]

Might be one soon, once Dotible is fully up (see my sig). But it won't have the ns of domains deleted BEFORE its launch.

What do you want to accomplish? Just asking so if I see how I can help out.

I want to know the domains that were on brandbucket and squadhelp, And her companions didn't renew her, and she fell. Now it is available again.

 

[Jonh Borin]

@Jonh Borin , @okaydomains , additional question.

What is the status of the domain you search? I guess you want a specific status. Expired, pending delete, or fully deleted only?

This raises quite some special requirements so I need to know.

It might require a massive Whois live poll to determine which are still available or not... some of them will be re-taken meantime... etc. That is quite a challenge so I will be looking at ways to optimize the tasks.

Edit: And I can offer live bulk Whois for certain TLDs only, .com and .net so far. Not so much for others. So it all depends on what data to pull exactly. Might have to mix in some other data sources for it.

deleted only