discuss Crawling Wikipedia for Expired Domains

[Tarabasa Domains Ltd]

Okay so I recently came across a website online called 'Expired Domain Tracker'

The way it works is you enter a keyword/niche, and then they email you a whole entire list of highly ranked expired domains all crawled from Wikipedia links relevant to your keyword

Here's a screenshot of what I mean:
imgur . com/BNNFQRl . png (Just copy + paste in browser without spaces since I cant post links in here)

This is a genius idea! I never would have thought to use Wikipedia, Wikipedia is a HUGE website full of millions of links! It works as a great platform for crawling for expired domains

The only thing is that this is a paid service,
and I wanted to know if anyone knows a way to crawl Wikipedia the same way but for free

My idea is to use Xenu, Iv'e tested it and it works well
I'd type into Google something like '"Wikipedia Links" + "Niche"', then download all the pages with the Moz toolbar and put them all into Xenu, and Iv'e found some decent results so far

I hope this helps any other people looking for ways to find expired domains, all you have to do is download Xenu and the Moz toolbar, they're both free just Google how to use them and hopefully this should help you find domains

Also if anybody else knows any search terms to put into Google for crawling Wikipedia please post them below
All I could come up with is:
"Wikipedia Links" + "Niche"
"Wikipedia" + "Niche"
Anything else would be appreciated

Hope this helps people
All feedback + comments is appreciated

Sincerely by your boyyyyyy Vladddddddddd

 

[Tarabasa Domains Ltd]

 

[Reality]

Keep in mind that links from Wikipedia are all "no-follow", so they will not help your page ranking. You might get some traffic from the link, but probably only people looking for citations/references. It won't be high volume, and probably not very good "commercial" traffic. Also, Google does not look favorably on redirected domain traffic.

 

[kostaki]

You don't need to crawl it! You can download everything you need.
https://en.wikipedia.org/wiki/Wikipedia:Database_download

 

[cdboard]

how about


"niche" site:wikipedia.org

Also if anybody else knows any search terms to put into Google for crawling Wikipedia please post them below
All I could come up with is:
"Wikipedia Links" + "Niche"
"Wikipedia" + "Niche"
Anything else would be appreciated

 

[Lakshan]

You don't need to crawl it! You can download everything you need.
https://en.wikipedia.org/wiki/Wikipedia:Database_download

Hi, Can you explain how you get the database only related to reference links. I couldn't figure it out.

Thanks

 

[Tarabasa Domains Ltd]

Hi, Can you explain how you get the database only related to reference links. I couldn't figure it out.

Thanks

Try this link

http://www.kiwix.org/downloads/

Kiwix lets you download all sorts of Wiki data which you can use even when you're without internet
It gives access to Wiki Pages, Public Domain Library's, Dictionaries, Guides etc,

Hope that helps

 

[Tarabasa Domains Ltd]

9

Try this link

http://www.kiwix.org/downloads/

Kiwix lets you download all sorts of Wiki data which you can use even when you're without internet
It gives access to Wiki Pages, Public Domain Library's, Dictionaries, Guides etc,

Hope that helps

Then from there you should be able to search for related reference links

 

[MrConor]

Hi,

This also works for different categories on Wikipedia if you plugin the URL in to the Expired Domains Module for Scrapebox, using multi threads it will go from 1 - 25 levels deep and list all expired domains

Not only Wikipedia but you can do this method on any High Authority sites. It is also a bonus if the High Authority site is a DoFollow as in this case many of the expired domains that will be scraped will have a backlink from the high authority site and still be passing link juice to the domain.

 

[kostaki]

You can now also find Wikipedia linked domains on ExpiredDomains.net. I've added a new field that represents the number of links a domain has from *.wikipedia.org and filters for that.
All domains (expired/deleted/marketplaces...) are checked when they are added to ExpiredDomains.net and everything will be update once per month. If there is interest, I can also provide the same for other wikimedia websites like wiktionary, wikibooks, wikiquote... Let me know.

 

[Lakshan]

You can now also find Wikipedia linked domains on ExpiredDomains.net. I've added a new field that represents the number of links a domain has from *.wikipedia.org and filters for that.
All domains (expired/deleted/marketplaces...) are checked when they are added to ExpiredDomains.net and everything will be update once per month. If there is interest, I can also provide the same for other wikimedia websites like wiktionary, wikibooks, wikiquote... Let me know.

Awsome feature @kostaki
Keepup the good work...

 

[rafaelo0]

@kostaki great info. wikipedia database is a great resource but how can you know witch domains are expired? you have to check manually if they are available right??

I love expireddomains.net its another great resource and the wikipedia option to.

@MrConor scrapbox looks great its a shame that you have to pay, but it looks like it worth the money because the resources there looks great. Unfortunately i dont have the budget at this moment.

Does any one know a website or software like scrapbox nut only free that has the option to crawl authority sites like CNN.com for expired domains linked to them and check if they are available.

Would be a great resource and the help would be appreciated.

Best Regards
Mcloud

 

[vivaldi]

@kostaki great info. wikipedia database is a great resource but how can you know witch domains are expired? you have to check manually if they are available right??

You gotta be joking?

 

[rafaelo0]

@vivaldi im not joking. Sorry if i said something ignorant or sound like one, but im new in this crawl and wikepedia expired domains area. That's why im asking to get more information and educate myself in this area.

Best Regards
Mcloud

 

[FPForum]

This is nothing new. You can use Scrapebox and mine tons of high authority sites like CNN, Foxnews, HuffingtonPost and other sites to scrape for available domains. The method is pretty saturated now and most of what you'll find is pretty spammy. With all the demand for these types of names not many go unregistered these days.

 

[vivaldi]

@vivaldi im not joking. Sorry if i said something ignorant or sound like one, but im new in this crawl and wikepedia expired domains area. That's why im asking to get more information and educate myself in this area.

Best Regards
Mcloud

Cool, it would take him forever to check all that manually.

 

[kostaki]

@kostaki great info. wikipedia database is a great resource but how can you know witch domains are expired? you have to check manually if they are available right??

I love expireddomains.net its another great resource and the wikipedia option to.

I already have the information what domain is expired. I only had to add a new check if these domains are linked on wikipedia and that is not that hard, because you can download wikipedia. No crawling required. With that I can now easily make a lookup if a certain domain is linked anywhere on wikipedia.

 

[buywebsites.biz]

I used to use Screaming Frog to do this, but as already mentioned, these type of domains get picked over pretty quickly. Ahrefs is also good for finding these expired domains (among other things - it's become my swiss army knife), but it's a $99+ monthly service.