NameSilo

[FREE] Useful Python Script to filter expiring domain lists by customizable criteria

Labeled as FREE in More Free Stuff, started by CrocBuzz, Jul 3, 2018

Replies:
17
Views:
4,466

  1. Crock

    Crock Software Engineer Gold Account

    Posts:
    179
    Likes Received:
    126
    I wrote a little Python script this week to aid myself in filtering daily expiring domain lists by criteria I specify. I have it set to run automatically using the built-in Automator application on macOS, but it can easily be run with a UNIX cronjob or Windows Task Scheduler as well.

    You can review the full source code at github.com/crock/dropcatch-filter
    If you are knowledgeable in Python, contributions are welcome. Just make a pull request anytime and I'll review it as soon as possible.

    Comes with a simple JSON configuration file that looks like this:
    Code:
    {
      "maxDomainLength": 10,
      "keywords": [
        "minecraft",
        "coin",
        "craft",
        "tech",
        "pvp"
      ],
      "tlds": [
        "com",
        "net",
        "org",
        "io",
        "co"
      ]
    }
    The script programmatically downloads the next day's expiring list from Namejet's website and parses it. It then filters the entire list in mere seconds based on the criteria you specify in the config.json file.

    If running on a timer of some sort, please set the script to run AFTER 7:00 AM PST because the next day's list is not available until that time the day before.

    Installation Instructions

    1. Download the Python 3.6.x+ installer from python.org/downloads
    2. Run the installer, but please make sure to tick the box labeled "Add Python to environment variables". It is unchecked by default. This is important!
    3. Open Terminal (Mac/Linux) or Command Prompt (Windows) and type the following command to let the terminal know where the script files are located. Replace [file path] with the full path to the directory containing the files: cd [file path]
    4. Next, run the following command: pip install -r requirements.txt
    5. The above steps only need to be run once. Now every time you want to run this utility, you simply run the following command: python main.py (if that doesn't work, try replacing python in that command with python3)
    6. If executed correctly, you will see domains being output according to your criteria and they will also be saved to a file in the results directory with the name results_MM-DD-YYYY.txt



     
    The views expressed on this page by users and staff are their own, not those of NamePros.
  2. creataweb

    creataweb Some Guy with Awesome Senior High School Photo VIP ★★★★★★★★★★

    Posts:
    6,338
    Likes Received:
    7,689
    Cool I’ll check this out today.
     
  3. Crock

    Crock Software Engineer Gold Account

    Posts:
    179
    Likes Received:
    126
    Thanks to a generous soul from another forum for making a pull request. With the latest change, Python 3.6.x is now the base required version going forward.

    If you need a backported version (3.x.x only) for some reason, please send me a message.
     
  4. Crock

    Crock Software Engineer Gold Account

    Posts:
    179
    Likes Received:
    126
    Just pushed a critical fix for a bug on Windows systems that prevented the script from running properly. Go download the latest zip from Github when you get the chance.

    Thanks to a Discord user for reporting this bug!
     
  5. Walter Eason

    Walter Eason Established Member

    Posts:
    148
    Likes Received:
    27
    Nice.......i'll check it out
     
  6. anantj

    anantj Top Contributor VIP

    Posts:
    4,173
    Likes Received:
    4,000
    Can you also add configs to search for alphabetical domains only (i.e. exclude hyphens and alphanumeric domains)?

    Thanks
     
  7. anantj

    anantj Top Contributor VIP

    Posts:
    4,173
    Likes Received:
    4,000
    For now, I've edited the Regex in Filters.py

    Thanks
     
  8. Crock

    Crock Software Engineer Gold Account

    Posts:
    179
    Likes Received:
    126
    I built a new config generator for those non-developers that want to use my tool. Also, I have ported this application to an exclusive Discord bot command for my domain name community.
    https:// rule_violation [dot] com /dropfilter/config/
     
  9. Mukesh01

    Mukesh01 Established Member

    Posts:
    73
    Likes Received:
    28
    thanks, will check it out soon
     
  10. Primary Names

    Primary Names ****** xXx ****** VIP

    Posts:
    2,539
    Likes Received:
    3,925
    @CrocBuzz Getting this error - Couldn't get file.
     
  11. Crock

    Crock Software Engineer Gold Account

    Posts:
    179
    Likes Received:
    126
    I would need to see the full error from the console. Reply or DM me a screenshot.
     
  12. Primary Names

    Primary Names ****** xXx ****** VIP

    Posts:
    2,539
    Likes Received:
    3,925
    Here is the entire error log from console.
     
  13. Primary Names

    Primary Names ****** xXx ****** VIP

    Posts:
    2,539
    Likes Received:
    3,925
    Trying this on a remote shared server on linux
    python 2.7.5
    pip 20.0.2
     
  14. Crock

    Crock Software Engineer Gold Account

    Posts:
    179
    Likes Received:
    126
    Ah I see, is your timezone somewhere around Australia because that's a day ahead and Namejet's list is only available at 7am PST. I've fixed this issue using the exclusive Discord bot version of my program, but you would have to join my community in order to use it. Here's an invite https://domain-api.now.sh/api/discord/invite
     
  15. Primary Names

    Primary Names ****** xXx ****** VIP

    Posts:
    2,539
    Likes Received:
    3,925
    Unfortunately my host admin says that they dont "officially" support python apps. So they wont be upgrading to Python 3.x.x. Before I give up, I want to give this one last try, if it can it work in Python 2.7 with some code modifications or installing dependencies etc.

    Do we need to convert the american timezone to our local.
    Code:
    local = arrow.now("America/New_York")
    tomorrow = local.shift(days=1).format("M-DD-YYYY")
    I can get this done in javascript or php but Python is said to be second just to Perl when it comes to text parsing. Wish I knew Perl. So it is a performance thing. Beause I do not know yet how big the data will be in the future, so I wanna start on the right foot.
     
    Last edited: Feb 12, 2020
  16. Crock

    Crock Software Engineer Gold Account

    Posts:
    179
    Likes Received:
    126
    The Python cli is behind on updates. That's why it's free and open-source. The bot command version is a Node.js/Javascript port with more functionality, but I haven't open-sourced it yet. You can interact with the bot command on my Discord server.
     
  17. Primary Names

    Primary Names ****** xXx ****** VIP

    Posts:
    2,539
    Likes Received:
    3,925
    @Crock
    Currently trying this on a remote shared server on linux
    python 3.6.0
    pip 21.1.1

    I am getting following errors when I run the script with python3 main.py

    [[email protected] pyapp]$ python3 main.py
    Downloading tomorrow's list of expiring domains...
    Traceback (most recent call last):
    File "main.py", line 72, in <module>
    main()
    File "main.py", line 64, in main
    download_file(url, filename)
    File "main.py", line 28, in download_file
    file_size = int(response.headers['Content-Length'])
    File "/home/torje/pybase/lib/python3.6/site-packages/requests/structures.py", line 52, in __getitem__
    return self._store[key.lower()][1]
    KeyError: 'content-length'
     
  18. Crock

    Crock Software Engineer Gold Account

    Posts:
    179
    Likes Received:
    126
    I don’t really support this script anymore. Use the web version instead as it is better and has more functionality at https://dropfilter.app
     
    Last edited: May 16, 2021

Want to reply or ask your own question?

It only takes a minute to sign up – and it's free!
biix
  1. NamePros uses cookies and similar technologies. By using this site, you are agreeing to our privacy policy, terms, and use of cookies.
    Dismiss Notice
Loading...