Cleaning an email list

[TalkDevelopment]

I have an email list that goes back to 2005. Does anyone know of an opensource software or another free way to clean this email list from fake email addresses, etc?

Most of the only services are so expensive.

Thanks

 

[FPForum]

So, you know most of the emails might not be valid..but heck with doing that to your own server..You want a piece of software that will do that to another persons server? Or better yet..you just wanna use random SMTP servers to ruin the reputation on those servers instead?..Oh wait, and all for next to nothing!

I think your options are to either download a mailer..and send them using either your IP (or proxies) and other peoples mail servers (which could come back and bit you in the butt)
Or....you're gonna have to pay someone for this...

 

[Paul]

@buyselldoms and @FPForum, if you don't have experience with something, you're best off not bashing the people who know what they're talking about. @TalkDevelopment didn't post a question here just so you two could make fools of yourselves in front of everyone who knows how e-mail works.

Spam is a big problem. Very big. Not just for the people receiving mail, but also for the people sending legitimate mail. For example, until very recently, Yahoo marked almost all e-mails from NamePros as spam automatically. Sending e-mails to a large quantity of users is a very delicate process that involves many technicalities your typical small website owner would never encounter. Here's a small checklist for sending legitimate mail:

• Sign messages with DKIM
• Use DMARC policies
• Use SPF policies (not really necessary, but can help)
• Make sure Sender and From match
• Mark bulk e-mail as such
• Implement Return-Path handling
• Provide an Unsubscribe link with support for easy, instantaneous unsubscribing--usually required by law in the United States
• Don't send all messages from the same server
• Proxy through third-party servers that have good reputations (can be costly)
• Avoid blacklisted words (for example, Google likes to blacklist anything Bitcoin-related)

It can even get much more complicated than that. Sure, you can send a handful of e-mails without doing anything fancy, but once you get up into the thousands, the big providers start to notice. They flag the silliest things, especially Yahoo. Yahoo flags all sorts of things with no good explanation and refuses to provide postmaster services to resolve such problems--oh, and they let the spammy e-mails right through. (Moral of the side-story: anything is better than Yahoo.)

E-mails can be divided into three groups:

• Personal: These e-mails are sent by hand directly to other e-mail addresses. They're usually not marked as spam.
• Transactional: These are e-mail that are triggered by events specific to a user. In most contexts, this includes notification e-mails For example, if you receive a private message on NamePros, the resulting e-mail that we send you is transactional. This also includes subscription e-mails that are necessary to provide a primary service to a user, such as monthly bank statements. Opt-in subscription e-mails, like those that you can choose to receive on NamePros, are considered transactional.
• Marketing: Anything that is not personal or transactional, whether or not it is actually intended to be promotional. Newsletters and all other bulk e-mails fall here. This also includes most opt-out subscription e-mails and many opt-in e-mail, as well. (If the checkbox on a form is checked by default, it's opt-out.)

There are companies that provide e-mail proxying services for businesses sending legitimate e-mails. They track each e-mail, and if a sender gets too many spam reports, they'll have their account suspended. This way, their servers gain high reputations with the big providers: there's a high probability that e-mails originating from them are not spam. For most large forums, sending e-mail directly via SMTP wouldn't work well; they'd need to use one of these services.

There's a catch, though: most of the e-mail proxying services out there only allow transactional e-mails. The ones that do allow marketing e-mails charge a lot more, or have separate prices for transactional and marketing e-mails. Bulk e-mailing becomes very costly.

@TalkDevelopment, the reason you can't find any open source software to fully validate the e-mails is that you'd need a full e-mail proxying service. While such a service could be open source, it couldn't realistically be free--at least, not without some catch. Servers are expensive. Amazon Web Services offers a relatively cheap e-mail service called SES. It's not the most featureful, but it's definitely cheap, and more reputable than many alternatives. 100,000 e-mails would cost you about $10.

 

[FPForum]

@Paul Buonopane - Thanks for your nice detailed explanation (Although some* probably won't read all that..lol) Aside from that, you have no idea on my past and what I might know about emailing. Sure, I'm not gonna sit here and write an entire book like you did, we all know what proxies are..SPF records...and most the other stuff you mentioned. If you think "Or....you're gonna have to pay someone for this..." is incorrect information and I don't know what I'm talking then maybe should refer to the last paragraph of your own reply. Hmm, proxies..email services..cost.... You crack me up

 

[buyselldoms]

FPForum what makes you think people wont read it all? Its detailed and to the point and presented nicely rather than being one big chunk of unjustified text. This said, I am not sure what triggered Paul into referring what he did about me but I am sure for the most part it was referring to FPForum.

My views on the matter remain the same... its an old database in terms of the internet....

1) people who haven't logged in for years aren't interested, unless you can specifically track them to visiting the website without logging in... IMO emailing an outdated list is spam.

In the past I heavily used catch-all email and used the format (for example...) ebay-a2Dx1lO0S @ whatever... so if I got an email I am 99% certain whether or not it was leaked, stolen, shared or sold on... I knew it was (in this example) ebay... "ebay @" wouldn't be enough as anyone could put that, having a random combination of alphanumeric characters eliminates this unless someone was in the knowledge of me doing this for that domain. obviously, not happened with ebay, not the best example, but short and sweet. You get the message.

Although not my primary email, I still get emails from sites going back 10+ years... some are "enter your email to download" like sites. Visited the site one off, never gone back... this email is now sold on and used for goodness knows what other sites. A decade later, still get them.

2) the forum never had verification (from the start) and there is the strong belief of many fake emails, which have absolutely zero value whatsoever.

3) the follow up from open source/free ways to clean email lists, was just as much about checking if real emails were still valid/checked, than fake emails (by intent, not necessarily invalid - could be someone making them up or putting someone elses email in) (i.e. a @ b.com, bill.gates @ microsoft.com, two @ four.six etc. depends what forum software and how it checks email address formats)

4) overtime its highly likely that people stop using their email account and get a new one

5) if the forum/website still gets traffic IMO its better to convert *current* users than inactive members from the past. I would email recently inactive members (say within last 6 months) but any longer than that, despite them receiving the email, I don't think it will convert to many hits. At best, never beyond 2 years of inactivity

Its important to understand peoples behavioural patterns especially in regards to online communities. Take this forum for instance, whereas some popular posters have been using the site less (I assume focusing more elsewhere, we can only assume without the facts), some people, like myself are using other forums less and using this one more.

With forums you either visit many and post infrequent, or you keep coming back to one or two posting frequently.. if this isn't clear from running your own forum for coming up to 10 years... you aren't going to be able to market a new website, that forum or a product. The problem you have had is you failed at customer retention... you let the forum slip when you should have been reviewing the forum every 6 months to ensure everyone remain active. Everyone tries it, but web forums are really difficult to start-up and to keep going.

I am on a very popular business forum, however, even they employ freelancers to keep the posts coming. You have to keep your veteran members happy and engaged by stimulating them with continuous content - they double or triple this. Then and only when the forums are very frequent do you get new people joining to be part of it. I am sure we have all done it... "ooh, that looks like a nice forum but no one has posted in half the boards for about 3 months and the rest only get a few posts per week"... so we don't register. This causes a chain reaction where everyone does the same and results in a lack of new members and content.

As this is under Web Dev and Server Admin... I think its important to highlight by allowing members to become inactive you start to damage the forums reputation. Do you have a link to the forum?

 

[FPForum]

I think a number of people in this thread are reading into things way more than needed. It seams like some of you are sitting here trying to explain to the OP how email works ... or you want to give your 2 cents about how you used to have an email and you don't use it anymore..yadda yadda yadda.. Let's try to get back to the question at hand, shall we? He has an email list..and he wants to clean it of fake or invalid emails.

Does he really need a checklist for what is considered a legitimate email? Does he really need your story about how you used to have an email and you got rid of it years ago and still get emails to it now? The guy has a 3 sentence question and you guys are responding with answers that practically need chapters! lol

You have an email list and you want it cleaned..for free or as affordable as possible. Do you know of a provider, solution, or piece of freeware that can do it? That's the question here so why not stick to it. It's already been said that you have a few options, most of which are not going to be free but they're available if you want to try them. I think now we can move onto the next topic and stop talking about your opinion on what you think peoples habits are, or how long you owned an email address for..etc.

If the guy wants to email people from 2005 then that's his choice..I think he's probably gonna do it the way he wants either way.

 

[buyselldoms]

Are you okay FPForum? You just seem to lash out at everyone for no reason whatsoever.

 

[Paul]

The purpose of this thread was to ask for names of services and software that can be used. The OP was not asking for opinions on the usefulness of his data. A single wayward post wouldn't have been a problem, but this is spiraling out of control. Please stop posting off-topic information and opinions. It'd make a great debate elsewhere, but it's distracting from the original purpose of this thread. Feel free to start an additional thread on the matter. I think you've already scared away @TalkDevelopment.

 

[nielsencl]

Boy, posting quality for responses to someone who is asking for help have really gone downhill here! Between people not reading or not understanding what the OP was asking for, to those with their personal opinion that they think are so important that they dismiss what the OP wanted.

On the other hand free advice is often worth what you pay for it.

For what it's worth I will add mine to the pile.

I have seen scripts in the past that will take an email list and validate addresses by starting a SMTP session but after sending the address and having it accepted, the session is dropped before any message is sent. It should be a fast and inexpensive way to validate that an address exists without any risk of spam. The problem is that it doesn't tell you if the email address is really being used any longer. The person could have lost the password, stopped using it because of spam, or died. If you want this and can't find it I can try and help you.

But my advice as a consultant would be to actually use the list and send a message to all the addresses. Wait, I know you have objections. There are a number of ways to do this and not run into problems. But if your return email address is at Gmail or Hotmail, then I would advise against using that. If you have your own domain then I would use that. Using email software that comes with most Linux hosting accounts, you could set up the list and send out a message to help you do what it is you are trying to do (not sure what they is exactly). I would make the message really short and NOT make it opt-out, but make it opt-in and be ready to delete any addresses that don't opt-in or confirm they want to be contacted by you again in the future. The message is crucial and should avoid any possibility of appearing as spam and not an email list confirmation / opt-in invitation to an old forum member.

If you are still scared about the spam thing, I would get a new domain and cheap hosting account and run the list from there after explaining what you are doing to the hosting provider.

Chances are you WILL have issues with a large number of addresses no longer being valid.Most average user seem to change them every few years due to job changes, ISP changes, or to escape from spam. But I have been using the same basic email address since 1999, so there are those that may still have the email email address after "only" 9 years.

 

[Paul]

@nielsencl, half-sending e-mails is a good idea--I hadn't thought of that. You'll at least know if the e-mail exists, and as long as you don't hammer any servers, you probably wouldn't get blocked. Spam filtering wouldn't be a problem. Honeypots might.

It's worth noting that most ISPs put their residential IP allocations in various "no-SMTP" lists. For this to work, you'd have to run the script on a server or at an office, not on your home computer. Otherwise, you'd risk getting rejected immediately, which would yield a false positive--or, worse, your ISP might notice.

 

[TalkDevelopment]

To those that have offered assistance, thank you.

I believe I finally have things under control at least for now.

Just to clarify a bit, I do not usually mail to the entire list at once. I do normally only send emails out to members that have been active with in a certain amount of time.

There are times though emails go out automatically to older members like when someone posts on an old thread or something like that. Many times these posts get members back after years of absence.

I do not want to post the link here but the type of forum it is has a member base that sort of loses interest and then gets back into it. We have many members come back after years and years of absence. it happens all the time and through poling them when they come back I discovered that most of these members still view the site and read the emails. Many just do not log in or have not in a long time. For this reason is was just not wise to throw out those email addresses.

if a member who was very active with a major profile and thousands of posts leaves the site for a few years but then decides he wants to come back we could lose him if there is no email address there for him to reset his password, etc.

At any rate, things are going ok now though it was a major hassle. We only send out legitimate opt in email but these days that is not enough. You can get blacklisted as spam just for not having your domain reverse mapped to your ip. I had to do some major work to get going again including filling out forms with Microsoft, signing up for spam reporting programs and some server and dns work also.

I have been able to remove most of the bounced emails though it is an ongoing process. I have also posted on the site to let members know to make sure they have valid emails.

Thanks again for the help,
Rich

 

[N-A]

GreenArrow with professional installation included would have done all that for you.

 

[TalkDevelopment]

GreenArrow with professional installation included would have done all that for you. :P

That was one of the first solutions I checked out but I never buy a product like that does not have a price easily visible.

 

[N-A]

I was skeptical at first because of this. I've tried both Port25 and GreenArrow. Both do not disclose pricing as it detracts users. I mean, who wants to buy Port25/GA when Exim is free? Serious buyers will inquire. Both systems are in the same league as far as licensing, pricing and operations go, but I find GA more user friendly.

They don't work on commission (so I've been told), so they don't pester you to make a sale if you just ask for the price. This is why I put it in the thread, as I've inquired on monthly, yearly and lifetime licence prices on the various send rates that they have available, which I believe is 3 levels. (By that, once you tell them you don't show interest, they will no longer email or call you [I got one email at the end of the trial asking if I liked it and would like to call in to discuss more]).

I can only speculate that it's not easily visible because there is no "buy now" option as you have to physically sign documents agreeing to use the software as it's either monthly or yearly (though, you can get a lifetime license... but I think that has contracts to go with it), etc. and they also are flexible at pricing and structuring a plan just for you in the beginning.

To anyone else thinking about doing what @TalkDevelopment did, it's well worth the 10-40 minute conversation to have with the sales rep about the product so you can get demo access before purchasing it to see if you need to hire someone to help you run it (GA has authorized companies that they can forward you to) or get to it yourself because you've had practice.

GA literally took me 10 minutes to figure out (it's so uncluttered, unlike WordPress—in fact—if you were to rank the two by being user friendly, GA over WordPress hands down) and a cool option about it is the ability to create an opt-in form. The form takes data X, Y, Z for the storage, confirms email and redirects to a specific 'thank you' page. It returns it in their own HTML code after "designing" it, but you can modify the form to fit just about any subscription widget (or build your own) by modifying the posting URL, text box names, etc. and having the thank you page already setup.

Either way, I'm glad you got this resolved and that others couldn't see past what you were trying to accomplish and/or attacked you. If you need any assistance, feel free to drop me a convo.

 

[Paul]

Given your update, @TalkDevelopment, I would recommend setting up DMARC, if you haven't already done so. It seems to help decrease the number of false positives with all major providers except Yahoo. Yahoo just does whatever the hell they want--which means marking everything as spam.