Ask HN: Is it possible to run your own mail server for personal use?
Recently I decided that I wanted to run my own MTA. Downloaded qmail, applied a couple of patches and it was done. The problem is making sure my mail is not marked as spam by the major MTAs out there, gmail and hotmail both mark my mails as spam. So far I've:
- made sure I'm not running an open relay (obviously)
- made sure a reverse lookup to my IP matches my host
- made sure my IP wasn't already in any spam IP blacklist
- added SPF
- added DKIM
- run a lot of tests at glockapps.com
It's still not enough. I don't know what else to do, is it possible to run your own MTA, for personal use in 2016? Who, here, is doing it successfully?
313 comments
[ 4.3 ms ] story [ 276 ms ] threadEDIT: Looks like this is one of the things https://glockapps.com/ checks.
https://news.ycombinator.com/item?id=12109727
about some hard-to-find tools that help with Microsoft mail (Live.com / outlook.com / hotmail.com etc). There may have been some similar strange link for Gmail - i only have access to my phone right now, so I can't check my notes.
I believe that was the thing I was missing from my setup (and it was frustrating to debug as mail simply disappeared).
Between a (for now) cacert-certificate (which I think is treated as "self-signed" for most purposes) and SPF - both Google and MS appear to accept my mail. I've turned off ipv6 - if you want it, it might take some extra work.
I'm planning to move to opensmtp "when I have time" and maybe dbmail for the store - now I'm on exim and dovecot.
I get little spam - I do have greylisting set up. What I get is generally to emails leaked in the linked-in and Adobe attacks. I should change those addresses, and reject/black hole the old ones. But for now it's manageable - and is filtered out in my "misc" folder by simple address-basrd filtering (exim .forward file)
(I generally give each site a different mail, se eg my hn profile).
It's like trying to make a toaster from scratch, or growing your own wheat to make your own bread. It's possible, but it's also impractical and you'll end up with a worse result than you can get off the shelf.
I've been running my own e-mail server for years. I have DKIM, SPF, PTR, and everything else set up and my e-mail domains and IP addresses are many years old. Every so often, Gmail still decides to spam-bin me for a week or so.
Eventually though, gmail will typically accept email from home IP addresses. Google apps for business seems to be even trickier than that: they have an opt-in strict setting which blacklists (outright rejects) anything remotely suspicious. It'll tell you where you stand through delivery failures, but it also means you'll end up having to call whomever you're trying to reach.
I'm sure if you're willing to put in the effort you can do it. But from my point of view, I'd probably just get a managed VPS with a hosting company who will take care of all the headaches of dealing with spam filters for me. They can be had pretty cheaply and the money is well worth it if you get good support.
Or sometimes your IP will just get caught when a spam company decides to spam flag an entire block of IPs.
I think if OP keeps at it, they will find that it's not worth the hassle and for a very reasonable amount of money they could have just had a personal email server where someone else deals with the headaches.
Of course, again, I work for a hosting company so I may be biased here but I'd probably be saying the same thing even if I didn't.
Therefore I'd advise OP to keep at it, not just switch to a hosting company. Also for the learning experience, though that is secondary.
Here's what I've done on top of your list:
- backscatter prevention (using my own https://github.com/pflanze/better-qmail-remote)
- do the Google domain verification dance (postmaster tools, configuring their entry in the DNS); still didn't prevent mails ending up in spam, but who knows whether it might still have helped.
- started running mailing lists on it anyway, in spite of me knowing that mails end up in people's spam folders, and simply tell all new subscribers that mails first end up in their spam folders and that once they mark them as non-spam the problem goes away. This seems to be working (people haven't complained), and will over time hopefully give my server the reputation I need.
(PS. I'm also still using DJBDNS, with a config generator written in Scheme, look out for tinydns-scm on my Github)
I'm using maradns for DNS but I respect djb's software a lot. I'm using his publicfile as an httpd.
I have already added my domains in google's postmaster tools. So far it hasn't helped much
Small users generally don't use Qmail anymore, so if you're a small sender and use Qmail, then perhaps that's odd, and iff spammers (perhaps for Qmail's efficiency (1)) are using Qmail often, then...
(1) although you can attain higher efficiency with a custom MTA that's not safe, i.e. big spammers generally don't even retry for efficiency (that's why greylisting works).
A friend of mine runs several mail servers on OpenBSD, using OpenSMTPD, and says he's got almost zero problems and scratches his head on what I'm doing. So there's 3 hypothetical reasons for that for me:
- basically no spammers will be using OpenBSD (why should they?), - and hence also none be using OpenSMTPD, - and hence also none of them will be using a OpenBSD hoster (netblock!).
Relatively many spammers (perhaps not the biggest ones, but still spammy senders) will be using services like Hetzner, and will most likely be running Linux. Now if Gmail etc. do check the MTA you're running, and perhaps do OS fingerprinting, they'll "know" how to judge you.
Anyway, I don't have evidence that Qmail is a big reason, it seems likely the netblock is much more important (or at least that's what I'm inclined to think because the alternative would indeed be rather frustrating).
That said, SPF is worth looking into - but it's more a "good neighbour" thing. Rather than having any direct benefit for you, it helps other hosts spot mails that claim to be from you. "Be the change you want to see" type deal.
Be careful testing your configuration when sending emails to the large providers, you can inadvertently score negative marks against your own reputation, which is hard to recover from.
The reporting sets my mind at ease that those big guys are blocking it and that the (low) legitimate volume of mail to those guys is reasonable.
It's also interesting to note that with DNSSEC, DKIM, SPF and DMARC, the pattern seems to be that some large Chinese mail providers drop DNS responses to try to overcome the "-all" token in the SPF record and "p=reject" token in DMARC. At least the reports show that the authentication (by SPF/DKIM) failed, so that makes me sleep a little better.
Par for the course, I guess. :P
*edit: grammar
This is a little bit like "correlation is not causation". Companies are moving a lot of stuff to the 'cloud', and email is just one of the services that goes with it. It's simply cost-saving to not have dedicated IT personnel. Even if hosting your own mail server is easy, you need someone who takes care of it every now and then (disk full, adding users, blacklist incident once a year, etc.).
The internet seems to have decided that if your mail doesn't come from one of the major commercial mail brokers, it's as good as spam. I ran several private mail servers for several years before giving up. I'm generally suspicious of moving to cloud/services, but outgoing email is the one thing that I can safely say 99% of companies would be better off moving to a service.
I remember fondly the days you could send an email to postmaster@domain.com and get a response from a real human being and at least attempt to have a rational conversation to resolve your mail delivery issues. I've tried that with google, yahoo, et al. and have never received a response.
I just use mailgun/mandrill/etc. these days and forget about it. I also pay $10 a year to a provider for my personal email address.
Postfix occasionally drops a legitimate incoming email due to a misconfigured sender, usually from a domain that doesn't resolve to anything, but I just log those in case I miss something important.
Not really. They'll suspect them, especially ones that newly start sending email, but only very rarely block them.
Two real problems I have faced...I once (embarrassingly) created a unix user with test/test credentials for messing about and forgot that my postfix setup at the time reused unix credentials (ssh was locked down to only allow specific user to log in). I sent a few million spams a hour for a couple of weeks, getting my host into all the DNS blocklists. This took some time to fix (you have to apply to have your host removed from the blacklist). While I'm sure sending all that spam was annoying to many other people it didn't actually affect delivery for my mail...so it seems other administrators aren't using DNS blocklists?
Second, after a while I started to get a lot of spam. Maybe 10 per day. I tried various things to handle this, including setting up a proper bayesian spam filter (amavis-new) and using DNS blocklists myself. None of this worked for me. Greylisting however worked great.
So my suggestions to you: use defence in depth for mail as well as ssh. That means, fail2ban, different creds for both, unusual ports, user whitelists, high patchlevels (auto-patch and restart is great for a personal mail server) maybe client side TLS certs...etc. If you're relying on a single layer of defence eventually you'll make a mistake with it and then you're in trouble. I guess that really applies to anything you're trying to secure.
One of the things that is the most frustrating is that if you end up on a blacklist, or a large provider decides independently not to trust you, it's often completely silent when it blackholes all your outbound emails to that service.
I've just moved over to a hybrid of hosting my own MX servers for incoming email, and forwarding all my outgoing emails to an email-as-a-service provider for outgoing messages. Their trusted IPs usually help delivery, and they're actively paid by their users to have employees making sure that their IP ranges are whitelisted.
Even when I switched mail server IPs twice over the last few years I didn't run into the issues you ran into. A large part of it depends on where you run it - if you, say, run it on your home Internet connection that's usually an immediate strike against you because of the insane number of spammers using backdoor'd PCs to do exactly that.
The only time I ran an MTA out of my home was when I was on a commercial ISP with a fixed IP address, that seemed to be good enough for most services including gmail and hotmail.
These days I run my MTA on a VPS with a reputable hosting provider and don't seem to have that many issues with outgoing mails marked as spam.
SPF and DKIM are pretty much a must these days, so that's a good starting point, as are the rest of the precautions you already too. I assume you're using your own domain, how "old" is that domain? That might also have an impact giving how many phishers and spammers register odd domains and use them for a short amount of time. I've used the same domain since about 1999 so that could make a difference.
I use postfix instead of qmail, but I've used qmail in the past. Both work well and are easier to configure than sendmail or exim IMHO. On top of that I do run amavisd/spamassassin/clamav for the incoming emails as well.
One more thing I've got set up that I didn't see in your list is that I've got TLS set up with a non-self signed certificate for both incoming and outgoing email. I suspect that this also makes a difference even if the other email server won't request a client certificate (most, if not all, won't). Certainly shows up when I send an email over to gmail.
My biggest issues these days are more with incoming email:
- You'll never get to the level of spam filtering that, say, gmail offers. To me, that's OK
- I use greylisting to weed out a lot of the spam that would normally make it through spamassassin, but unfortunately that's when you find out how many people have misconfigured servers that bounce emails when they encounter temporary failures
While true, some IP addresses were not previously used, either by home connections or by anyone, so you may be in luck there. There seems to be a correlation with my latest IP switch (three years ago or so) and less mail server trouble.
Most services are also smart enough to learn to trust IP addresses after a little while. Even when SPF and DKIM is ignored, this usually solves it in a few emails.
ISPs definitely seem to have an impact as well, but some spamfilters at least used to automatically assign a negative score to anything originating from an IP block used by an ISP for domestic Internet connections.
ISP state in their TOS you are not to run "servers" from your home connection, as such they simply blacklist all Residential IP;s from the start, because if you are running a email server on a Residential Connection it is either a Botnet, or you are violating the TOS of your plan.
I don't know what evil ISP that is but running your own server is 1) encouraged by my ISP since forever (XS4ALL, the Netherlands) and 2) these days it's illegal to tell people not to run servers because of net neutrality laws.
Current Comcast AUP (http://www.xfinity.com/corporate/Customers/Policies/HighSpee...)
>>use or run dedicated, stand-alone equipment or servers from the Premises that provide network content or any other services to anyone outside of your Premises local area network (“Premises LAN”), also commonly referred to as public services or servers. Examples of prohibited equipment and servers include, but are not limited to, email, web hosting, file sharing, and proxy services and servers;
For years it was qmail, but when I wanted to use SMTP/SSL as much as possible, switching to Postfix was easier than maintaining all the qmail patches.
I switched over to Let's Encrypt certs several months ago, and those have been working out quite well for me.
Didn't you mean "outbound"?
As far as I can tell Comcast employs a number of very competent network engineers and an astounding number of horrendous customer service and executive managers.
I don't actually greylist any more, though -- I've found that postfix's protocol violation detection (postscreen[0]) is just as good, if not better, and I like my mail to come through immediately :).
Amavis is set up to do pre-queue filtering, which means that mail is either rejected or delivered (possibly to the spam folder), my server shouldn't be generating its own bounces or dropping messages on the floor.
[0]: http://www.postfix.org/POSTSCREEN_README.html
Make sure you don't send spam and your server is properly configured. If you are sending mails to people that don't want it then it is spam. "They silently agreed to get our newsletter because it was listed in our ToS on page 357" is not acceptable. No other excuse for sending spam is acceptable. Whenever you send any automated mail there must be an easy way to unsubscribe.
A few more tips:
* Check your mail server on http://multirbl.valli.org/ - if it's in any blacklist try to find out why (there are a few rogue blacklists, ignore them).
* Hotmail allows you to receive a report for every mail that a hotmail user thinks is spam. Use that. Act on it.
* Check your logs for messages that indicate that others think you're spamming.
* E-Mail forwarding is a tricky business these days. Avoid it.
I occasionally get dubious spam rejections, but they don't come from the large hosts. They usually come from some small ISP using a proprietary antispam solution that gives you no insight what's going on.
My suspicion would be that qmail is your problem. There are a great many details that a mail software has to get right, qmail often doesn't do what the email ecosystem expects.
For spam defence I use grey-listing based on DNSBL lookups, some standards-enforcement ACLs and then SpamAssasin via MailScanner on the messages that get through.
Reverse DNS very important and the SPF
Linode have excellent setup documentation ( https://www.linode.com/docs/email/postfix/email-with-postfix... )
[0] https://www.linode.com/docs/email/postfix/email-with-postfix...
[0] https://wiki.centos.org/HowTos/postgrey
Fun Fact: Microsoft Exchange has no native support for DKIM and tons of business run that in house, often on 'business class' connections without correct reverse IP records.
I don't know whether OP uses a home connection, but that makes a big difference. It has some pros and cons:
A VPS (with its dedicated IP address):
- Usually has a good reputation, though you might have bad luck. Switch early on when you notice this. Doing it later just nullifies any trouble you had building its reputation.
- Is hosted somewhere in the cloud. (I used to hate the term "the cloud" but it's actually pretty apt and funny if you start seeing it as "they have no idea where their shit is and who has access to it, both physically and network traffic".)
Hosting at home:
- Might be trouble with your ISP if you don't live in a country with net neutrality and port 25 is closed.
- Gets you a lower reputation score by default, so it takes slightly longer to build.
- Gives you full access control over your email.
Hello, I run my own mail server, though admittedly with an attitude of "your spam filter thinks I'm spam? That filter is broken; have fun reading your spambox."
Company email addresses are actually never any trouble anyway, only personal ones (the free ones like yahoo, gmail and hotmail) are the ones where people have trouble with broken filters, so I don't think I'm missing out on anything. And even those filters usually learn (after a few emails to different accounts on the service) that my IP address is not to fear.
I add SPF records but don't sign with DKIM (too much trouble; I set this up years ago when I didn't have much experience yet).
The last time I had trouble sending email was with (of course) google apps. Some company, whose product we were required to use by school, had no privacy policy so I wanted to ask after it. Sending them an email, google's mail server outright refused my IP address to deliver a message (this is extremely rare, usually it goes into a spambox). Google is the only one that can get away with this, given the near monopoly, without people thinking it's an issue on google's side. In the end I just didn't send them an email. Also quite ironic that google, of all companies, is the one standing in the way of my email trying to ask after a company's privacy policy.
This was half a year ago. Before that I can't remember having issues with anyone or anything for another year or so. Given how much I use email and how little spam I receive (catch-all with a blacklist), owning a mail server is totally worth it. Also because I don't have to accept anyone else's privacy policy or consider how many people have access to my inbox (I host at home, not colocated nor VPS).
I actually use sendmail with bogofilter-milter.pl script (there is also bogom program with does similar thing) as opposed to usual way of running it with procmail, it essentially does the filtering at the moment it is being sent to you and rejects spam right on the spot. The advantage of it is that sender will get notification that spam was not delivered, also many spammers might drop you from their lists when they can't deliver spam.
These days you also have to get a bunch of different VPSs and test sending e-mails from their IPs and choose the ones working. That's what e-mail marketing companies do. Because a lot of IPs and subnetworks out there have poor reputation or even completely blacklisted and that reputation is not generally recoverable.
... and that, in turn, is why so many IPs have poor reputations.