I wrote this article because I couldn't really find a good email tutorial that included DNS setup. I hope this helps you out. It's 2017 and it's surprisingly difficult to set up an email server, but hopefully this will make it a little bit easier. Please let me know if you need any help or you think I missed something major :D
Thanks for writing it. Do you intend to include a reverse DNS record for IPv6? I remember doing it once and the syntax was mind-bending. Here is a generator: http://rdns6.com/hostRecord
Thank you very much! I never actually learned how to set a reverse DNS record for IPv6 because because vultr doesn't use IPv6 by default. I think that it might be a good idea to look into though.
Alas, you haven't extended the self-hosting idea to your blogging. You publish on Blogger, which throws out some garbled js, and otherwise refuses to show up on my screen.
Yeah that's true. I set up blogger a few years ago and I just haven't gotten around to self hosting it yet. I have a todo list of things I want to switch over to.
This is very interesting. Recently I realized that if tomorrow the big G decides to take over or close my email address I lose access to all my online accounts.
Step one to freedom was to use my own domain, redirected to my old account, but I'm seriously thinking about doing it all myself.
But keeping a whole server up to date, secure, etc.... that's a full time job. Is there a good solution?
They stopped scanning for purposes of targeted advertising.
However, they still scan and read email. People can't be inconvenienced. I tend to think this is worse than scanning for ads.
I also tend to think anyone concerned about privacy moved from Google long ago. After all, its terrifying when you allow a company to determine if perhaps your received/sent links are fake, or malware, maybe they should report that you like certain sites, or have a flight.. People lived the idea of the President maxing out his power with executive orders, until they didn't.. Life is ask about tradeoffs i suppose..
I have been down that route before. My realization come when I was blocked out of my main GMail account for ~24 hours. I have consider setting up my own email server, but in the end I just use FastMail (paid plan).
Keeping the server up to date and secure is mostly done by your distro maintainers. The only issues you will have to deal with is configuration and deliverability.
However, that $2 per user per month only applies if you buy a minimum of 5 user licences, so it worked out more expensive for me than the 5 Euros a month for ProtonMail Plus:
Tutorials for setting up a personal email server are fine for educational purposes -- or -- if you're primarily using it to receive emails.
On the other hand, if you absolutely depend on the ability to send emails such that your recipients reliably get them, hosting your own email server is extremely tricky. One could carefully go through the checklist of SPF, DKIM, ip blacklists, etc and emails will still be rejected by MS Hotmail, GMail, Yahoo, etc. Those giants are "black boxes" when it comes to their heuristics for rejecting incoming mail as spam. E.g. your hosted email server does nothing wrong but some other bad actor on your ip block sends spam which then makes MS/Google block you because of "guilty by association". Trying to debug your "sender reputation" is not easy.
In the 1990s, I hosted my own home email server over ISDN lines. These days, I have a million other things I'd rather do than babysit a personal email server with software updates, SpamAssassin lists, etc. I get the whole decentralized ethos but it's just not worth the effort for email servers.
No, it's real. I gave it up a year ago, after hosting my own mail since forever. Exactly as stated: Hotmail (or whatever MS mail is called these days), GMail etc. are the main culprits. So yes, 'colossal' is the word.
Yep I did the same. I ran my own email server off the same IP address for 15 years with zero problems, but I had to give up recently because email is not something that you can do on you own anymore. You now have to pay some somebody to vouch for you or else MS will send your emails to spam or even worse just randomly delete them.
Something should eventually come along and replace email. Like bitcoin and other technologies, what was originally designed to be decentralized became centralized in practice.
Could be you had a customer/other recipient like my ex-boss: I'm fairly certain he used the spam button instead of delete on his Google apps for your domain (now gsuite I guess) account.
Hosted my own email for 18 years, same experience, switched to pobox.com about two years ago. It really burns me up, but I need my emails to arrive where I sent them.
> E.g. your hosted email server does nothing wrong but some other bad actor on your ip block sends spam which then makes MS/Google block you because of "guilty by association".
No. That is not why, the reason must be something else. The only explanation that makes sense is deliberate sabotage to eradicate competition. But I'm not suggesting that, it is just that I can not think of any other reason.
What is the motivation for the "guilty by association"? Because you can not verify who sent it.
With SPF and DKIM you can - that is the whole point, you know beyond any doubt that the machine supposed to send that mail actually did send it.
Now, if the person behind that machine is a spammer (or has been hijacked by a spammer) and sends out a lot of spam (and gets reported as such) then just nuke that sender. It is dead easy to do regardless of what IP it was sent from - each mail have identification credentials built in. It could not get any easier to do. The sender did all of the work for you by going through the extra effort of identifying itself.
Yeah, but they don’t identify themselves correctly. They lie.
Sometimes they claim to be real people with real e-mail addresses that they don’t actually own, and maybe that real person is you — in which case it is your reputation that gets permanently dinged.
Sometimes they claim to be fake people with fake e-mail addresses that don’t actually exist, in which case all you really know is the IP address they used to send the message. When they do this, they frequently arrange to get a large block of IP addresses they can use, and then cycle through them very quickly — maybe no more than five minutes per IP address. Anyone else who happens to be in that same range now gets their reputation dinged by association.
There’s a lot of tricks spammers use to try to hide their activity. And lots of ways to try to detect and reject that on the other end. It’s a never-ending arms race, and on the Internet it is easy to hide and hard to take out miscreants without causing excessive collateral damage.
And many big providers don’t care if they cause collateral damage.
Spammers have SPF and DKIM set up for their domains, too. Why does it seem so impossible to you that if mail hosters see "this IP block hosted a lot of domains that turned out be spammy" they react with "block it" or at least "block it if new sites pop up there"? (and as a private user, you're often enough "new" because you don't send all that much mail)
Yes, it'd be nice if they didn't do that, but small sites are and will be acceptable collateral damage to them.
> What is the motivation for the "guilty by association"?
One that has been given in the past is "Stop paying money to companies that host spammer websites and spammer email servers, or we'll block your email too"
(I don't particularly agree with it, but it was common to see that argument on news.admin.net-abuse.email
SPF / DKIM let you know that who you think you're receiving mail from and who you're actually receiving mail from are the same person. How that that come into this at all? Spammers can say they're from onlinechemist@spam.com via SPF / DKIM and it's still going to come through.
I don't know who you're even trying to convince. You're talking in hypotheticals and being extremely idealistic. How the email ecosystem works and how it actually works are two very different things. You cannot say "the reason must be something else!" as if - just because you can't make sense of the situation - it is not so.
Then you reduce the credibility of spam.com and possibly any links to whoever setup that domain etc.
Not whatever IP was used.
I'm not being idealistic when I say that I should be able to send an email over internet and have it delivered without someone banning me for reasons beyond my control.
> What kind of colossal idiot would block an email because it comes from a certain IP-block IF SPF and DKIM has been properly setup?
I don't know if anything like SPEWS is still around but they used to do large blocks. I still don't know if it was genuine attempt to reduce spam, or performance art, or god-tier trolling.
Statistically, a random mailserver on a random IP address is overwhelmingly likely to be sending spam. All the large mail providers use Bayesian filtering and are willing to tolerate a modest false-positive rate to ensure a low false-negative rate.
A spam blocklist is not a court of law. As far as the mail providers are concerned, you're guilty until proven innocent. SPF and DKIM are no guarantee that you're not a spammer.
No, in my extremely limited experience with email deliverability, this happens. I'd imagine a person that actually does this stuff day-to-day sees it a lot more than I do.
You can do "everything right" rolling your own email, but you are still relying on the receiving parties to do their job and implement sane spam prevention policies. This is far from what actually happens.
I lead development on a project that sends a lot of (legitimate, user-requested, transactional) email to many mailboxes belonging to a certain government department. I used Amazon SES as they have a pretty good track record. The receiving mail servers were configured to use a few overzealous blacklists that were causing my email to be silently swallowed (and reported as delivered). This was based on an IP black-level ban. Simply Googling the names of these blacklists showed report after report after report of them being completely useless, with many seasoned sysadmins saying that nobody worth their salt should pay any attention to them.
This was extremely hard to troubleshoot as this department was not a direct client and was of limited helpfulness. It became apparent that using SES's newly-launched dedicated IP functionality wouldn't have helped. I made full use of my AWS Business Support, the multiple people I talked to made it clear that SES put a great deal of effort into ensuring their IP ranges are "clean" - but there are some spam lists that do not play ball. The most important takeaway was that I was doing everything right. I was using a reputable mail provider who have very stringent spam prevention controls. I utilised SPF and DKIM. The content of my mail shouldn't have (and didn't) trigger a content-based spam detector. It was because the people on the other side weren't doing their job to an acceptable standard. When I was eventually able to open a channel of contact with the receiving department, they confirmed it was an IP range-based ban.
The fact that I was doing everything in my power to do the right thing had no effect on the situation. I still wasn't getting mail to customers. Saying "darm, these sysadmins really aren't doing their job properly!" had no effect on the situation at all.
We still live in the age of almost-but-not-quite-plug-and-play email solutions like Exchange that can be configured by people that probably shouldn't be configuring mail servers. For all I know, there was a list of blacklists on some Hosted Outlook control panel and the person in charge ticked all of them because more is better. As long as this is the reality I live in, I'm going to do all I can to play ball and succumb to the black magic. I can imagine my experience would've been a lot worse if I had been operating a mail server out of a dirtier IP range (generic cloud hosting provider, retail ISP customer IP range.etc). If I recall correctly, AWS severely limits sending mail from EC2 instances because they don't want their IP addresses / IP ranges to end up on mail blacklists.
There are more collosal idiots around than you allow for.
Also, it isn't in said idiot's interest to deliver your email properly. They would rather fart around with machine learning and blame whatever happens on incomplete training data...
MS actually have a service where you can register IP’s you own and it gives you a digest of what people have marked as spam that originated from said IP’s.
Their headers also tell you a lot, excepting the content filter one which is a bit of a black box and I don’t believe actually is purely a content filter based on messing with it.
(Responding to black box comment)
Smart Network Data Service[1] which, like Google's postmaster[2], is not terribly useful for small servers as it requires a certain volume of messages before it shows any info. A volume you won't reach from a personal server.
I don't have time to reply at length on a touchscreen but this topic comes up every time someone links "how to self-host email". Saying it's extremely tricky is exaggerating. Everyone agrees it takes more work than it does to create a gmail account, but it's not that hard either.
I don't think we should dissuade people from doing it, especially if the fact that more people doing it means that it'll be easier next time because it'll be slightly more common. Many of us are in tech and the field is a small subset of the population. Even if it's a small amount of servers setup by us, that could make a noticeable on those working at bigcorps who write the hostile receivers.
Setting up is easy, maintaining and responding to all the blocks you may get is ongoing and simply cumbersome. That is what people are referring to when they say its tricky.
You may wake up one morning, and find your host is blocked through association (if you are lucky - some hosts will simply silently swallow your mail - i.e. it never arrives and you dont know about it). You can apply for removal in some cases (and wait for it to update before you resend), and in other cases find your hosts entire subnet is blocked, resulting in you having to set up another server on a different subnet in order to relay your email.
There are ways around a lot of this, but hosting on AWS/DO/Etc
not having your own assigned subnet, etc, will most definitely result in the above when you least want it to happen.
Of course SPF and DKIM were supposed to alleviate the need for these IP based blocks, but the reality is that they haven't changed a thing when it comes to sending email to large hosts.
> Setting up is easy, maintaining and responding to all the blocks you may get is ongoing and simply cumbersome.
The only real issues I had were with Gmail for a few months, until they got used to the IP address sending legit email. You have to instruct people to look in their spam folder (of which Gmail hides the existence, an evil thing in itself) until people marked like 10 emails as being non-spam over the course of a few months. Hardly had any trouble otherwise, and I don't even use DKIM nor a reverse DNS for all domains (only SPF because that was easy to do, and with 1 IP address I can do only 1 reverse DNS, and only v4 at that).
It helps that I've got a static address, and it might also help that XS4ALL (ISP) has a good abuse team that probably keeps the netblock quite clean. I don't know to what degree this affected my experience.
It's nice that you think all this stuff. The reality is that the technical side of SMTP service is the easy part (hasn't changed a hell of a lot in the last 20 years, anyhow) and the political side is a real burden (and it shifts with the wind).
You're not just setting up mail, you're dealing with an email cartel that does not have any desire for small players to exist. Note that I said "small players", not "spammers". Enough spam makes it to its destination that there are certainly palms being greased.
I also have a personal mail server that works fine, but I wouldn't recommend most people to do it, because getting spamholed is quite a serious liability.
I've been hosting my own personal mail server for about a year. Had to go through some arcane process to be "allowed" to send mails to Hotmail and my mails have been marked as spam once or twice by Gmail, but I don't usually have problems with delivery.
Running your own mail server does require investing some time in learning the software, the protocols and all the ways spam filters will sabotage your attempts to send legitimate, non-bulk e-mail, as well as a little time for maintenance, but it's really not as bad as people say, at least in my experience. It's not something you "just" do, but it is doable.
Could you share info about the arcane process with me ( cariaso@gmail.com ) . I manage delivery to most sites with no issue, but hotmail nearly always ends up in spam, and I'd love to try and address this.
I used the process described in this blog post to be able to deliver to Hotmail: https://blog.stickleback.dk/getting-off-hotmails-blocklist/ (not my blog). I got the same strange automated responses as this guy got, but I got off their blacklist after a couple of hours.
More generally:
- I made sure my server's IP addresses weren't on any blacklists.
- I host my mail server with a reputable hosting provider (RamNode). Stay clear of companies like OVH that have reputations for poor abuse handling.
- I have configured SPF, DKIM, and DMARC.
- I registered my server with https://www.dnswl.org/, a whitelist for legitimate mail servers.
- I don't send spam or other bulk e-mail, only personal correspondence.
I also tried sending e-mails to a couple of people I know who use Gmail. I asked them to mark my mails as 'not spam' if they went into their junk folders. I don't know how much this helps "globally" in the Gmail system, but at least it'll teach Gmail not to mark my mails as spam for those people.
I had used my current personal domain for e-mail for a couple of years but I paid a company to host my e-mail for me. It might have helped that my domain had already built up a good reputation in big mail companies' spam filters, but who knows... It definitely helps, however, that my domain wasn't brand new. Newly registered domains are often penalized by spam filters. A poor choice of TLD can also hurt delivery: I know that many mail admins penalize or out-right block many of the new gTLDs like .top and .download.
I am not discounting those who complain about deliverability, but so far I haven't had much trouble with my low-volume, personal mail server.
I used to (in the early 00's) manage email servers for a small company that among other things did web and email hosting for other small companies. It was annoying as hell and I will never do it again, I no longer have the patience for it.
Or you do nothing wrong on your server, but another user using your mail server ends up with a compromised password, or reuses the same username/password from elsewhere as their mail login. You wake up and see thousands of outgoing emails in your mail queue, all from the spammer, followed by other users who can't send outgoing emails.
You could attempt to limit the outgoing emails per account per hour, however if that's set too low then you end up with other users who can't send out emails to their "mailing lists" consisting of hundreds of contacts, instead of using a real mailing list to manage it.
Cool article. I personally use Virtualmin for this, it's very easy to set up, you can manage hosting and mail accounts for a bunch of different domains, easily do backups via S3, easy to set up SPF and DKIM, has a good web admin interface, easy to set up automatic updates, etc.
By the way, here's the obligatory HN-style critic of the format, not the content: the blog template in this is a little annoying, it fiddles with scrolling by making it really slow on Safari on Mac, and Reader mode doesn't work.
Virtualmin looks pretty cool, that looks like a great option.
Yeah thanks for the feedback, I need to get around to self-hosting my blog as well. I set up blogger a few years ago and I never got around to changing it.
The moment Yahoo, Microsoft or google decide that they don’t like you, you’re SOL.
Yahoo are the worst. If you try and deliver to them you get a deferral with an error message in your log with a URL. Then you have to open the link in the URL and fill in a form. They don’t have to accept the form and they ignore you for 3 months if it goes wrong.
This happens even if you’re not on an RBL and have set up DKIM and SPF properly.
Edit: you want to see the trouble we had to go to so we could run an SMTP server in AWS for outbound/abuse address inbound only and get that talking to Office 365 for internal use only. Two days of hell.
I've worked for a real estate webhost and a major cloud provider/isp. Both had nightmarish problems simply ensuring email delivery. It did not help that the real estate "bulk mail newsletters" were legitimate spam. At the ISP, we offered an open internal relay to customers for some reason, and thus were again legitimate spammers.
MS has 2 layers of spam blocking - one for Outlook.com, and a much stricter invisible layer for O365 with no support team.
Yeah I wouldn’t want to do it now. I worked for a major pollster about 10 years ago and they had monumental problems back then just sending invites out that people had actually signed up for.
I've been self-hosting for the past 4 years and this hasn't been my experience. Things like fixing DKIM/SPF eat an hour or two occasionally, but I've never encountered an issue where the time investment overtook the learning + other benefits - it's been hands-off 364 days of the year.
I've only seen one blocked send happen -- blocked by my grandma's @att.net account. Since it happens so infrequently and nobody uses @att.net, I just re-sent from a Hotmail account instead. No issues with the other major players. But for my use-case it's easy to mitigate and if the problem persists I can invest more time in it, but one recipient blocking me in 4 years isn't bad.
It's the only way to have ownership, which is is one of the benefits I really like - Google, Yahoo, etc. still get pieces of my personal email history because nobody else self-hosts or uses PGP, which is disappointing, but I prefer it over handing one player ownership the full history.
BTW, I'm running it on the same 512MB DigitalOcean droplet that I use to host my static sites (personal website, small product sites, etc), so it's basically free since I'd need to host those things anyway, which is nice. Needs some swap though.
Edit: Not saying these points are invalid. They're certainly valid, a service like Gmail _will_ be more reliable and easier. If you're blocked for some reason or have any other email probs, there's nobody else to fix it besides you.
You can try sending it to yourself from different email providers and check to see if it went to spam. For example I created a gmail and outlook account and sent emails to it from my hosted email server, once I had things set correctly I was able to validate the emails I was sending weren't going into spam.
How many different hosts are you going to create accounts on and send test emails to? Are you going to check the MX records for grandma's email to figure out who's hosting her email to test with that one?
Testing is not a one-time thing, the hosts keep changing their rules and if your ip is close to a spammer's that could change the treatment of your email as well. I sent email from my own domain address via an authenticated university SMTP server for years without hearing of any problems. Then this year, family members using Gmail started finding my messages in the spam folder. My best guess is Google started caring about the lack of an SPF record for the SMTP server, associating it with my domain but there's no way to know.
This is all way the advice for most people is: Don't run your own mail server, it's too much work and less reliable.
Certainty is difficult. I use the approach mentioned - I maintain email accounts at 3-4 different popular providers anyway. If I do ever notice something fishy I'll send to a couple of them.
Since I'm only using this for personal email, and that's at a lower volume vs. work email with different communication patterns, I think it's a little easier to detect failed sends (in other words, usually some response is expected, even if it's just "ha"/"cool"). But you're right, some might have been lost in the spam folder and never seen.
Uncertainty is the cost of gaining more ownership, and I don't want to downplay that. If I'm sending messages where I want to maintain as much personal ownership as possible, I use my personal mailserver and accept the risks. If I'm sending mail where I need higher certainty and don't care about ownership, I use other providers.
Another commenter said: "if you absolutely depend on the ability to send emails such that your recipients reliably get them, hosting your own email server is extremely tricky." I agree with that -- different communication has different needs/requirements, and a self-hosted mailserver gives some benefits that I really like and that you can't get any other way. I'm just saying for me and for my common uses, it doesn't feel like a constant headache + battle.
I wrote my own receive-only Haraka-based server. The web UI is pretty simple: x-y matrix of small blocks representing received emails with basic info (subject and from address). A column is the account. Clicking an email block brings up full text or html mime content (not rendered).
It’s setup to receive everything that’s sent to it, which means I occasionally have to delete rando spam. TLS is setup too. But it’s an interesting system because you can keep tabs on what exactly you are receiving per service (e.g. using instagram@mydomain.com), and maybe one day will tip me off to services giving away email addresses.
I've been signing up on sites using unique addresses for each one for about 10 years. I've had two addresses that eventually started receiving spam, one for a small local business and one for Adobe. I'm sure both were due to hacks of those sites, not the customer addresses being sold.
I have been running my own mail server for a number of years. I recently was forced to move to a new IP. I had to get my server listed on DNSWL, then send in manual requests to Microsoft, AOL, and Yahoo to not block my email because they all seem to deny by default.
I also ran through a couple of IPs with my provider before I found one that wasn't on any meaningful RBLs (my IP's on SpamGrouper, but that list is clearly run by an insane person and nobody seems to use it).
As other people point out, some mail providers are just complete assholes and will blackhole your mail with no indication to the sender or recipient that it happened.
I've been hosting my own email server for 10 years until a couple of weeks ago, when I switched everything to gsuite. During the sign up process there is a step claiming "email just got awesome". That's as close of a description of my experience so far as you can possibly get. Email just got awesome.
Just another data point: I've been running my own mail for more than 6 years and I've never had a problem with receivers.
Hosted on a cheap Strato VServer in Germany, I've never cared for the technical details, could not explain right now what DKIM and SPF are (and they're not configured), and only recently installed a self-signed SSL certificate in my Exim configuration to be able to use it with a Desktop client for submission (pretty sure outbound traffic still runs unencrypted).
Echoing to other's thoughts that hosting a server on your own would probably flag you in spam and just be a very tiresome process.
A reasonably good and cheap service I could not recommend enough is https://www.migadu.com . They allow you to use unlimited email domains, storage, addresses with the only limit being on total daily outgoing emails. The mini plan allows 100 outgoing emails a day which is more than sufficient for most of my purposes.
To me it falls under the same category as assembling my own computer. I can do it but to me it's not worth the trouble.
Between DKIM, DMARC, and SPF, security, backup strategy, the fact if you are an open relay for even a day a bot net will find you and get your IP blacklisted for life... or an ISP could just blacklist you because they saw other spam from your same subnet on a shared hosting provider...
Granted this article covers a lot of that (it talks about DKIM, DMARC, and SPF) I'm still counting this as one of the things I outsource.
Friends don't let friends host their own email. It's just not worth the hassle these days unless you're managing hundreds of accounts and it's a full time endeavour.
Even at the hundreds level, I'm sure there are people who'd rather outsource. It's not just email that these guys (Gmail, Outlook, Fastmail, etc) provide.
81 comments
[ 3.5 ms ] story [ 156 ms ] threadStep one to freedom was to use my own domain, redirected to my old account, but I'm seriously thinking about doing it all myself.
But keeping a whole server up to date, secure, etc.... that's a full time job. Is there a good solution?
I believe they stopped this recently.
But you could have always become a paying customer of Google.
However, they still scan and read email. People can't be inconvenienced. I tend to think this is worse than scanning for ads.
I also tend to think anyone concerned about privacy moved from Google long ago. After all, its terrifying when you allow a company to determine if perhaps your received/sent links are fake, or malware, maybe they should report that you like certain sites, or have a flight.. People lived the idea of the President maxing out his power with executive orders, until they didn't.. Life is ask about tradeoffs i suppose..
http://variety.com/2017/digital/news/google-gmail-ads-emails...
I recently signed up for ProtonMail.
https://protonmail.com/
I wanted a reasonably cheap and secure email service that allowed me to use custom domains. Proton Mail seemed like a good fit. No complaints so far.
One other option that I almost went with was Rackspace Webmail:
https://www.rackspace.com/en-gb/email-hosting/webmail
However, that $2 per user per month only applies if you buy a minimum of 5 user licences, so it worked out more expensive for me than the 5 Euros a month for ProtonMail Plus:
https://protonmail.com/pricing
On the other hand, if you absolutely depend on the ability to send emails such that your recipients reliably get them, hosting your own email server is extremely tricky. One could carefully go through the checklist of SPF, DKIM, ip blacklists, etc and emails will still be rejected by MS Hotmail, GMail, Yahoo, etc. Those giants are "black boxes" when it comes to their heuristics for rejecting incoming mail as spam. E.g. your hosted email server does nothing wrong but some other bad actor on your ip block sends spam which then makes MS/Google block you because of "guilty by association". Trying to debug your "sender reputation" is not easy.
In the 1990s, I hosted my own home email server over ISDN lines. These days, I have a million other things I'd rather do than babysit a personal email server with software updates, SpamAssassin lists, etc. I get the whole decentralized ethos but it's just not worth the effort for email servers.
Is this an urban legend that just keeps getting repeated?
Hashcash, Bitcoin's PoW game, was intended for use as an email attachment to create a real cost to spam without any micropayments.
"If you want to mail me, prove that you burned five seconds of CPU time."
That’s why
What is the motivation for the "guilty by association"? Because you can not verify who sent it.
With SPF and DKIM you can - that is the whole point, you know beyond any doubt that the machine supposed to send that mail actually did send it.
Now, if the person behind that machine is a spammer (or has been hijacked by a spammer) and sends out a lot of spam (and gets reported as such) then just nuke that sender. It is dead easy to do regardless of what IP it was sent from - each mail have identification credentials built in. It could not get any easier to do. The sender did all of the work for you by going through the extra effort of identifying itself.
Sometimes they claim to be real people with real e-mail addresses that they don’t actually own, and maybe that real person is you — in which case it is your reputation that gets permanently dinged.
Sometimes they claim to be fake people with fake e-mail addresses that don’t actually exist, in which case all you really know is the IP address they used to send the message. When they do this, they frequently arrange to get a large block of IP addresses they can use, and then cycle through them very quickly — maybe no more than five minutes per IP address. Anyone else who happens to be in that same range now gets their reputation dinged by association.
There’s a lot of tricks spammers use to try to hide their activity. And lots of ways to try to detect and reject that on the other end. It’s a never-ending arms race, and on the Internet it is easy to hide and hard to take out miscreants without causing excessive collateral damage.
And many big providers don’t care if they cause collateral damage.
Now, if SPF and DKIM has not been configured then fine. That is something else, resort to whatever technique you have to.
Yes, it'd be nice if they didn't do that, but small sites are and will be acceptable collateral damage to them.
One that has been given in the past is "Stop paying money to companies that host spammer websites and spammer email servers, or we'll block your email too"
(I don't particularly agree with it, but it was common to see that argument on news.admin.net-abuse.email
I don't know who you're even trying to convince. You're talking in hypotheticals and being extremely idealistic. How the email ecosystem works and how it actually works are two very different things. You cannot say "the reason must be something else!" as if - just because you can't make sense of the situation - it is not so.
Not whatever IP was used.
I'm not being idealistic when I say that I should be able to send an email over internet and have it delivered without someone banning me for reasons beyond my control.
That could be said about lots of things, but this is kind of remarkable in how much effort is being made to make something as bad as possible.
I don't know if anything like SPEWS is still around but they used to do large blocks. I still don't know if it was genuine attempt to reduce spam, or performance art, or god-tier trolling.
https://en.wikipedia.org/wiki/Spam_Prevention_Early_Warning_...
https://web-beta.archive.org/web/20030609200813/http://www.s...
A spam blocklist is not a court of law. As far as the mail providers are concerned, you're guilty until proven innocent. SPF and DKIM are no guarantee that you're not a spammer.
You can do "everything right" rolling your own email, but you are still relying on the receiving parties to do their job and implement sane spam prevention policies. This is far from what actually happens.
I lead development on a project that sends a lot of (legitimate, user-requested, transactional) email to many mailboxes belonging to a certain government department. I used Amazon SES as they have a pretty good track record. The receiving mail servers were configured to use a few overzealous blacklists that were causing my email to be silently swallowed (and reported as delivered). This was based on an IP black-level ban. Simply Googling the names of these blacklists showed report after report after report of them being completely useless, with many seasoned sysadmins saying that nobody worth their salt should pay any attention to them.
This was extremely hard to troubleshoot as this department was not a direct client and was of limited helpfulness. It became apparent that using SES's newly-launched dedicated IP functionality wouldn't have helped. I made full use of my AWS Business Support, the multiple people I talked to made it clear that SES put a great deal of effort into ensuring their IP ranges are "clean" - but there are some spam lists that do not play ball. The most important takeaway was that I was doing everything right. I was using a reputable mail provider who have very stringent spam prevention controls. I utilised SPF and DKIM. The content of my mail shouldn't have (and didn't) trigger a content-based spam detector. It was because the people on the other side weren't doing their job to an acceptable standard. When I was eventually able to open a channel of contact with the receiving department, they confirmed it was an IP range-based ban.
The fact that I was doing everything in my power to do the right thing had no effect on the situation. I still wasn't getting mail to customers. Saying "darm, these sysadmins really aren't doing their job properly!" had no effect on the situation at all.
We still live in the age of almost-but-not-quite-plug-and-play email solutions like Exchange that can be configured by people that probably shouldn't be configuring mail servers. For all I know, there was a list of blacklists on some Hosted Outlook control panel and the person in charge ticked all of them because more is better. As long as this is the reality I live in, I'm going to do all I can to play ball and succumb to the black magic. I can imagine my experience would've been a lot worse if I had been operating a mail server out of a dirtier IP range (generic cloud hosting provider, retail ISP customer IP range.etc). If I recall correctly, AWS severely limits sending mail from EC2 instances because they don't want their IP addresses / IP ranges to end up on mail blacklists.
Also, it isn't in said idiot's interest to deliver your email properly. They would rather fart around with machine learning and blame whatever happens on incomplete training data...
[1] https://postmaster.live.com/snds/
[2] https://postmaster.google.com/
I don't think we should dissuade people from doing it, especially if the fact that more people doing it means that it'll be easier next time because it'll be slightly more common. Many of us are in tech and the field is a small subset of the population. Even if it's a small amount of servers setup by us, that could make a noticeable on those working at bigcorps who write the hostile receivers.
Setting up is easy, maintaining and responding to all the blocks you may get is ongoing and simply cumbersome. That is what people are referring to when they say its tricky.
You may wake up one morning, and find your host is blocked through association (if you are lucky - some hosts will simply silently swallow your mail - i.e. it never arrives and you dont know about it). You can apply for removal in some cases (and wait for it to update before you resend), and in other cases find your hosts entire subnet is blocked, resulting in you having to set up another server on a different subnet in order to relay your email.
There are ways around a lot of this, but hosting on AWS/DO/Etc not having your own assigned subnet, etc, will most definitely result in the above when you least want it to happen.
Of course SPF and DKIM were supposed to alleviate the need for these IP based blocks, but the reality is that they haven't changed a thing when it comes to sending email to large hosts.
Thats why we say its tricky.
The only real issues I had were with Gmail for a few months, until they got used to the IP address sending legit email. You have to instruct people to look in their spam folder (of which Gmail hides the existence, an evil thing in itself) until people marked like 10 emails as being non-spam over the course of a few months. Hardly had any trouble otherwise, and I don't even use DKIM nor a reverse DNS for all domains (only SPF because that was easy to do, and with 1 IP address I can do only 1 reverse DNS, and only v4 at that).
It helps that I've got a static address, and it might also help that XS4ALL (ISP) has a good abuse team that probably keeps the netblock quite clean. I don't know to what degree this affected my experience.
You're not just setting up mail, you're dealing with an email cartel that does not have any desire for small players to exist. Note that I said "small players", not "spammers". Enough spam makes it to its destination that there are certainly palms being greased.
No need to get snarky. I do run a personal mail server myself so I'm reasonably sure I know what reality is like.
I also have a personal mail server that works fine, but I wouldn't recommend most people to do it, because getting spamholed is quite a serious liability.
Running your own mail server does require investing some time in learning the software, the protocols and all the ways spam filters will sabotage your attempts to send legitimate, non-bulk e-mail, as well as a little time for maintenance, but it's really not as bad as people say, at least in my experience. It's not something you "just" do, but it is doable.
More generally:
- I made sure my server's IP addresses weren't on any blacklists.
- I host my mail server with a reputable hosting provider (RamNode). Stay clear of companies like OVH that have reputations for poor abuse handling.
- I have configured SPF, DKIM, and DMARC.
- I registered my server with https://www.dnswl.org/, a whitelist for legitimate mail servers.
- I don't send spam or other bulk e-mail, only personal correspondence.
I also tried sending e-mails to a couple of people I know who use Gmail. I asked them to mark my mails as 'not spam' if they went into their junk folders. I don't know how much this helps "globally" in the Gmail system, but at least it'll teach Gmail not to mark my mails as spam for those people.
I had used my current personal domain for e-mail for a couple of years but I paid a company to host my e-mail for me. It might have helped that my domain had already built up a good reputation in big mail companies' spam filters, but who knows... It definitely helps, however, that my domain wasn't brand new. Newly registered domains are often penalized by spam filters. A poor choice of TLD can also hurt delivery: I know that many mail admins penalize or out-right block many of the new gTLDs like .top and .download.
I am not discounting those who complain about deliverability, but so far I haven't had much trouble with my low-volume, personal mail server.
You could attempt to limit the outgoing emails per account per hour, however if that's set too low then you end up with other users who can't send out emails to their "mailing lists" consisting of hundreds of contacts, instead of using a real mailing list to manage it.
The effort just isn't worth it.
By the way, here's the obligatory HN-style critic of the format, not the content: the blog template in this is a little annoying, it fiddles with scrolling by making it really slow on Safari on Mac, and Reader mode doesn't work.
Yeah thanks for the feedback, I need to get around to self-hosting my blog as well. I set up blogger a few years ago and I never got around to changing it.
The moment Yahoo, Microsoft or google decide that they don’t like you, you’re SOL.
Yahoo are the worst. If you try and deliver to them you get a deferral with an error message in your log with a URL. Then you have to open the link in the URL and fill in a form. They don’t have to accept the form and they ignore you for 3 months if it goes wrong.
This happens even if you’re not on an RBL and have set up DKIM and SPF properly.
Edit: you want to see the trouble we had to go to so we could run an SMTP server in AWS for outbound/abuse address inbound only and get that talking to Office 365 for internal use only. Two days of hell.
MS has 2 layers of spam blocking - one for Outlook.com, and a much stricter invisible layer for O365 with no support team.
I've only seen one blocked send happen -- blocked by my grandma's @att.net account. Since it happens so infrequently and nobody uses @att.net, I just re-sent from a Hotmail account instead. No issues with the other major players. But for my use-case it's easy to mitigate and if the problem persists I can invest more time in it, but one recipient blocking me in 4 years isn't bad.
It's the only way to have ownership, which is is one of the benefits I really like - Google, Yahoo, etc. still get pieces of my personal email history because nobody else self-hosts or uses PGP, which is disappointing, but I prefer it over handing one player ownership the full history.
BTW, I'm running it on the same 512MB DigitalOcean droplet that I use to host my static sites (personal website, small product sites, etc), so it's basically free since I'd need to host those things anyway, which is nice. Needs some swap though.
Edit: Not saying these points are invalid. They're certainly valid, a service like Gmail _will_ be more reliable and easier. If you're blocked for some reason or have any other email probs, there's nobody else to fix it besides you.
BTW, AT&T's customer email is now hosted by Yahoo.
Testing is not a one-time thing, the hosts keep changing their rules and if your ip is close to a spammer's that could change the treatment of your email as well. I sent email from my own domain address via an authenticated university SMTP server for years without hearing of any problems. Then this year, family members using Gmail started finding my messages in the spam folder. My best guess is Google started caring about the lack of an SPF record for the SMTP server, associating it with my domain but there's no way to know.
This is all way the advice for most people is: Don't run your own mail server, it's too much work and less reliable.
Since I'm only using this for personal email, and that's at a lower volume vs. work email with different communication patterns, I think it's a little easier to detect failed sends (in other words, usually some response is expected, even if it's just "ha"/"cool"). But you're right, some might have been lost in the spam folder and never seen.
Uncertainty is the cost of gaining more ownership, and I don't want to downplay that. If I'm sending messages where I want to maintain as much personal ownership as possible, I use my personal mailserver and accept the risks. If I'm sending mail where I need higher certainty and don't care about ownership, I use other providers.
Another commenter said: "if you absolutely depend on the ability to send emails such that your recipients reliably get them, hosting your own email server is extremely tricky." I agree with that -- different communication has different needs/requirements, and a self-hosted mailserver gives some benefits that I really like and that you can't get any other way. I'm just saying for me and for my common uses, it doesn't feel like a constant headache + battle.
It’s setup to receive everything that’s sent to it, which means I occasionally have to delete rando spam. TLS is setup too. But it’s an interesting system because you can keep tabs on what exactly you are receiving per service (e.g. using instagram@mydomain.com), and maybe one day will tip me off to services giving away email addresses.
Otherwise I’m using gmail for personal sending.
I also ran through a couple of IPs with my provider before I found one that wasn't on any meaningful RBLs (my IP's on SpamGrouper, but that list is clearly run by an insane person and nobody seems to use it).
As other people point out, some mail providers are just complete assholes and will blackhole your mail with no indication to the sender or recipient that it happened.
Hosted on a cheap Strato VServer in Germany, I've never cared for the technical details, could not explain right now what DKIM and SPF are (and they're not configured), and only recently installed a self-signed SSL certificate in my Exim configuration to be able to use it with a Desktop client for submission (pretty sure outbound traffic still runs unencrypted).
A reasonably good and cheap service I could not recommend enough is https://www.migadu.com . They allow you to use unlimited email domains, storage, addresses with the only limit being on total daily outgoing emails. The mini plan allows 100 outgoing emails a day which is more than sufficient for most of my purposes.
We used this to setup a multi-domain email server. Works reliably and fast at both sending and receiving mail.
To me it falls under the same category as assembling my own computer. I can do it but to me it's not worth the trouble.
Between DKIM, DMARC, and SPF, security, backup strategy, the fact if you are an open relay for even a day a bot net will find you and get your IP blacklisted for life... or an ISP could just blacklist you because they saw other spam from your same subnet on a shared hosting provider...
Granted this article covers a lot of that (it talks about DKIM, DMARC, and SPF) I'm still counting this as one of the things I outsource.
Even at the hundreds level, I'm sure there are people who'd rather outsource. It's not just email that these guys (Gmail, Outlook, Fastmail, etc) provide.