The wild west days of email, with bang paths and "store and pray" delivery systems, those were fun. By the time `sendmail.cf` hacking was no longer a necessary skill, email had become industrialized. Today, why would you even want to try routing internet email through an RBBS net to WWIV net to some hackers custom Amiga board?
SMTP was the Ford Model T of electronic messaging. It slaughtered the previous visions of what the field needed to be. We can look back fondly at the older ideas and even re-implement their insights now, but the lessons of the market are written in big bold letters now.
I've got a personal email server and an old gmail address from gmail-beta days.
I've never used gmail itself (that model doesn't fit my mind), but O do use that Google account for some minor stuff. Unfortunazely, I've repeatedly gotten email targetted at someone else having same first name initial and same last name on gmail (address is in the form of FLastname@gmail.com). I've usually been able to get through to those people to stop them and to get them to reach their targets, but in the last 24 months, a lady from Michigan is repeatedly giving out my email address for everything (I've got covid appts, doctor appts, movie tickets, responses to home buying inquires...). I have no idea how to stop this: this would've never happened with any provider that's not owning like 60% of the market.
I am constantly annoyed and I've considered both stopping mail forwarding from this account to mine (but then I might miss that YT premium notification) and I've tried reaching out to many humans on the other side. But I've so far resisted the urge to cancel those movie tickets or vaccination appts, but things just keep coming in.
I can't imagine how are people not overwhelmed by wrongly targetted email: there's more of it than spam I get on my personal server, so spam filtering would definitely not move me towards gmail. And actual spam also gets through on gmail!
I don’t think you can stop it. I have to imagine that this woman doesn’t have an email address and just gives out what she thought was a fake one, since so many things require it. Otherwise, how could she not notice she never receives anything?
> Otherwise, how could she not notice she never receives anything?
She may be getting it at f.lastname@gmail.com.
I signed up for Gmail the first or second day it came out with first.last@gmail.com, but hardly ever log(ged) in over the years. A little while ago I did go into it and noticed getting a bunch of message to firstlast@gmail.com.
Now Gmail allows for 'customized' addresses in that you can drop a period anywhere and it will still go to your address. But this raised the risk of one person signing up with first.last and another person signing up with firstlast. Supposedly this is prevented, but I think that they did not catch this situation in the early days of the service, and so a bunch of OG accounts have cross-contamination.
Someone signed up for an Amex card using my wife's gmail address. This was years ago, and she still gets the Amex emails regularly (like, between daily and weekly). It's incredible both that the Amex customer doesn't notice, and also that Amex has absolutely no method whatsoever for a non-customer to contact them and fix the problem.
I have a very short gmail address from the beta days and get probably 5-10 emails a day addressed to accounts that absolutely aren’t mine, Eg my username then a dot and another word, or my username spelled differently. I don’t know why google thought fuzzy-matching emails was a smart idea, but it really isn’t. I’ve gotten a lot of very private information and direct login links to a lot of stuff. I also get countless people putting my email address as theirs, to the point where I’ve given up trying to fix it and just delete it instantly.
To even use that gmail address I need to basically whitelist senders and filter them into folders and ignore the inbox completely.
Gmail, like so many of Googles services these days, is an absolute mess. Features no one asked for, blatant spam that gets through their checks while your actual emails go to the spam folder, and a constantly degrading UI that seems to be an experiment in how much you can annoy the user.
>Unfortunazely, I've repeatedly gotten email targetted at someone else having same first name initial and same last name on gmail (address is in the form of FLastname@gmail.com).
That's a user problem, not a technical one. (there is, of course, an XKCD for that).
I've got a popular initials/surname combo and I have a number of doppelgängers giving it out. The one I feel most sorry for is the trumpist and his scary NRA/pro-gun mail. It's really fierce stuff, I'm glad it's going into my spam folder rather than in front of a real human!
I also get an endless amount of email for a few different people with the same name as me at my Gmail from 2005. I used to try to deal with it but now it's just amusing, especially the photos, family chain emails etc. My name alike is Canadian so it's a little glimpse across the border
What you've described is common among many first adopters. I know someone who has a common first name and last name combo, signed up for gmail in mid-2000s and use the firstname+lastname@gmail.com.
He got many "weird" messages over the years from messages addressed to a religious minister, to a pro-gun individual and to NSFW account that he didn't sign up for.
Wait, is email spam still a thing? I only get a few spam a day and Spamassassin easily identifies them.
I had assumed that the spammers had moved off to other mediums. Either that or they are specifically targeting big servers like Gmail and are leaving the smaller servers, with their varied (artisanal) anti-spam approaches alone.
It never stopped. What is however more annoying are all the "marketing messages" from all the companies you ever bought a single thing from in your life. It's incredible how often some companies spam you with this (often more than once a day). Really not sure what they are trying to achieve, but for me it's the resolution never to buy anything from them again as I filter their whole domain permanently.
Those I usually forward to their legal department with a little extra bit that says that I have stopped doing business with them on account of their UCE.
A lot of it is luck. You can go years not seeing much of the really malicious spam and then one day you're on the list and you'll find yourself being bombarded with stuff Spamassassin doesn't touch.
Especially if your organization is a potential financial target.
If anything, it's worse than ever. The favored technique these days is "snowshoe" spamming, where $SPAMMER sends a trickle of spam from a large number of IP addresses. About the only way to stop it is to block the /24, and then they just move on to the next block of IPs they want to ruin. RBLs like Spamhaus are helpful, but there will always be a few spammer IPs that haven't yet been listed. SpamAssassin can be handy, but it's a pain and there's a lot of rope you can hang yourself with. Blocking certain TLDs outright (.cam is a good candidate not just because of this, but also the phishing potential) can be an option.
Even a service as big as Microsoft's has pretty bad spamfiltering though. It's slightly better than an untuned SpamAssassin config but it's really not a lot better.
Another reason could be that I live in a country with very strict anti-spam and privacy laws (Canada). I have always assumed that spammers wouldn't care but who knows...
Someone should try "artisanal email server" using cloudron or yunohost !
the bigger problem is that "authoritative" email monopolies such as Gmail, 365 and the other big ones arbitrarily define and impose what is a legit email server or not and even with better score than gmail an "artisanal" email server can suffer from being classified into spam by the big tech players just because they can and will do anything to maintain their monopoly.
This is exactly it. Email is now just another way to squeeze companies and private individuals alike instead of a cheap, secure and free way to communicate. Peer-to-peer email was worth having, in spite of the downsides.
I’ve seen a lot of small businesses go from $50 / year to $500+ / year. And from their point of view all they get is a bunch of nagging about 2FA and a much bigger target on their back when it comes to phishing.
I'm running an email server and I can tell you that this is by and large not the case.
If you put some decent effort into making sure that you don't send spam, try to monitor if anyone thinks you send spam and react when someone complains that you send spam (and stop it), it works.
In my experience people telling these stories often do send spam, but they don't believe they do. ("It's not spam, it's a Newsletter. No, it has no unsubscribe link. These are people that agreed to be put on the newsletter by clicking on some ToS they never read, and they can unsubscribe by some arcane mechanism that we will make as complicated as we can. But we're definitely not spammers.")
To most, including Gmail, it's actually no problem with DMARC in my experience too.
However, one of my servers IPs is on a Microsoft blacklist since many years now.
It sends <10 messages / day. I've tried every unlist form I could find, even called MS but it does not get taken of that list and they "won't disclose why".
I'm routing SMTP to MS via another relay now :)
relay_passwd.db: # if necessary / not authenticated by IP
relay.server.tld user:pass
The relay can/should rewrite the Return-Path to pass SPF. It's no problem for DMARC as the DKIM signature added by the initial server still authenticates it.
It requires manually adding domains of custom 365 installations to the list - at this size I do this manually, but should probably be automated "on bounce" or maybe even by a smart rule based on the MX record.
In Exim4 it's also possible to conditionally rewrite based on for example the recipient domain.
Please stop spreading falsehoods. If you were able to somehow get your own email server to deliver email to Gmail and Outlook, great, good for you - but stop pretending that anybody can do it.
> If you were able to somehow get your own email server to deliver email to Gmail and Outlook, great, good for you - but stop pretending that anybody can do it.
Yes, that's probably true. I've been running my own server for 20 years now, and I guess that in itself helps with getting my mail delivered (apart from t-online, but who cares about them). At some time I also hosted some mailing lists, but I quickly abandoned that because that's a surefire way to get your IP blacklisted sooner or later. If you set up a completely new mail server, there probably is a lot of luck involved, and I wouldn't recommend it to anyone, at least not for your critical business mails. I pretty much keep doing it only out of nostalgia, it doesn't really make any sense otherwise.
Haha I have the same experience... I have given up trying to send emails to t-online, but every other email-provider accept emails from the server I manage. It sends a few thousand emails per day.
A few years ago we had problems, but then I realized some of the emails sent from our servers had non-ascii characters in headers (subject, from, to) which caused email-providers to distrust our server. Using encoded-words syntax ("=?UTF-8?B?" + BASE64(text) + "?=") fixed that problem:
I did run my own email server for 20+ years. As you may imagine, I had to learn a thing or two about DMARC, DKIM and SPF, but spread over the years it is not a big investment to make.
Most of the time, delivery problems were of my own creation. Like running out of disk space or accidentally disabling TLS.
Once in a while, Microsoft would start swallowing emails or Google would push everyone to use DMARC.
But overall, the experience has been very pleasant. I host my mails, I own my data. I am not shy of using Google, but my work is not defined by their whims. When Google tells me I ran out of space in my account I just delete stuff because I have copies of everything outside of Google infrastructure.
My mail server running on DigitalOcean has been relatively trouble free over the last 9 years. It runs docker-mailserver and is used by me and a dozen employees of my various small businesses.
It requires some effort to maintain and understand, and I’ve had a few deliverability issues over the years but they are generally with niche providers. I’ve never had trouble sending mail to the big providers.
Every time I read comments about the impracticality of self-hosted email, I scratch my head. Maybe I’ve just been lucky.
I think DO is really good about policing their IP space. When I signed up for the Microsoft JMRP [0], DO was already a contact of record for the IP I was using. I just appended myself to the list to get any abuse reports as well.
>Every time I read comments about the impracticality of self-hosted email, I scratch my head. Maybe I’ve just been lucky.
I feel the same. I've had one or two hiccups but smooth sailing for the most part. I'm also happy to provide receipts that show how the recipient's mail server is responding when I send the emails. It's a powerful tool to say, "your mail provider is misbehaving, look!" They will wonder how many people tried to send them email that didn't get to them.
That's what I've been using for some years and it's never been a problem for me. You are right that you have to have at least a basic understanding of how a mail server works, and there is some configuration to know about. But I think of all the things I host myself, docker-mailserver is the least cumbersome and among the most reliable.
I setup and have been running an email server for around 20 domains for over a decade. There have been no issues delivering to Gmail or Outlook, AOL, or Hotmail. There was some work I did initiallity to remove our IP addresses from blackhole lists, which had resulted from whatever the prior owners of the addresses had done. That was, however, minor and didn't take much time. Similarly, setting up DKIM, SPF, and the like were necessary and ugly to do, but they didn't take much time.
Lots of sister comments here saying that they've been running a mail server for X amount of years, where X is a rather large number. That will obviously come with some reputation for your mail server, reducing the curve of being classified as down. I would be interested in hearing from someone who tried to setup a new mail server in the last 1-2 years who was able to run it without a hitch.
According to AWS, most mail servers will not even count statistics for low-volume senders. If that is true, then it won't matter whether your personal email server has been up for 20 years or 1 year, it won't have any reputation.
This matches my exprience. I switched recently after 10+ years, and was cautious that this might be a problem but it hasn't been at all. I think it has more to do with the choice of ISP.
Interesting, I misinterpreted what was being said!
I'm doubtful a default block would work, as that would even penalise the 'big boys' of email when they make basic network changes and piss off existing customers of both sender and receiver... Its easier and logical to conclude something without reputation yet is therefore sending too few mails to be useful to a spammer.
I've had good experiences with smaller ISPs (currently Mythic Beasts). In contrast, OVH was a poor experience.
I find that reputation (beyond the known "block-lists") appears more likely being tracked for the whole AS number, therefore a lot more to do with your "neighbours" than anything else.
What matters most is if the IP address they issue you has been blacklisted for spamming. DigitalOcean is fine but you need to check the IP address before you do the work of building a mail server. Some of their IPs are on a lot of blacklists.
If it's only on a very few you need to look into who's blacklisting it. There are some that offer a way to get delisted and make it easy, there are others that block pretty much every IP address DigitalOcean has (or large ranges of them) and they won't de-list anything within them. Many of those blacklists are managed overseas and not used much in the U.S.
No matter the ISP you should check the IP address they issue for a VPS before you build the email server.
Yeah, I have that problem with gmail, I had a test account with a weak password get exploited a few years ago. Now for any new gmail address I want to send to seems to endup in spam. The problem here is there is no getting out of jail easily for low volume email users.
My personal gmail account is full of spam and emails I do want from email lists end up the spam folder randomly.
In my case x>10y for a personal server, but that reputation got ruined when some test email account I had created with a weak password and forgotten, got breached and some spammers started sending spam. My mail server (smartermail) notified me within an hour of the abnormal number of emails and I disabled the account immediately. But that was it for the reputation of that IP. Fortunately I could switch to a spare, clean IP.
That being said, now I monitor and auto-ban failed authentication attempts to smtp/imap (among others) and running the service is fairly low maintenance.
But the morale of the story is that you are only one weak password from one of your users away from your mail server getting blacklisted as a spam server. So while I think it is fairly easy to run a personal server, running one for a small organisation is another matter.
I've been running my own email server for around 7-8 years and just setup a new email server on a DigitalOcean vps earlier this year using "Mail-in-a-Box".
That's about as easy as it gets but it still requires some work and you need to check the IP address DigitalOcean issues to see if it's blacklisted before you set it up.
Google makes it easy to get whitelisted. Microsoft email services (Hotmail/Outlook) are a pain though. I tried to get through their process but got nowhere. Other services I had to submit a request to get de-listed. So it does take awhile to go through all that.
Still, I prefer that to hitching that wagon to a 3rd party provider like Google, or any other.
Before I set mine up the 1st time I'd been screwed a few times by 3rd party providers. The last one, I can't recall which, but it was either "MailChimp" or whomever bought them, that I'd configured an app to use and almost as soon as I'd released it they announce they'd been acquired and I would have to use the new services APIs, and of course they cost more, and their services were geared towards mass mailing, and that's not what my apps do, and their API sucked for my needs.
It was about 12 years of dealing with 3rd party bullshit that motivated me to set up my own email server.
If you just want to fiddle around with one to get a feel for it Mail-in-a-Box is a good place to get started: https://mailinabox.email/
I've been running my own mail server since mid '00s. Initially hosted with one of West coast Canadian colos and subsequently moved to an EU colo. Had some deliverability issues with Outlook and Yahoo, but these were episodic and rare even though I set up DKIM only last year and have been running with just SPF and DNS/PTR before that.
I know at least a dozen of others with similar setups and timelines. But we all use dedicated colo'ed boxes on IPs from clean netblocks that weren't previously used for shared hosting. I strongly suspect that attempting to run a mailserver on Digital Ocean, OVH, 1and1 and similar mass-hosting providers will not go well. Just like it will be an uphill battle to run it on a residental IP.
What did I generalize, exactly? Parent poster was claiming that anybody can set up a mail server with good deliverability - that's a generalization. I said good for them (acknowledging they managed to make it work) and said that I also tried and couldn't make it work - therefore, clearly not everybody can make it work. Did I not argue against generalization there?
Exactly. I’ve been running my own email server for over a decade not because I think I’m artisanal, but for practical reasons. But I don’t send out spam or “newletters”.
I've run a hobby website for about 15 years that does not even have a newsletter of any kind, and includes "stop sending me emails" in each transactional email (all users are double opt-in verified), and password resets are still not delivered half the time to gmail addresses.
I've run an email server for 10 years now and by and large this is the case. I am the only person that uses my domain/IP/mailserver. I know it doesn't send spam. I've still been blocked by MS Office 365, marked as spam by google, etc, every few years. It's quite a hassle to get unblocked involving lots of lying about having a Microsoft account or the like to tech support till you get to techs who actually know what a mailserver is.
Frankly, I'm shocked you've never been arbitrarily blocked and I find your insinuations offensive.
The last time I was getting blocked it was the solarwinds fiasco where their internal mail tunneling/forwarding and filtering setup broke all DKIM and suddenly solarwinds users like NOAA.gov were rejecting me and adding me to naughty lists. There was no fallout for the megacorps and their broken setups. There was only damage to independent mailserver operators doing the right thing.
I've run my own email for decades and I've designed and run some pretty big commercial installations.
As a small provider, you run the risk of existing in a netblock used by other people sending spam. A small co-op I ran encountered this problem once. They were operating on the cheap and while they weren't sending spam their neighbors had been.
Even as a large provider at a billion dollar company, figuring out delivery issues is a huge pain and generally not worth it. There are unofficial professional postmaster meetups around the bay and these can be helpful in getting escalation contacts to fix issues, but even with entire teams of people dedicated it's a lot to handle and is usually worthwhile to outsource the work to other companies who already have these types of relationships established.
I think the "decent effort" part is the key thing. We had to change our mail routing temporarily earlier in the year (after having sent via Office 365 for multiple years) and keeping on top of emails that were being blocked was a non-trivial amount of effort (and stress) for a period of time.
Unlike the person to you're replying to we had no issues with Google or Microsoft (once we did the requisite things) - it was Yahoo (and the people they provide email for) and then multiple mid-size organisations who used IP based block lists. At one point our mails were being rejected by our local NHS trust, the London Fire Brigade and a mental health agency we make referrals to. None of this was complicated to resolve but it was energy that could have been better used elsewhere.
I'm not usually part of the "let's go cloud without doing any cost-benefit analysis" movement but with email delivery I was happy when we could go back to routing via Office365 again. If a recipient decides to ban Microsoft's IPs that's usually going to be a bigger problem for them than me.
You need control of the entire netblock you send email from. Everything was going smoothly for me for 7 years until the entire Digital Ocean netblock my static IP was in landed in a permanent blacklist due to enough of the other IPs in that block having repeated complaints. I don't remember the mailing blacklist it was on but unblocking that single IP required the netblock owner (Digital Ocean) contacting the blacklist provider directly
this is why persons self-hosting email servers are much more likely to have success using a small to medium sized, trusted local ISP where you can establish a relationship with the persons who run the ASN. And determine for certain that the ipv4 /24 your mail server's /32 is contained within does not contain random other $5 to $30/month people buying VPS/VMs/low-budget-dedicated-servers with credit cards.
If you can have a high degree of confidence that no outgoing smtp spam traffic has ever been emitted from any of the other IPs adjacent to where you're hosted, the opaque blacklists of the big mail receiving providers (gmail, etc) are much less likely to consider your legit traffic as spam.
I deal with this every day. Personal fully controlled server. I don’t conduct business over this server, have only one email, a personal email, associated with it.
I've been running mailservers using free software for 20 years. I've run two for personal use, and several for groups like companies. In the old days, you could indeed throw up a server, and provided you don't spam, and you're not in a bad neighbourhood, outgoing mail would be accepted.
In more recent years, my experience has been that it takes time for a new mail sender to be acccepted; could be a year or two to build reputation. That's assuming you do everything right.
My personal mail, by the way, has been on the same domain since about 2001. I've quit running a mailserver now. My small ISP runs a setup that's basically what I would have built, so I use that; the support is excellent. But it's still on the same domain.
Last company I was at ran their mail on their ISPs mailserver. The ISP got taken over; service deteriorated, to the point it became unacceptable. So I built $EMPLOYER a mailserver; it took me longer than I predicted, because the bosses had all kinds of finicky requirements (don't they always) that I had to figure out how to provide after the fact. But that "artisanal" server beat the bejabers out of the ISP system; it was fast, reliable, and when anything went wrong I could fix it - which that ISP couldn't.
I think the vastly different experiences have a lot to do with the quality and scale of the ISP. Best results with small, good quality ISPs.
Also running my own servers for personal and business, and working well.
But when we tried to use one of the large VM providers the experience was much less reliable. Despite ensuring the IP was not on the various block lists etc. mails would be accepted and silently discarded by recipients ISPs, perhaps due to the level of abuse of these IP ranges.
In my experience people telling these stories often do send spam, but they don't believe they do. ("It's not spam, it's a Newsletter. No, it has no unsubscribe link. These are people that agreed to be put on the newsletter by clicking on some ToS they never read, and they can unsubscribe by some arcane mechanism that we will make as complicated as we can. But we're definitely not spammers.")
Yes. I do get that impression from most complainers.
I send from my own domains, and if I sent it, I wrote and addressed it personally.
I hear this all the time, but I question how true it is. I've been running my own mail servers for decades, and I've never had any problems with sending or receiving mail. I suspect anyone who properly configures their server will be fine.
The problem is delivery problems are almost undetectable.
If I send an email to a corporation’s customer support, or to a distant relation, or to an open source mailing list, and I don’t get a reply, it could be a delivery problem - or it could just be that they didn’t decide to reply.
For corporate support, that’s totally on them. Checking the spam folder is customer support. The “oh sorry your email went to spam!” is one of the few times I express dissatisfaction to a company. It’s not my responsibility to make sure my email doesn’t go to their spam folder. Not when I’ve taken all the right steps to make sure my emails are not marked as spam. If you have customers, you have to check your spam folder! It’s not foolproof.
You seem to not realize many email providers just drop emails (often after accepting it) instead of putting it in spam folders. So even your suggestion is of no help in that situation.
> I hear this all the time, but I question how true it is. I've been running my own mail servers for decades, and I've never had any problems with sending or receiving mail. I suspect anyone who properly configures their server will be fine.
At work I ran email servers professionally and with good deliverability for years. My own email server was arguably longer lived than those at work, just much lower volumes. IP block was clean, DKIM, SPF, rDNS, etc. all setup correctly.
I thought I had no deliverability issues. I interacted with mailing lists regularly, the odd email to friends and family and I was firmly in your camp until I had to deal with a death in the family.
I think this was shortly after Microsoft BPOS became Office365. It became very very clear very very rapidly that to certain orgs I just wasn't hitting the inbox. And there was jack shit I could do about it. That was the end of my mail server, and it's certainly got worse over time.
Hmm, guess it's time for a counter-attack : "Sorry, but it's too much hassle to send e-mail to gmail/outlook, please use another provider if you want to communicate." ?
I post on a mailing list where one member has configured his server to reject all emails from Gmail. Inevitably we end up getting messages sent to the list which begin, "Direct emails to <guy> are being rejected, so I'm sending it via the list, sorry for the noise!"
The unspoken "you silly prick" gets louder every time this happens.
And my retort is “that you know of”. If mine isn’t getting to you, who else’s isn’t? (For businesspeople) how much business are you losing because Google isn’t letting mail through? It’s one thing if the server is declining email and telling you why. It’s another thing to silently hide email.
Having run my own email for the last two years, this is the number one problem I encounter. Somehow every official step published in terms of standards for securing email servers is not enough to appease large provider such that they’ll deliver your mail and not relegate it to spam.
Every time my artisanal mails went to spam it was an overzealous corporate spam filter (mostly for mails with attachments). Never from one of the big hosts.
Sending to Gmail/Workspace and O365 is by far the easiest case for me. It's the random enterprise email servers that don't like new gTLDs like '.xyz' who cause the most headache.
The only problem I've had are with small players. You can't seem to reach anybody in charge of configuring and they do stupid shit that doesn't actually work.
The big players all have a process and followup within days.
My own experience from running a private e-mail server the past 5-6 years is that the problem more than anything else is garbage "e-mail gateway" products, like e.g. Cyren GlobalView and Proofpoint, that gets in the way.
There's a tendency to perma-reject e-mail coming from "not seen before" domains despite the e-mail passing FCrDNS + SPF + DKIM + DMARC validation, which makes it difficult for private e-mail server users to get through to people.
The suggestion that they will try muscle you out to maintain their monopoly is a bit alarmist. If you’re not sending spam, and your email infrastructure includes strong DMARC and SPF policies, then it’s unlikely that your reputation will be tarnished simply because it isn’t part of the Gmail or Exchange Online ecosystems.
I’d argue that the vast bulk of email is sent from dedicated providers like Sendgrid which are built on the same tech that might be found in any given ‘artisanal’ on-prem service.
>just because they can and will do anything to maintain their monopoly.
This is a popular opinion on HN but it doesn't seem at all inline with reality. Email isn't exactly a real money maker for anyone. And the amount of email spam and abuse is immense. Filtering out most unknown providers is unfortunately extremely effective. Almost all spam wiped out with a simple check.
Maybe the ideal solution would be to let you link your custom email domain with a google account so you can have your google account vouch for the legitimacy of your custom domain. But even then, some of the time your email server actually is just blasting out spam without you knowing it.
Email itself maybe is not a money maker, but my company just went to 365 and 90% of the justification to management is "were switching email providers". Microsoft and Google's small business offerings are inextricably dependent on email first.
I run my own email server and it's not so bad. gmail is obviously the most troublesome "peer" - if it wasn't for the fact I need to communicate with some gmail users regularly then I'd just cut gmail off. You have to set some headers and DNS stuff, having a static IP helps, and obviously not allowing that IP to send spam too. Largely it works fine.
I do too, gmail is not a problem at all, but I have had the same static IP on AWS for about a decade, reverse DNS is mandatory for deliverability. My problem with Gmail is it is difficult to filter SPAM coming from their servers.
My biggest problem with mail delivery is sending mail to Microsoft properties. I've had to resort to sending those messages via SES.
Open source webmail solutions suck, so now I'm paying fastmail and forwarding incoming messages there.
Also run my one server for years without too much trouble. Hotmail is the one giving me the most hassle - for some reason they periodically block my IP address for apparently belonging to an ASN they block … except it belongs to an entirely different ASN. My provider (RamNode) say they’ve been trying to get someone there for years to fix their system but didn’t help, so I just fill in their unblock form now and again.
If you have any neighbourgh on your block that has sent spam, then the entire IP block gets a bad reputation. I moved away from OVH because of this. I no longer get the issue )on Vultr) but still get 100% blocked by outlook.com and blocked by Gmail if I'm the one initiating the conversation (I can mail people once they've emailed me at least once)
Their email referenced AS3150 ... Here is what the (clearly frustrated) RamNode support person had to say last time I enquired:
AS3150 is NTT, a large backbone provider: https://bgp.he.net /AS3150
RamNode runs AS3842 and AS198203. We have contacted them before - they don't know why nor care why their system is raising issue with AS3150 in regard to emails from our network.
But this issue isn't exclusive to our network either, and no other major email provider blocks emails like this.
> … general security will not be as good as they have.
> Entirely "on premise" email is now an inferior thing for almost everyone.
I disagree on this one. Placing your email with a big player means that by definition, they have access to your mailbox (with sensitive stuff hopefully encrypted). To allow that you have to trust the big player and the countries where they reside.
They can drop you any time for political reasons, for dealing with a country that is considered an enemy of the host country of your provider. They may sell out your data.
You may still choose a big player, but understand how screwed you are.
Once you’ve sent or received email containing “sensitive stuff” you no longer control your data. Folks need to come to grips with the fact that email security is dead and hosting it yourself doesn’t fix this.
The author of the post should have included that in his post.
"Folks need to come to grips with the fact that email security is dead and hosting it yourself doesn’t fix this."
Sort of, but not entirely true ...
If you run your own mailserver then users of that mailserver can send and receive mail, to each other, without traversing a network. The mail never goes out on the net. That can be valuable/interesting.
This is true in both the webmail use-case (the text goes to the browser, ephemerally, encrypted with SSL) or the terminal/console (alpine) use-case (the text goes to the terminal, ephemerally, encrypted with SSH).
There's a certain cloud storage provider I know of whose internal / intra-company emails have never traversed the Internet ...
> (with DMARC signatures and other modern email practices)
DMARC does not provide signatures, DKIM does.
DMARC adds the DKIM 'alignment' requirement. Meaning that not just any DKIM signature will do, the public key (the DKIM DNS record) must be published under the administrative domain (the part after the '@' in the sender address).
DMARC also mandates SPF alignment (not that your should rely on SPF), meaning that the rfc5321.MailFrom and rfc5322.From address should be from the same administrative domain for the SPF to pass DMARC.
When either SPF or DKIM is aligned, you have a DMARC pass. Because SPF breaks with forwarding services, you shouldn't rely on it. DKIM + DMARC is the way to go.
Also funny that the author calls DMARC 'modern practice', since DMARC was introduced in March 2015, almost 7 years ago.
It's obvious when your domain (well, the IP really) is blacklisted. You either get a nasty error from the mail system saying it didn't go through or that the server was unreachable or what have you. You're always going to get some notice that it failed. In that case I contact the recipient a different way and say, "hey, you're mail server is broken, can you please fix it?" Usually this means them contacting their mail provider and asking why they are blocking mail from someone they need to communicate with. If it's someone like Google, they'll realize they can't actually get any support and think twice about using Google for mail.
A lot of people will say, "no they'll blame you for it not going through", but that's rare. Most people will be receptive to your insistence that you're trying to send them mail but their provider is in the wrong.
Now, if you mean blacklisting in the sense that the server has shadowbanned you and is sending back "221 OK", then, again, you have an affirmative defense: "hey, you're mail server said it accepted the mail, can you please check with your provider on what they did with it?"
In either case, this is actually not a good thing for the recipient's mail provider especially if they pay for that provider. "Why did you accept the email from the sender but not put it in my mailbox? Who else have you done that for? Why am I using you as my mail provider again?"
I have a circle of influence... about 100 people. All 100 of those people would switch mail providers if I asked them to. And I think a lot of people have a circle of influence around this size as well. So there actually is an amount of control over these bigger mail providers. They will be receptive to "I just told your customer to switch because you won't let me email them" especially if it's widespread. In exchange for that "power", I make sure my mail server is as clean as possible and quickly respond to any notices sent to my abuse@ address.
Also, a lot of software in the field has never made sense to me. I know what the parts do, but I couldn't tell you how to assemble it all. It all seems very old and seperate software more for historical reasons than anything else.
I ran mailinabox for a year or two, but eventually I just didn't want to maintain a piece of software I didn't understand where the documentation seemed actively hostile and presumptuous about me having read all the other parts. I'm sure the postfix docs make an okay reference, but understanding it as a whole, god no. I'd rather do Kubernetes from scratch.
Postfix and Dovecot are classic magic word projects - completely useless unless you can work out the magic words, and then they work fine.
The docs for magic word projects never to seem to prioritise essentials. So [obscure feature someone last used in 1984] gets equal billing with [essential fundamentals] and you have no idea which is which because - you haven't understood the docs yet.
I'm still running my own servers. I sorted out the spam issues, and they're basically zero maintenance now. But it certainly took a while, and a fair amount of copying other people's ideas of what a config file should look like, with plenty of trial and error.
Agree regarding learning the magic words. There''s probably no substitute to just diving in and setting up a server with a domain you can afford to get blacklisted for a few weeks while you make mistakes. But once you know the magic words, Dovecot's documentation is actually fairly decent these days: https://doc.dovecot.org/.
> I ran mailinabox for a year or two ... the documentation seemed actively hostile
Maintainer of Mail-in-a-Box here. I'm sorry you had that experience. Definitely was not the intention of the project to be hostile (but I can see how it might come off that way).
Hi Josh. I should've specified I meant the postfix documentation here. MiaB was wonderful, save for maybe skipping one Ubuntu LTS and leaving not quite a lot of time to migrate.
It is a wonderful world we live in. Everything from salary negotiations to love affairs is on someone else's servers, all in plaintext. Apart from email this also applies to Slack, which is a goldmine for keeping dossiers on developers.
This is one thing the authoritarians like Biden (Clipper Chip, Patriot Act) won't want to fix. There will be no law that companies with more than 100 employees must accept mail from individual servers (they would still have the correspondence anyway, but it would be a start). There will be no law that all mail must be encrypted.
Except of course that it isn't an artisanal choice, a very practical one that is made increasingly impossible by the few very large email providers that are left. It should be as simple as hosting a web server.
Speaking of which, how long before it won't be possible to host your own web server?
On another note: the biggest source of spam is gmail itself, and guess what, that makes it to my inbox just fine, because what could possibly be wrong with someone using google as their source. Spam was annoying but it was never an actual problem. The consolidation of the internet into a handful of players is a problem.
"Except of course that it isn't an artisanal choice, a very practical one that is made increasingly impossible by the few very large email providers that are left. It should be as simple as hosting a web server."
I don't get this one. How do large email providers make it difficult to host your own email?
I host my own email. It was a pain to setup so I try not to touch it since it is running fine.
Setting up email on your own server is just complicated unless you install server management software. I am not sure big email providers are to blame for this.
> How do large email providers make it difficult to host your own email?
By randomly marking your email as spam without any recourse. This may be because they blacklist your provider en bloc, your IP address or some subnet, because they feel like it, it's Tuesday or because their spam filters suck.
But it happens and it happens often enough that running a business in that way will cost you money, sometimes lots of it.
Business use of email tends to look a lot like spam and people mark it as such. An appointment reminder or notification that something just shipped is generally fine. Send out mass notification of your holiday sales and that’s going into someone’s spam folder.
Not so much a question of luck, sending out sipping notifications that for example include advertising is risky. Sending a high volume of appointment reminders for the same appointment is similarly problematic.
I don’t mean that’s the only way to trip up, there are a lot of unspoken self hosing email rules. Don’t use public data centers, don’t send news letters etc.
My email server is only used as a personal server for a few select friends and family. They absolutely do not send and have never sent anything that could remotely be considered spam. Everything in our setup is picture perfect (SPF, DKIM, DMARC, PTR records, etc). We still can't get email onto Microsoft's servers without it being marked as spam.
"By randomly marking your email as spam without any recourse."
Correct.
I'd like to describe how badly this is implemented:
I run my own mail server and I have a 15+ year history of emailing (mywife)@gmail.com.
On a regular basis (mywife)@gmail.com will email me, and I will respond to her email and my response will go to her junk/spam folder.
And there is no alert, no bounce, no notification.
Let's unpack this:
Google (gmail) knows that these two email addresses converse back and forth, regularly, with a 15+ year history. Google knows that their own user initiated this conversation. Google knows my email is a response to their users email. Google knows my address has never been marked as spam/junk.
So, what kind of unimaginably bad heuristics would have to be in employ to allow this to happen ?
To be honest, this wouldn't bother me that much - I don't think google owes me anything and my wife doesn't pay for their service. What makes me so, so angry is that they behave this way without any notification or bounce email.
You are still hosting it fine. They just decide that you or your messages are suspect.
Also one thing - if people actually want your email they will contact you if they don't get an expected email. If they don't want your messages it is spam.
The problem with e-mail, and with other forms of communication, is that two parties (or their service providers) need to co-operate. You can run your own e-mail server just fine, but Google, Microsoft and friends might consider you to be a spammer or silently block your e-mail just because.
What if email was based on a whitelist instead of a blacklist? So you'd only receive email from addresses of people you've already established contact with some other way (maybe using conventional email)? This eliminates spam and if the big providers supported this, it could also enable them to stop blackholing innocent servers (though whether they care is another question).
Even if your ip address/domain is not in the blacklist right now, it only takes a few people marking your correspondence as spam for it to be blacklisted. Since everyone is on these big free providers, nobody will ever see a single email from you any more. With less centrally controlled email, that would not be possible. I think that is the problem everyone is talking about.
I generally don't check my personal spam folder. I've honestly not seen any false positives with Fastmail. But I certainly do have to check every now and then for my work O365 account which is pretty bad at marking legitimate mail as spam. YMMV of course.
> How do large email providers make it difficult to host your own email?
By not delivering mail sent by your mailserver to mailboxes hosted by them. There's not much use for an own server, if your mail won't be received by most users on gmail or hotmail.
You'll get it when Microsoft decides you are a spammer for no other reason then sending email from port 25 from your house. Or when you can't seem to sign up for a service... until you use your old Gmail address.
Yeah, there was (is?) a period of time where viruses were used to send spam so if you got infected you'd suddenly be sending out a lot of SMTP traffic from a residential IP address. The entire industry adopted the practice of not trusting residential ips. Then the spammers shifted to cheap VPS providers and ip and netblock black lists became more common.
[citation needed]; is this actually going out from gmail or does it just use gmail return addresses?
I too used to run my own email from about 2000-2010, but the maintenance overhead is quite stressful especially because it always happens for critical times or critical emails.
The citation you are looking for is my inbox. That's the spam that still makes it through and there is quite a bit of it, conversely some ham consistently gets misclassified as spam or just simply disappears entirely.
I will echo this experience. An example of an email that made it through from a gmail.com account (abbreviated, it also contained links to some apps (the main purpose I assume) and much more text):
شهر مجاني عند الاشتراك السنوي $ الأسئلة الشائعة
In 1979, LA residents were wearing masks — because of smog Los Angeles Times staff photographer Boris Yaro photographed Sera Segal-Alsberg on Crescent Heights Boulevard in West Hollywood Segal-Alsberg, an artist-instructor, was en route to teach a class at the Los Angeles County Museum of Art
للمزيد من الأسئلة
— In another sign of live entertainment’s rebirth, Bruce Springsteen returned to Broadway over the weekend
يقوم الموظفين بتسجيل حضورهم ، انصرافهم الشركات العصرية مع الاستفادة القصوى من الإمكانيات الهائلة التي تقدمها لنا تكنولوجيا العصر أو الهاتف المتحرك ( الجوال ) أو إذا كنت تستخدم الحاسب فيمكنك استخدام أو يمكن للإداري تحميل هذا التطبيق على جهاز تابلت اشترك في النظام 10
Diverse yet divided cities
واحد أو عدة أجهزة ثم يضعها اشترك في النظام تسجيل دخول فقط في الأفرع المسموح له بالتبصيم فيها | أسبوعين كما أن الموظف لديك يستطيع التبصيم في ثوان قليلة بريد إلكتروني هو نظام إلكتروني قوي وحديث يستخدم لتسجيل فهل يمكن استخدام النظام في جميع هذه الأفرع أدخل بيانات الأفرع إن وجدت الموظفين
Experts say the Delta variant poses a greater chance of infection for unvaccinated people if they are exposed The variant, first identified in India, may be twice as transmissible as the conventional coronavirus strains It has been responsible for the rise in cases recently in India, the United Kingdom and elsewhere
في مداخل الشركة أو أفرعها المختلفة يمكنك الاعتماد على أي جهاز إلكتورني حديث أو حتى قديم في تسجيل ومتابعة تبصيمات الموظفين ، إعرف المزيد اشترك في النظام مجانا , أدخل بيانات الشركة والموظفين
Almost all spam I receive is from Gmail. It's gotten so bad I've actually setup a filter that routes everything from @gmail.com into spam - except for some whitelisted email addresses. G Suite is fine, it's only @gmail.com that is an issue
And yes, it's genuinely from Gmail; valid SPF, valid DKIM, came from a Google IP address, etc...
To say the biggest source is Gmail might be technically wrong though - I suspect there's a large volume of spam that Migadu (my provider) is dropping before it even reaches my inbox, i.e. emails that it is 100% sure are spam and it can just drop. Nevertheless, an overwhelming amount of spam I observe/have to deal with is coming from Gmail. Second to that is outlook/hotmail.
> To say the biggest source is Gmail might be technically wrong though - I suspect there's a large volume of spam that Migadu (my provider) is dropping before it even reaches my inbox, i.e. emails that it is 100% sure are spam and it can just drop. Nevertheless, an overwhelming amount of spam I observe/have to deal with is coming from Gmail. Second to that is outlook/hotmail.
This. It's more likely to be survivorship bias -- the gmail emails happen to survive because gmail is more trusted.
When I ran my own mail server some years ago I was shocked at the amount of spam originating from Google. Definitely their IP addresses as I would routinely get other legitimite mail from the same IP ranges. Was quite a challenge dealing with these spam as it wasn't as simple as blocking their ip ranges as the vast majority of my personal contacts use Google. Never saw the same from Microsoft, Apple, etc.
> Speaking of which, how long before it won't be possible to host your own web server?
It's increasingly getting harder and harder. Recently I was trying to watch a TV show with my friends using a self-hosted Plex server, which was located in one of my friend's house, connected via a gigabit, albeit residential link. Another friend was using LTE internet at that time. He couldn't watch the show, because his connection was so slow, but when he did a speed test the download speed was good enough (100+ Mbit).
Turns out the mobile carrier was throttling connections to select IP ranges to about 1 Mbit (we tested that with a few other IPs). I reckon it was to cripple peer-to-peer protocols. So I guess it's a matter of time until you will be allowed only to connect to certain IP addresses owned by the biggest companies (AWS, Azure, GCP) and nothing else.
Net neutrality wouldn’t fix this is if the issue is a peering problem (which is very common today). The internet has become so centralized that ISPs cheap out on transit and just direct peer to all of the big content providers.
Speaking of which, how long before it won't be possible to host your own web server?
Maybe its just a matter of time for some. For me personally, I could not possibly care less if all the free mail providers blocked me some day. If something is important I can call people and tell them to go to https://mydomain.tld/theirName/ to grab files. I have used this method with non technical people including lawyers without issue. They prefer of course to use their own secure portals. I do acknowledge that running my own mail server may get more expensive with time as I may have to use providers that and more vigilant about keeping abusers off their network.
As for web servers why would I not be able to run my own servers? I can rent VM's, physical servers, racks, cages.
I am just speaking for myself but I will never give in to the bully anti-competitive behavior of the likes of Google and as for ISP's I will not use one that blocks ports or protocols. If there is any blocking to be done it must be done by me. I would never fund an ISP that uses CG-NAT or rate limits something by protocol or port. I realize some people have limited options but at least in terms of blocking and rate limiting, those ISP's are shooting themselves in the feet given that providers like Starlink and various 5G providers will be more common place soon.
There's quite a lot of small providers left and thriving - I've recently migrated from Gmail to mailbox.org, set up inbox encryption with my own key and can't be happier about it.
It's not as feature-rich as Gmail, and webmail with your own encryption key is not usable, but desktop (Thunderbird) and mobile (K9 mail) clients fully cover my use cases. Cheaper than Google Workspace, too.
Yeah I stopped running my own. I kept getting blocked by Microsoft in particular (mainly consumer recipients at live.com and outlook.com, strange enough not corporate O365 users!). I'm 100% sure I did not send any spam, the only emails going to those addresses were legit from a family member. DMARC and SPF were all set perfectly, relays blocked, I was not on any spamlist and I never have been either.
Literally every month I got blocked again because my server did not have enough reputation. Kept logging tickets to get it unblocked and then a month later it was back. One time I did manage to get a personal email back from a guy in India. Said that it was because my mailserver did not send enough legitimate mail for their algorithm to trust it.
So the lack of spam is not enough anymore to be blocked. You actually have to send a load of legit traffic to build up 'reputation'. Now just being a small time sender is a problem. This way the big players can just carve out a bigger market for themselves. They basically break the decentralised concept of email by doing this.
In the end I moved to O365, which felt bad because I didn't want to reward them for their behaviour. But we moved to it at work too and I wanted an instance with full admin rights to explore. My contract is up next year so I may change then if I can find a party that does it well and ideally cheaper.
> Said that it was because my mailserver did not send enough legitimate mail for their algorithm to trust it.
In other words, a small self-hosted email server will be considered a spammer until it starts sending out large amounts of email? Maybe that can be automated...
In theory it would be simple to provide a "cloudworkers cooperative" kind of service that just bundles the outbound mail so that traffic is sufficiently large to be whitelisted by the big providers. The two biggest problems are A. Scaling up sufficiently without attracting Spammers. Because even a single Spammer can ruin your reputation forever. So ideally you'd have a tight knit group of friends or similar. Even then you could hardly assure than no one ever gets hacked. B. Edge Cases. Even if your US or West European Traffic is sufficient to be whitelisted by all major Providers, how do you ensure that the occasional Email to a customer of an Indonesian ISP does not get blocked by their provider...
Yeah SMTP relaying is quite common. The problem is due to email architecture, to my knowledge, that same relay is going to be able to read your incoming emails because remote servers will block emails from user@endserver.org sent from relay.net unless endserver.org has MX entries pointing to relay.net.
'large amounts' is also pretty relative, I'm sure if you had a small team of 10-25 employees on a self-hosted mail server (preferably with a static IP via the ISP) you'd be taken seriously pretty quickly versus only you sending an email once a week or less.
You can configure postfix to relay emails to certain domains through a 3rd party SMTP service like SES. The MS domains give all of us the same problems, there is no other solution.
I had a very similar experience. Is there any cheaper option than just using aws SES nowadays ? (for outbound only! don't understand why people would pay the same rate for inbound)
My concern is what happens when aws decides to massively increase rates...
I tried office 365 for email this year but couldn’t get the marketing emails from Microsoft under control. No matter how much time I spent trawling through the settings menus. Almost every email I got was about some security update or promotion from some ms product I did not use and had no intention of using. And I was paying for O365 too!
It is possible. I managed to stop them in the end. One of the many admin sites if I recall correctly (seriously, they have an office admin portal, exchange online admin, Azure Ad portal and everything is spread out across those)
You may just have been unlucky with your IP block having spammers. How were you hosting it - own ISP or another provider?
I have not had deliverability issues for years with my Kimsufi (OVH France) server. While I am confident my server is well configured using best practices, I suspect some of it is also just luck not to be in the same IP block as a spammer.
I was using a colocation hoster in Belgium. They actually moved me to another netblock to test (they were a really nice small company). But the same happened.
I heard Kimsufi is indeed pretty bad as it's so cheap people tend to use it for 'throwaway' purposes. It's basically the white label budget brand of OVH :)
I've had 2 kimsufi email servers and both were fine. It shares the same data centres as OVH so I guess IP ranges are similar. No problems with blacklisting based on anything other than my own misconfiguration so far, and it's been maybe 8 years.
I'm sorry this happened to you, but it's a shame. You end up giving money to your perpetrators and leaving the rest of us in the same situation you were in previously.
Maybe a hosting coop could be an option? Large enough for reputation but ethical enough to still federate with smaller hosts?
Agreed. But I did learn a lot from it. I needed that because in our large organisation at work the admin rights are highly compartmentalized. And this way I was able to understand what other admins were and weren't able to do.
You can actually get a free test tenant from MS for 3 months but setting up a real production environment is much better than doing some tests.
> This way the big players can just carve out a bigger market for themselves.
Or it’s because there is a near infinite number of domains so it’s relatively simple for spammers to avoid bad rep blocks by grabbing new domains and starting fresh.
I host my own email and I have the same problem with MS. Perhaps this is something for the new Digital Markets Act and interoperability laws in the EU to handle.
Moved away from outlook.com hosting a while ago since so much legitimate transactional email went to spam whilst actual spam easily got through. Now, when outlook forwards to gmail, gmail catch it before it hits the inbox.
I would agree that setting up a robust personal email system is difficult and may be exceedingly so for an organization of any significant size but I am not sure I would attribute that to the quality of services from large providers. Our organization (a university) outsourced its email to MS (outlook) and it is simply awful. No IMAP or POP, no forwarding (this is of course the policy of the university, not MS' fault per se), important emails getting lost as 'spam' (including some once in a lifetime conference invitations) just because they came from the outside. The search is a complete nightmare. No way to send mass emails to one's class with, say individually generated temporary passwords, which I can easily do from my own server. The interface is clunky to say the least. In comparison, even Thunderbird shines (although why TB cannot implement its quick search function for years now is beyond me). So it is not the quality that the big providers supply. Security, maybe (even though they are a large target by default) but not convenience.
I suspect OP’s organization disabled it. Not really Microsoft’s fault but that how a lot of Enterprise products get a bad name - user hostile configuration.
As I mention in my post, this is the setting our admins chose so I do not blame MS for this (they support both, I know). Another setting they chose is to delete emails over six months of age and there is nothing I can do about it either.
The search and the interface are entirely the fault of MS as are the lack of more subtle features such as mass emails (which I have to use often while teaching online).
As is sharing links to OneDrive content with a password on it: https://support.microsoft.com/en-us/office/share-onedrive-fi... ("Set password: lets you set a password to access the file. When a user clicks the link, they will be prompted to enter a password before they can access the file. You'll need to provide this password separately to anyone you want to share the file with.")
(Poor quality sluggish Outlook client, dropping important emails into Spam, not being as configurable as a custom mailserver, hit-or-miss search results, those are all things I can agree with, I'm not just defending it).
Thank you for the links but just to clarify: I have to send each student a random individual password (so that they can login to set up their test taking software). The password is currently generated by a script that then proceeds to email it to each student. Sometimes I have to send individualized assignments (also randomly generated by computer algebra software). My personal email server makes it straightforward but I do not see how it is possible with MS. I guess a cleverly written IMAP client can do that for me but I cannot use IMAP, alas. The university policy is basically 'disable everything that can make using outlook bearable' (this last sentence should compensate for the ambiguous grammar you pointed out :)
They are going to block standard authentication soon though, leaving only their 'modern' webbased authentication. So basically IMAP/POP as we (and our mail clients) know it will no longer work with O365.
They've already delayed it a few times but they keep pushing for this.
I run my own personal SMTP and IMAP server and haven't found it too hard to maintain after the initial setup phase. The main problem I had with getting emails accepted was my lack of reverse DNS PTR records on my domain. If you're unable to fix that (eg, you're not using a commericial internet connection that allows this) the solution is just to use another SMTP relay service. Some, like SMTP2go, are free if you're only sending a personal-use number of emails a day/month. That way you're still in charge of everything except the relaying of outgoing mail, which is easy enough to swap out.
Do you have ~30,000 EUR / year for skilled admin? Plus ~20,000 EUR / year for hardware and other running costs? If you do, you can have your emails safely and reliably exchanged from your basement. If you don't, you can rent whatever on the 'net that suits your budget.
I think those numbers are too high. Even if not, companies like Airbus have that money and are using G-Suite. So Google has all corporate information (does the swamp redirect some of it to Boeing?).
In my experience the time sink is high when you're setting it up. Then it's mostly zero.
Though I'm not sure I'd recommend to invest into it. Back in the days that unix knowledge was valuable and setting up your own e-mail was a good way to learn thing or two. These days those skills are useless for most people, so I'd say use hosted mail and spend that time learning some more valuable skills.
This is not true. Regardless of how secure and properly setup your server is:
Outlook blocks anybody that does not send enough mail, even if you've never sent any spam and are on a clean block. They're happy to let their users to send you spam, but ironically they still block you when you you try to report it to abuse. The good thing is they block you, so the email bounced and you know it wasn't delivered.
Gmail classify messages to a user who has never communicated with you before as spam. This is silent, so you never really know if an email to a Gmail box has been filtered out as spam or not. Their abuse inbox accepts messages but I'm not sure they do anything with it.
Basically, email has been hijacked by two companies.
I do away with the middle ground. I rent email services from businesses that specialize in email hosting and I own domain name. No way I will have my email serviced by Google / MS / Apple and the likes.
Who do you use? After seeing all the horror stories posted on HN, I have been worried about getting banned from $MegaCorp randomly and would like to minimize the blast radius if that were to happen.
I still run my own email system (postfix/dovecot for imap), mostly for one reason: the virtual username function of postfix:
I configured postfix with:
recipient_delimiter = .
which gives me unlimited dynamic virtual addresses (username.<something>@mydomain), so I know where spam/leaks come from if I get unsolicited mail directed to `username.<unique_name_per_registration>`, and it makes it trivial to block.
I know that you can do the same thing with google addresses using + as a delimited, but the + sign is often not allowed in dumb email checks. Also spammers probably know about + and strip it automatically anyway...
I believe you can insert dots as you wish but not use it like the plus sign. So abc@gmail.com is the same as a.b.c@gmail.com but abc.new@gmail.com is another account than abc@gmail.com.
You don't need to run your own email server for this. I do this with a catch-all in Fastmail for $50/year or something. I'm pretty sure Gmail and most others can do this too.
Love that, too! I've always been amazed that spammers aren't able (to my knowledge) to defeat such a simple scheme by removing the . or + in the local part.
If you're running your own server, you can just setup catch-all account and use something like $(printf %s news.ycombinator.com | sha256sum | head -c 12)@mymail.com for further privacy.
I've been doing something with dash as the recipient delimiter since the late 1990s and it's been great. But that became a pain when I wanted to switch to hosted email, as many providers wouldn't support it.
I eventually ended up at Fastmail, as they let you build custom Sieve scripts that can do this kind of remapping without having to run your own mail server.
I've been running my own email since i was 14 years old, back then it was hosted on my home ADSL connection. Now I'm on fiber and still running my own email setup, but the end is near..
For the reason not mentioned in the article, it's getting increasingly harder to actually get a public routable static IP address and also be allowed to either send traffic on port 25, or use the ISPs relay host to actually send the email.
This defeats the entire point of selfhosting whether it's for security concerns or autonomy. The ecological impact of VPS hosting every single service you need is also not negligible: datacenters require huge amounts of resources and infrastructure which a simple second-hand machine at home doesn't.
(also worth mentioning: email protocols were explicitly conceived so that uptime is not a worry)
Datacenters use energy and computer resources more efficiently than a machine at home, unless the machine at home is already running some other tasks that you can't move to the datacenter. A computer that's 99% idle is wasting most of the energy it consumes.
> Datacenters use energy and computer resources more efficiently
That is both true and misleading. Once the datacenter and all surrounding infrastructure (optic fibers, fuel pit, dedicated electricity lines, cooling equipment) and all server/networking hardware has been built, then you start having a better efficiency. If the whole cycle is taken into account, there's no way VPS can be as "green" as selfhosting.
A computer will usually take more energy to build than it will consume over its entire lifetime, so repurposing an existing machine is a good way to go (if you consider minerals-related pollution, even more so).
Also, when you're in a datacenter, servers will be changed every few years. For something as simple/lightweight as email, a 20y old computer will do just fine. A datacenter will renew its entire hardware a few times in that timeframe.
> A computer that's 99% idle is wasting most of the energy it consumes
That is true whether it's in a datacenter or at home. But of course you can share/mutualize resources with other people in order to mitigate this.
Some ISPs provide VPNs on their AS at cheapish price. For instant milkywan in France provide one with public constant IPv4 for 5€/month in France.
It obviously make the whole setup much more complicated, because it leads to a kinda multi-homing setup, but I think it's still reasonable.
How often do you need to send on port 25? In my experience also running my own mail server, never. Receive from servers that don't support encryption yes, but never send. I always send using TLS and since maybe 5 years I've not had an issue with a receiving server not supporting it.
If you want to deliver email to other domains then you need to connect to port 25 on the destination domain MX server. As far as I know, best practice for the other ports (465, 587) is to require authentication and to reject anonymous submissions.
Port 25 is only required if the destination doesn't support TLS, I think. I've not opened port 25 outgoing on my server and I've had zero issues delivering sent mail to other servers for maybe 5 years.
I think OP might have meant "receiving on port 25 is getting difficult" rather than sending. The spec requires servers to support unencrypted deliveries over port 25, even though almost all servers use TLS these days.
Even with TLS, that is usually handled by issuing STARTTLS on TCP port 25. I can't find anything in the RFCs mentioning server-to-server smtp delivery happening on anything but port 25? Do you have a reference for that? In fact, even the MX for google domains (aspmx.l.google.com) does not listen on TCP port 465 or 587, only 25.
I'm not certain, so you might be more informed than me. It's possible my server is sending on port 25 and since the firewall I use doesn't block outgoing connection I just didn't notice. TIL!
There's a few ways to approach this question. One is to mention community networks (DIY ISPs) which will ensure you always have a public IP without filtering. Some even provide VPN access so that you can use your filtered internet to acquire a publicly-routable IP. This is a common pattern in the ffdn.org federation of non-profit ISPs.
Another one is to mention hosting coops (libreho.st/chatons.org) and how they could be employed in limited-network situations. On the web, we have SNI/eSNI-aware proxying which enables multiple servers to share a single IP without revealing their private keys to the reverse proxy. I don't know of an equivalent in the email world (because it's assumed there is only one MX with a canonical domain/DKIM per IP), but i'm all ears if you have suggestions!
Of course, we could mention onionMX and other key-routing systems (CJDNS..) but the problem is you need it to be supported on the other side as well, which is highly unlikely.
The main problem I've run into while running my own email server is IP reputation issues. It is still an ongoing issue for me.
You can read my previous IP reputation issue in this comment [11].
You should make sure your server's IP address isn't blacklisted. If it is, you're going to have major delivery issues with some email server providers (ESP). Some blacklists you can check are listed here [0]-[3].
I think my main problem now is the UCE Protect [5] blacklist. I think some of the major ESPs use their harshest blacklist from UCE Protect, which is their Level 3 blacklist [6]. This blacklist will include your IP if your ISP meets a spam threshold for any of their other IP addresses. This makes running a mail server on cheap hosting providers like Digital Ocean or Linode very difficult.
My conclusion is I should switch to a more expensive ISP that isn't in danger of getting on the UCEPROTECTL3 list or find an email forwarding service for a next hop destination for outgoing mail.
You can read more about UCE Protect here [7]-[10].
I think I am showing my age when I say, I remember when Linux by default came with sendmail enabled by default. You could use your Linux box to send e-mail anywhere without it getting caught up in spam filters. Fun times.
I think you misunderstood. If you fire up a mail server on your Linux box today and try using it for mail the big mail servers are sending your mail to spam due to a bunch of spam rules. These days, you need an organization with a known reputation for sending good e-mail to handle your e-mail so you don't end up in a spam folder. It's how the big players stay big.
As a person running his own email system, I think it's taking more efforts from people to convince everyone to don't run their email systems than managing it effectively, and I don't have any bouncing even towards gmail et similia. As a company the only concern for me is not managing an email, is more to give third party private for profit companies access to all my communications (even if I guess sending it to other unaware people using gmail / 365 has the same effect), but for me it's crazy that the world has accepted that
Not the best choice of title. I didn't realize until I got to the end that this was about large organizations. No doubt for a large university or company this is true.
I run my own email. But I think it is a software problem.
It would be nice of you could just install an email program that will set all the right settings for you. DNS, database, roles and rights, certificates, firewall and so on.
There is server management software that can do this but then you have the same problem: it is just complicated for most people.
518 comments
[ 4.0 ms ] story [ 280 ms ] threadThe wild west days of email, with bang paths and "store and pray" delivery systems, those were fun. By the time `sendmail.cf` hacking was no longer a necessary skill, email had become industrialized. Today, why would you even want to try routing internet email through an RBBS net to WWIV net to some hackers custom Amiga board?
SMTP was the Ford Model T of electronic messaging. It slaughtered the previous visions of what the field needed to be. We can look back fondly at the older ideas and even re-implement their insights now, but the lessons of the market are written in big bold letters now.
I never linked it to WWIV though I was my area code coordinator for WWIVLink. That had to be around 1988 or 1989?
It was wilder before that, think "B news" times.
I've never used gmail itself (that model doesn't fit my mind), but O do use that Google account for some minor stuff. Unfortunazely, I've repeatedly gotten email targetted at someone else having same first name initial and same last name on gmail (address is in the form of FLastname@gmail.com). I've usually been able to get through to those people to stop them and to get them to reach their targets, but in the last 24 months, a lady from Michigan is repeatedly giving out my email address for everything (I've got covid appts, doctor appts, movie tickets, responses to home buying inquires...). I have no idea how to stop this: this would've never happened with any provider that's not owning like 60% of the market.
I am constantly annoyed and I've considered both stopping mail forwarding from this account to mine (but then I might miss that YT premium notification) and I've tried reaching out to many humans on the other side. But I've so far resisted the urge to cancel those movie tickets or vaccination appts, but things just keep coming in.
I can't imagine how are people not overwhelmed by wrongly targetted email: there's more of it than spam I get on my personal server, so spam filtering would definitely not move me towards gmail. And actual spam also gets through on gmail!
She may be getting it at f.lastname@gmail.com.
I signed up for Gmail the first or second day it came out with first.last@gmail.com, but hardly ever log(ged) in over the years. A little while ago I did go into it and noticed getting a bunch of message to firstlast@gmail.com.
Now Gmail allows for 'customized' addresses in that you can drop a period anywhere and it will still go to your address. But this raised the risk of one person signing up with first.last and another person signing up with firstlast. Supposedly this is prevented, but I think that they did not catch this situation in the early days of the service, and so a bunch of OG accounts have cross-contamination.
To even use that gmail address I need to basically whitelist senders and filter them into folders and ignore the inbox completely.
Gmail, like so many of Googles services these days, is an absolute mess. Features no one asked for, blatant spam that gets through their checks while your actual emails go to the spam folder, and a constantly degrading UI that seems to be an experiment in how much you can annoy the user.
That's a user problem, not a technical one. (there is, of course, an XKCD for that).
I've got a popular initials/surname combo and I have a number of doppelgängers giving it out. The one I feel most sorry for is the trumpist and his scary NRA/pro-gun mail. It's really fierce stuff, I'm glad it's going into my spam folder rather than in front of a real human!
He got many "weird" messages over the years from messages addressed to a religious minister, to a pro-gun individual and to NSFW account that he didn't sign up for.
I had assumed that the spammers had moved off to other mediums. Either that or they are specifically targeting big servers like Gmail and are leaving the smaller servers, with their varied (artisanal) anti-spam approaches alone.
Especially if your organization is a potential financial target.
A lot of legit emails end up in my spam box.
Another reason could be that I live in a country with very strict anti-spam and privacy laws (Canada). I have always assumed that spammers wouldn't care but who knows...
If you put some decent effort into making sure that you don't send spam, try to monitor if anyone thinks you send spam and react when someone complains that you send spam (and stop it), it works.
In my experience people telling these stories often do send spam, but they don't believe they do. ("It's not spam, it's a Newsletter. No, it has no unsubscribe link. These are people that agreed to be put on the newsletter by clicking on some ToS they never read, and they can unsubscribe by some arcane mechanism that we will make as complicated as we can. But we're definitely not spammers.")
But what is a problem is providing a good enough web interface, search, and so on.
However, one of my servers IPs is on a Microsoft blacklist since many years now. It sends <10 messages / day. I've tried every unlist form I could find, even called MS but it does not get taken of that list and they "won't disclose why". I'm routing SMTP to MS via another relay now :)
How do you do this? Could you share details on the setup?
transport.db:
hotmail.com relay:[relay.server.tld]:587 # and other domains
main.cf:
transport_maps = hash:/etc/postfix/transport smtp_sasl_password_maps = hash:/etc/postfix/relay_passwd
relay_passwd.db: # if necessary / not authenticated by IP relay.server.tld user:pass
The relay can/should rewrite the Return-Path to pass SPF. It's no problem for DMARC as the DKIM signature added by the initial server still authenticates it.
It requires manually adding domains of custom 365 installations to the list - at this size I do this manually, but should probably be automated "on bounce" or maybe even by a smart rule based on the MX record.
In Exim4 it's also possible to conditionally rewrite based on for example the recipient domain.
Well, I put in more than some decent effort, and I didn't get it "to work". I detailed my efforts here:
https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...
Please stop spreading falsehoods. If you were able to somehow get your own email server to deliver email to Gmail and Outlook, great, good for you - but stop pretending that anybody can do it.
Yes, that's probably true. I've been running my own server for 20 years now, and I guess that in itself helps with getting my mail delivered (apart from t-online, but who cares about them). At some time I also hosted some mailing lists, but I quickly abandoned that because that's a surefire way to get your IP blacklisted sooner or later. If you set up a completely new mail server, there probably is a lot of luck involved, and I wouldn't recommend it to anyone, at least not for your critical business mails. I pretty much keep doing it only out of nostalgia, it doesn't really make any sense otherwise.
A few years ago we had problems, but then I realized some of the emails sent from our servers had non-ascii characters in headers (subject, from, to) which caused email-providers to distrust our server. Using encoded-words syntax ("=?UTF-8?B?" + BASE64(text) + "?=") fixed that problem:
https://en.wikipedia.org/wiki/MIME#Encoded-Word
Most of the time, delivery problems were of my own creation. Like running out of disk space or accidentally disabling TLS.
Once in a while, Microsoft would start swallowing emails or Google would push everyone to use DMARC.
But overall, the experience has been very pleasant. I host my mails, I own my data. I am not shy of using Google, but my work is not defined by their whims. When Google tells me I ran out of space in my account I just delete stuff because I have copies of everything outside of Google infrastructure.
It requires some effort to maintain and understand, and I’ve had a few deliverability issues over the years but they are generally with niche providers. I’ve never had trouble sending mail to the big providers.
Every time I read comments about the impracticality of self-hosted email, I scratch my head. Maybe I’ve just been lucky.
>Every time I read comments about the impracticality of self-hosted email, I scratch my head. Maybe I’ve just been lucky.
I feel the same. I've had one or two hiccups but smooth sailing for the most part. I'm also happy to provide receipts that show how the recipient's mail server is responding when I send the emails. It's a powerful tool to say, "your mail provider is misbehaving, look!" They will wonder how many people tried to send them email that didn't get to them.
[0]: https://postmaster.live.com/snds/JMRP.aspx
What ISP should someone choose?
I'm doubtful a default block would work, as that would even penalise the 'big boys' of email when they make basic network changes and piss off existing customers of both sender and receiver... Its easier and logical to conclude something without reputation yet is therefore sending too few mails to be useful to a spammer.
I've had good experiences with smaller ISPs (currently Mythic Beasts). In contrast, OVH was a poor experience.
I find that reputation (beyond the known "block-lists") appears more likely being tracked for the whole AS number, therefore a lot more to do with your "neighbours" than anything else.
What matters most is if the IP address they issue you has been blacklisted for spamming. DigitalOcean is fine but you need to check the IP address before you do the work of building a mail server. Some of their IPs are on a lot of blacklists.
If it's only on a very few you need to look into who's blacklisting it. There are some that offer a way to get delisted and make it easy, there are others that block pretty much every IP address DigitalOcean has (or large ranges of them) and they won't de-list anything within them. Many of those blacklists are managed overseas and not used much in the U.S.
No matter the ISP you should check the IP address they issue for a VPS before you build the email server.
My personal gmail account is full of spam and emails I do want from email lists end up the spam folder randomly.
That being said, now I monitor and auto-ban failed authentication attempts to smtp/imap (among others) and running the service is fairly low maintenance.
But the morale of the story is that you are only one weak password from one of your users away from your mail server getting blacklisted as a spam server. So while I think it is fairly easy to run a personal server, running one for a small organisation is another matter.
That's about as easy as it gets but it still requires some work and you need to check the IP address DigitalOcean issues to see if it's blacklisted before you set it up.
Google makes it easy to get whitelisted. Microsoft email services (Hotmail/Outlook) are a pain though. I tried to get through their process but got nowhere. Other services I had to submit a request to get de-listed. So it does take awhile to go through all that.
Still, I prefer that to hitching that wagon to a 3rd party provider like Google, or any other.
Before I set mine up the 1st time I'd been screwed a few times by 3rd party providers. The last one, I can't recall which, but it was either "MailChimp" or whomever bought them, that I'd configured an app to use and almost as soon as I'd released it they announce they'd been acquired and I would have to use the new services APIs, and of course they cost more, and their services were geared towards mass mailing, and that's not what my apps do, and their API sucked for my needs.
It was about 12 years of dealing with 3rd party bullshit that motivated me to set up my own email server.
If you just want to fiddle around with one to get a feel for it Mail-in-a-Box is a good place to get started: https://mailinabox.email/
I've been running my own mail server since mid '00s. Initially hosted with one of West coast Canadian colos and subsequently moved to an EU colo. Had some deliverability issues with Outlook and Yahoo, but these were episodic and rare even though I set up DKIM only last year and have been running with just SPF and DNS/PTR before that.
I know at least a dozen of others with similar setups and timelines. But we all use dedicated colo'ed boxes on IPs from clean netblocks that weren't previously used for shared hosting. I strongly suspect that attempting to run a mailserver on Digital Ocean, OVH, 1and1 and similar mass-hosting providers will not go well. Just like it will be an uphill battle to run it on a residental IP.
What did I generalize, exactly? Parent poster was claiming that anybody can set up a mail server with good deliverability - that's a generalization. I said good for them (acknowledging they managed to make it work) and said that I also tried and couldn't make it work - therefore, clearly not everybody can make it work. Did I not argue against generalization there?
I run my mail server on Linode, no issues at all.
I do it as well and apparently so do many others.
Not sure why it seems problematic for some, but it hasn't been an issue at all for me.
I've run a hobby website for about 15 years that does not even have a newsletter of any kind, and includes "stop sending me emails" in each transactional email (all users are double opt-in verified), and password resets are still not delivered half the time to gmail addresses.
Frankly, I'm shocked you've never been arbitrarily blocked and I find your insinuations offensive.
The last time I was getting blocked it was the solarwinds fiasco where their internal mail tunneling/forwarding and filtering setup broke all DKIM and suddenly solarwinds users like NOAA.gov were rejecting me and adding me to naughty lists. There was no fallout for the megacorps and their broken setups. There was only damage to independent mailserver operators doing the right thing.
I've run my own email for decades and I've designed and run some pretty big commercial installations.
As a small provider, you run the risk of existing in a netblock used by other people sending spam. A small co-op I ran encountered this problem once. They were operating on the cheap and while they weren't sending spam their neighbors had been.
Even as a large provider at a billion dollar company, figuring out delivery issues is a huge pain and generally not worth it. There are unofficial professional postmaster meetups around the bay and these can be helpful in getting escalation contacts to fix issues, but even with entire teams of people dedicated it's a lot to handle and is usually worthwhile to outsource the work to other companies who already have these types of relationships established.
Unlike the person to you're replying to we had no issues with Google or Microsoft (once we did the requisite things) - it was Yahoo (and the people they provide email for) and then multiple mid-size organisations who used IP based block lists. At one point our mails were being rejected by our local NHS trust, the London Fire Brigade and a mental health agency we make referrals to. None of this was complicated to resolve but it was energy that could have been better used elsewhere.
I'm not usually part of the "let's go cloud without doing any cost-benefit analysis" movement but with email delivery I was happy when we could go back to routing via Office365 again. If a recipient decides to ban Microsoft's IPs that's usually going to be a bigger problem for them than me.
If you can have a high degree of confidence that no outgoing smtp spam traffic has ever been emitted from any of the other IPs adjacent to where you're hosted, the opaque blacklists of the big mail receiving providers (gmail, etc) are much less likely to consider your legit traffic as spam.
I've been running mailservers using free software for 20 years. I've run two for personal use, and several for groups like companies. In the old days, you could indeed throw up a server, and provided you don't spam, and you're not in a bad neighbourhood, outgoing mail would be accepted.
In more recent years, my experience has been that it takes time for a new mail sender to be acccepted; could be a year or two to build reputation. That's assuming you do everything right.
My personal mail, by the way, has been on the same domain since about 2001. I've quit running a mailserver now. My small ISP runs a setup that's basically what I would have built, so I use that; the support is excellent. But it's still on the same domain.
Last company I was at ran their mail on their ISPs mailserver. The ISP got taken over; service deteriorated, to the point it became unacceptable. So I built $EMPLOYER a mailserver; it took me longer than I predicted, because the bosses had all kinds of finicky requirements (don't they always) that I had to figure out how to provide after the fact. But that "artisanal" server beat the bejabers out of the ISP system; it was fast, reliable, and when anything went wrong I could fix it - which that ISP couldn't.
Also running my own servers for personal and business, and working well.
But when we tried to use one of the large VM providers the experience was much less reliable. Despite ensuring the IP was not on the various block lists etc. mails would be accepted and silently discarded by recipients ISPs, perhaps due to the level of abuse of these IP ranges.
Yes. I do get that impression from most complainers.
I send from my own domains, and if I sent it, I wrote and addressed it personally.
If I send an email to a corporation’s customer support, or to a distant relation, or to an open source mailing list, and I don’t get a reply, it could be a delivery problem - or it could just be that they didn’t decide to reply.
At work I ran email servers professionally and with good deliverability for years. My own email server was arguably longer lived than those at work, just much lower volumes. IP block was clean, DKIM, SPF, rDNS, etc. all setup correctly.
I thought I had no deliverability issues. I interacted with mailing lists regularly, the odd email to friends and family and I was firmly in your camp until I had to deal with a death in the family.
I think this was shortly after Microsoft BPOS became Office365. It became very very clear very very rapidly that to certain orgs I just wasn't hitting the inbox. And there was jack shit I could do about it. That was the end of my mail server, and it's certainly got worse over time.
The unspoken "you silly prick" gets louder every time this happens.
The big players all have a process and followup within days.
There's a tendency to perma-reject e-mail coming from "not seen before" domains despite the e-mail passing FCrDNS + SPF + DKIM + DMARC validation, which makes it difficult for private e-mail server users to get through to people.
I’d argue that the vast bulk of email is sent from dedicated providers like Sendgrid which are built on the same tech that might be found in any given ‘artisanal’ on-prem service.
This is a popular opinion on HN but it doesn't seem at all inline with reality. Email isn't exactly a real money maker for anyone. And the amount of email spam and abuse is immense. Filtering out most unknown providers is unfortunately extremely effective. Almost all spam wiped out with a simple check.
Maybe the ideal solution would be to let you link your custom email domain with a google account so you can have your google account vouch for the legitimacy of your custom domain. But even then, some of the time your email server actually is just blasting out spam without you knowing it.
My biggest problem with mail delivery is sending mail to Microsoft properties. I've had to resort to sending those messages via SES.
Open source webmail solutions suck, so now I'm paying fastmail and forwarding incoming messages there.
> Entirely "on premise" email is now an inferior thing for almost everyone.
I disagree on this one. Placing your email with a big player means that by definition, they have access to your mailbox (with sensitive stuff hopefully encrypted). To allow that you have to trust the big player and the countries where they reside.
They can drop you any time for political reasons, for dealing with a country that is considered an enemy of the host country of your provider. They may sell out your data.
You may still choose a big player, but understand how screwed you are.
"Your account has been suspended for suspicious activity."
The author of the post should have included that in his post.
Sort of, but not entirely true ...
If you run your own mailserver then users of that mailserver can send and receive mail, to each other, without traversing a network. The mail never goes out on the net. That can be valuable/interesting.
This is true in both the webmail use-case (the text goes to the browser, ephemerally, encrypted with SSL) or the terminal/console (alpine) use-case (the text goes to the terminal, ephemerally, encrypted with SSH).
There's a certain cloud storage provider I know of whose internal / intra-company emails have never traversed the Internet ...
DMARC does not provide signatures, DKIM does.
DMARC adds the DKIM 'alignment' requirement. Meaning that not just any DKIM signature will do, the public key (the DKIM DNS record) must be published under the administrative domain (the part after the '@' in the sender address).
DMARC also mandates SPF alignment (not that your should rely on SPF), meaning that the rfc5321.MailFrom and rfc5322.From address should be from the same administrative domain for the SPF to pass DMARC.
When either SPF or DKIM is aligned, you have a DMARC pass. Because SPF breaks with forwarding services, you shouldn't rely on it. DKIM + DMARC is the way to go.
Also funny that the author calls DMARC 'modern practice', since DMARC was introduced in March 2015, almost 7 years ago.
Oh come on, in terms of protocols, that's modern.
Your other points have merit but that's just a pointless dig at the author.
A lot of people will say, "no they'll blame you for it not going through", but that's rare. Most people will be receptive to your insistence that you're trying to send them mail but their provider is in the wrong.
Now, if you mean blacklisting in the sense that the server has shadowbanned you and is sending back "221 OK", then, again, you have an affirmative defense: "hey, you're mail server said it accepted the mail, can you please check with your provider on what they did with it?"
In either case, this is actually not a good thing for the recipient's mail provider especially if they pay for that provider. "Why did you accept the email from the sender but not put it in my mailbox? Who else have you done that for? Why am I using you as my mail provider again?"
I have a circle of influence... about 100 people. All 100 of those people would switch mail providers if I asked them to. And I think a lot of people have a circle of influence around this size as well. So there actually is an amount of control over these bigger mail providers. They will be receptive to "I just told your customer to switch because you won't let me email them" especially if it's widespread. In exchange for that "power", I make sure my mail server is as clean as possible and quickly respond to any notices sent to my abuse@ address.
I ran mailinabox for a year or two, but eventually I just didn't want to maintain a piece of software I didn't understand where the documentation seemed actively hostile and presumptuous about me having read all the other parts. I'm sure the postfix docs make an okay reference, but understanding it as a whole, god no. I'd rather do Kubernetes from scratch.
Fastmail is just fine for me.
The docs for magic word projects never to seem to prioritise essentials. So [obscure feature someone last used in 1984] gets equal billing with [essential fundamentals] and you have no idea which is which because - you haven't understood the docs yet.
I'm still running my own servers. I sorted out the spam issues, and they're basically zero maintenance now. But it certainly took a while, and a fair amount of copying other people's ideas of what a config file should look like, with plenty of trial and error.
* https://workaround.org/ispmail
Maintainer of Mail-in-a-Box here. I'm sorry you had that experience. Definitely was not the intention of the project to be hostile (but I can see how it might come off that way).
This is one thing the authoritarians like Biden (Clipper Chip, Patriot Act) won't want to fix. There will be no law that companies with more than 100 employees must accept mail from individual servers (they would still have the correspondence anyway, but it would be a start). There will be no law that all mail must be encrypted.
Speaking of which, how long before it won't be possible to host your own web server?
On another note: the biggest source of spam is gmail itself, and guess what, that makes it to my inbox just fine, because what could possibly be wrong with someone using google as their source. Spam was annoying but it was never an actual problem. The consolidation of the internet into a handful of players is a problem.
I don't get this one. How do large email providers make it difficult to host your own email?
I host my own email. It was a pain to setup so I try not to touch it since it is running fine. Setting up email on your own server is just complicated unless you install server management software. I am not sure big email providers are to blame for this.
By randomly marking your email as spam without any recourse. This may be because they blacklist your provider en bloc, your IP address or some subnet, because they feel like it, it's Tuesday or because their spam filters suck.
But it happens and it happens often enough that running a business in that way will cost you money, sometimes lots of it.
So you're saying that anything can get you blacklisted if you're unlucky enough? I think that's the point of the people you're arguing with.
At this point we just need to figure exactly how unlucky.
Correct.
I'd like to describe how badly this is implemented:
I run my own mail server and I have a 15+ year history of emailing (mywife)@gmail.com.
On a regular basis (mywife)@gmail.com will email me, and I will respond to her email and my response will go to her junk/spam folder.
And there is no alert, no bounce, no notification.
Let's unpack this:
Google (gmail) knows that these two email addresses converse back and forth, regularly, with a 15+ year history. Google knows that their own user initiated this conversation. Google knows my email is a response to their users email. Google knows my address has never been marked as spam/junk.
So, what kind of unimaginably bad heuristics would have to be in employ to allow this to happen ?
To be honest, this wouldn't bother me that much - I don't think google owes me anything and my wife doesn't pay for their service. What makes me so, so angry is that they behave this way without any notification or bounce email.
That's just shitty.
This is every Google product in a nutshell for me. Their "algorithms" are absurdly bad in every category.
Also one thing - if people actually want your email they will contact you if they don't get an expected email. If they don't want your messages it is spam.
By not delivering mail sent by your mailserver to mailboxes hosted by them. There's not much use for an own server, if your mail won't be received by most users on gmail or hotmail.
[citation needed]; is this actually going out from gmail or does it just use gmail return addresses?
I too used to run my own email from about 2000-2010, but the maintenance overhead is quite stressful especially because it always happens for critical times or critical emails.
You are of course welcome to not believe me.
شهر مجاني عند الاشتراك السنوي $ الأسئلة الشائعة
And yes, it's genuinely from Gmail; valid SPF, valid DKIM, came from a Google IP address, etc...
To say the biggest source is Gmail might be technically wrong though - I suspect there's a large volume of spam that Migadu (my provider) is dropping before it even reaches my inbox, i.e. emails that it is 100% sure are spam and it can just drop. Nevertheless, an overwhelming amount of spam I observe/have to deal with is coming from Gmail. Second to that is outlook/hotmail.
This. It's more likely to be survivorship bias -- the gmail emails happen to survive because gmail is more trusted.
> Gmail obvious spam still #1 in the quarantine folders..
-- Michael Peddemors, President/CEO LinuxMagic Inc.
https://www.mail-archive.com/mailop@mailop.org/msg14526.html
It's increasingly getting harder and harder. Recently I was trying to watch a TV show with my friends using a self-hosted Plex server, which was located in one of my friend's house, connected via a gigabit, albeit residential link. Another friend was using LTE internet at that time. He couldn't watch the show, because his connection was so slow, but when he did a speed test the download speed was good enough (100+ Mbit).
Turns out the mobile carrier was throttling connections to select IP ranges to about 1 Mbit (we tested that with a few other IPs). I reckon it was to cripple peer-to-peer protocols. So I guess it's a matter of time until you will be allowed only to connect to certain IP addresses owned by the biggest companies (AWS, Azure, GCP) and nothing else.
Maybe its just a matter of time for some. For me personally, I could not possibly care less if all the free mail providers blocked me some day. If something is important I can call people and tell them to go to https://mydomain.tld/theirName/ to grab files. I have used this method with non technical people including lawyers without issue. They prefer of course to use their own secure portals. I do acknowledge that running my own mail server may get more expensive with time as I may have to use providers that and more vigilant about keeping abusers off their network.
As for web servers why would I not be able to run my own servers? I can rent VM's, physical servers, racks, cages.
I am just speaking for myself but I will never give in to the bully anti-competitive behavior of the likes of Google and as for ISP's I will not use one that blocks ports or protocols. If there is any blocking to be done it must be done by me. I would never fund an ISP that uses CG-NAT or rate limits something by protocol or port. I realize some people have limited options but at least in terms of blocking and rate limiting, those ISP's are shooting themselves in the feet given that providers like Starlink and various 5G providers will be more common place soon.
It's not as feature-rich as Gmail, and webmail with your own encryption key is not usable, but desktop (Thunderbird) and mobile (K9 mail) clients fully cover my use cases. Cheaper than Google Workspace, too.
Literally every month I got blocked again because my server did not have enough reputation. Kept logging tickets to get it unblocked and then a month later it was back. One time I did manage to get a personal email back from a guy in India. Said that it was because my mailserver did not send enough legitimate mail for their algorithm to trust it.
So the lack of spam is not enough anymore to be blocked. You actually have to send a load of legit traffic to build up 'reputation'. Now just being a small time sender is a problem. This way the big players can just carve out a bigger market for themselves. They basically break the decentralised concept of email by doing this.
In the end I moved to O365, which felt bad because I didn't want to reward them for their behaviour. But we moved to it at work too and I wanted an instance with full admin rights to explore. My contract is up next year so I may change then if I can find a party that does it well and ideally cheaper.
In other words, a small self-hosted email server will be considered a spammer until it starts sending out large amounts of email? Maybe that can be automated...
I have not had deliverability issues for years with my Kimsufi (OVH France) server. While I am confident my server is well configured using best practices, I suspect some of it is also just luck not to be in the same IP block as a spammer.
I heard Kimsufi is indeed pretty bad as it's so cheap people tend to use it for 'throwaway' purposes. It's basically the white label budget brand of OVH :)
Maybe a hosting coop could be an option? Large enough for reputation but ethical enough to still federate with smaller hosts?
You can actually get a free test tenant from MS for 3 months but setting up a real production environment is much better than doing some tests.
But yeah I feel lousy about it.
Or it’s because there is a near infinite number of domains so it’s relatively simple for spammers to avoid bad rep blocks by grabbing new domains and starting fresh.
Microsoft seem to disagree with you there: https://support.microsoft.com/en-us/office/pop-imap-and-smtp...
The search and the interface are entirely the fault of MS as are the lack of more subtle features such as mass emails (which I have to use often while teaching online).
Mass emails via distribution lists are a thing - https://www.wisestamp.com/blog/managing-distribution-lists-i...
As is sharing links to OneDrive content with a password on it: https://support.microsoft.com/en-us/office/share-onedrive-fi... ("Set password: lets you set a password to access the file. When a user clicks the link, they will be prompted to enter a password before they can access the file. You'll need to provide this password separately to anyone you want to share the file with.")
(Poor quality sluggish Outlook client, dropping important emails into Spam, not being as configurable as a custom mailserver, hit-or-miss search results, those are all things I can agree with, I'm not just defending it).
They've already delayed it a few times but they keep pushing for this.
Do you have ~30,000 EUR / year for skilled admin? Plus ~20,000 EUR / year for hardware and other running costs? If you do, you can have your emails safely and reliably exchanged from your basement. If you don't, you can rent whatever on the 'net that suits your budget.
Though I'm not sure I'd recommend to invest into it. Back in the days that unix knowledge was valuable and setting up your own e-mail was a good way to learn thing or two. These days those skills are useless for most people, so I'd say use hosted mail and spend that time learning some more valuable skills.
Outlook blocks anybody that does not send enough mail, even if you've never sent any spam and are on a clean block. They're happy to let their users to send you spam, but ironically they still block you when you you try to report it to abuse. The good thing is they block you, so the email bounced and you know it wasn't delivered.
Gmail classify messages to a user who has never communicated with you before as spam. This is silent, so you never really know if an email to a Gmail box has been filtered out as spam or not. Their abuse inbox accepts messages but I'm not sure they do anything with it.
Basically, email has been hijacked by two companies.
I configured postfix with:
which gives me unlimited dynamic virtual addresses (username.<something>@mydomain), so I know where spam/leaks come from if I get unsolicited mail directed to `username.<unique_name_per_registration>`, and it makes it trivial to block.I know that you can do the same thing with google addresses using + as a delimited, but the + sign is often not allowed in dumb email checks. Also spammers probably know about + and strip it automatically anyway...
unless they use BCC
I eventually ended up at Fastmail, as they let you build custom Sieve scripts that can do this kind of remapping without having to run your own mail server.
(also worth mentioning: email protocols were explicitly conceived so that uptime is not a worry)
That is both true and misleading. Once the datacenter and all surrounding infrastructure (optic fibers, fuel pit, dedicated electricity lines, cooling equipment) and all server/networking hardware has been built, then you start having a better efficiency. If the whole cycle is taken into account, there's no way VPS can be as "green" as selfhosting.
A computer will usually take more energy to build than it will consume over its entire lifetime, so repurposing an existing machine is a good way to go (if you consider minerals-related pollution, even more so).
Also, when you're in a datacenter, servers will be changed every few years. For something as simple/lightweight as email, a 20y old computer will do just fine. A datacenter will renew its entire hardware a few times in that timeframe.
> A computer that's 99% idle is wasting most of the energy it consumes
That is true whether it's in a datacenter or at home. But of course you can share/mutualize resources with other people in order to mitigate this.
I think OP might have meant "receiving on port 25 is getting difficult" rather than sending. The spec requires servers to support unencrypted deliveries over port 25, even though almost all servers use TLS these days.
Another one is to mention hosting coops (libreho.st/chatons.org) and how they could be employed in limited-network situations. On the web, we have SNI/eSNI-aware proxying which enables multiple servers to share a single IP without revealing their private keys to the reverse proxy. I don't know of an equivalent in the email world (because it's assumed there is only one MX with a canonical domain/DKIM per IP), but i'm all ears if you have suggestions!
Of course, we could mention onionMX and other key-routing systems (CJDNS..) but the problem is you need it to be supported on the other side as well, which is highly unlikely.
You should make sure your server's IP address isn't blacklisted. If it is, you're going to have major delivery issues with some email server providers (ESP). Some blacklists you can check are listed here [0]-[3].
I think my main problem now is the UCE Protect [5] blacklist. I think some of the major ESPs use their harshest blacklist from UCE Protect, which is their Level 3 blacklist [6]. This blacklist will include your IP if your ISP meets a spam threshold for any of their other IP addresses. This makes running a mail server on cheap hosting providers like Digital Ocean or Linode very difficult.
My conclusion is I should switch to a more expensive ISP that isn't in danger of getting on the UCEPROTECTL3 list or find an email forwarding service for a next hop destination for outgoing mail.
You can read more about UCE Protect here [7]-[10].
[0]: mailtester: https://www.mail-tester.com/
[1]: mxtoolbox blacklists: https://mxtoolbox.com/blacklists.aspx
[2]: proofpoint blacklist: https://ipcheck.proofpoint.com
[4]: outlook blacklist: https://sendersupport.olc.protection.outlook.com/snds/index....
[5]: UCEPROTECT: http://www.uceprotect.net/en/index.php
[6]: UCEPROTECTL3 blacklist: http://www.uceprotect.net/en/index.php?m=3&s=5
[7]: UCEPROTECT Blacklist Scam https://community.spiceworks.com/topic/2170592-uceprotect-bl...
[8]: UCEPROTECT: When RBLs Go Bad https://blog.sucuri.net/2021/02/uceprotect-when-rbls-go-bad....
[9]: ASK HN thread https://news.ycombinator.com/item?id=26064722
[10]: SQLite3 IP blacklisted: https://sqlite.org/forum/forumpost/bb61881d7a?hist
[11]: Previous IP reputation issue https://news.ycombinator.com/item?id=25437841
It would be nice of you could just install an email program that will set all the right settings for you. DNS, database, roles and rights, certificates, firewall and so on.
There is server management software that can do this but then you have the same problem: it is just complicated for most people.
These days, it's probably easier than it was, back then (about twenty years ago).
It was a nightmare. I didn't do it for a living, so I was consumed by the task. It screwed up my other work, something fierce.