Ask HN: Why can't I host my own email?
I can host my own Mastodon server, or all kinds of other novelty / fun things which don't seem easily decentralized.
Email feels like one of the most decentralized internet concepts, and ironically it's seemingly the one thing I can't self-host unless, from what I've heard, I enjoy being permanently marked as spam / blacklisted. What's going on? How do we fix this?
405 comments
[ 3.1 ms ] story [ 326 ms ] threadhttps://www.google.com/search?client=firefox-b-1-d&q=self+ho...
https://www.google.com/search?q=self+hosted+email+server
https://www.google.com/search?q=dkim+dmarc+spf
What often happens is that virtual hosting firms (Linode, Digital Ocean, etc.) are often used by spammer's for their hosting too, and so if you try to host by renting a "cloud vm" or "cloud server" and are unlucky to have an IP address a spammer previously poisoned, or just happen to be in the same netblock as a prior spammer, you find your new IP often 'blocked' from the big services, for no good reason than you happen to be from a "bad neighborhood". And this is usually the genesis for all the scary stories about "can't self host".
But reality is, you can self host, but you do have to set things up with all the modern requirements (SPIF, DKIM, etc.) as well.
it’s too easy for scammer to create a domain name.
or is it only because smtp doesn’t have any kind of authentication and rely only on IP adress?
If you set everything up right, and choose the host for your mail server carefully, and never change IP, after a fairly short time you won't have much problem with being marked as spam. No more so than with any other email host.
As is so often the case, the people that say you should never do it probably have little relevant experience, they are just repeating something they heard.
More likely, they're saying that 99% of people don't know how to self-host, and for 99% of the rest it's not worth the trouble. Also, if you have to ask, then you shouldn't self-host it.
If you start self-hosting now, you should be prepared to lose quite a few emails randomly for the first X months while everyone else tries to figure out whether you're legit or not. Though I would encourage anyone who can to try to self-host at least some part of their email infrastructure, even if just for the learning experience, I would also recommend that they avoid using self-hosted email for anything business-critical until they're sure they've got the hang of it.
Also, little-known fact: if you register a UK company (probably more practical if you already have one, but the effort is not actually that big), you can register .uk domains directly with Nominet, the UK registry, by setting yourself up as a self-managed registrar. It doesn't cost anything (beyond the cost of the domain name) and is very easy. I'd love to know if there are any other registries that allow something similar.
And then prepared to lose quite a few emails consistently for the next 10 years when some decide you're not legit.
Source: I self-host.
Use mail-tester.com or similar tools to ensure everything is configured correctly.
And then just start sending. As long as your volume grows slowly over the first few months, you’ll get basically no rejects.
For a second, I thought I was on Stackoverflow. If you aren't starting by asking questions about the possibilities or limitations of a system you're about to work in, then you aren't starting properly.
We need to give newcomers a break and answer their questions well, and discuss to promote understanding, instead of swatting at them with our canes. The only way knowledge passes to the next generation of thinkers and tinkers is if we fuel that curiousity.
There's plenty of other tech they can screw with.
I've wanted to do this for a long time, and am using this very HN post as a source of information. I don't expect anyone to set it up for me, but I am hoping to identify the "gotchas" that are likely not discoverable by reading man pages.
Otherwise, how would anyone learn anything?
Certainly way more than 99% of the general population wouldn't know how to self host, but within a techie population like HN, easily ~50% can be capable of doing it if they wanted to. Whether it's worth the effort is a personal decision, but there's a lot of value in owning your own email so I recommend it to anyone who's curious about it and willing to do it.
> Also, if you have to ask, then you shouldn't self-host it.
We should be encouraging curiousity (a HN value) not stomping on it.
If anyone asks, I say go for it. Worst case you'll learn new things, best case now you own your email.
No, it's not "wild".
Its just that we're in 2022, not 1997.
Long gone are the days of "fire up Sendmail and you're good to go".
To those thinking of self-hosting, I would say they should start by understanding modern anti-spam.
Understanding modern anti-spam will not only help them with their inbound email, but will also help them understand how to ensure deliverability of their outbound email too.
2007..2014 were probably the worst. Gmail was chainging often, Microsoft was blocking everyone.
I think self-hosting is easier now than 10 years ago.
This is because the business department will try to force admins to ignore requests to remove spammers as the spammers generally pay decent money.
By blocking everyone, it forces the ISP to get off their ass and kill the spammer irrespective of how much they are willing to pay.
https://blog.sucuri.net/2021/02/uceprotect-when-rbls-go-bad....
ISPs are now so sensitive to not allowing spammers that the RBLs can't make money from it anymore.
I think it's also fair to say that personal mail for a small domain is much easier than even a small amount of transactional email and don't even try sending newsletters beyond your friend group.
I have run mail off three different IPs over the ~20 years I've been hosting, switching IP address didn't affect me all that much.
Another thing to note is that receiving mail is really easy. Sending it is hard, filtering out the spam (and only the spam) from your inbound email is harder.
Of course, this is not 100% reliable, as it's not too difficult for spammers to adapt and improve their scripts. Of course, vast majority of spammers are either not sophisticated enough, or do not care enough to do so, so if you don't mind your incoming mail to be slightly delayed, it's kind of a low-hanging fruit, as it cuts off a huge amount of low-effort spammers.
Doing it this way doesn't even delay mail much most of the time; many legitimate MTAs connect immediately to the priority-20 MX after failing to connect to the priority-10 one.
So a easy way to get started is to receive everything directly and use a commercial (often with low-volume free tier service) relay for outbound until you get comfortable enough to remove the training wheels. (Or never remove them, that's a legit choice as well.)
> filtering out the spam (and only the spam) from your inbound email is harder.
I don't find that at all. Filtering spam is the easiest part. All I do is if SPF doesn't match, goes to spam folder. Beyond that, apply a bayesian filter.
I get no false positives and the spam that gets through to my inbox can be counted on one hand per quarter. Basically none.
That's yet another benefit of self hosting, since my bayesian filter is trained on my personal email specifically, it tends to become very good. Unlike generic gmail filters for example, where there'll always be some mail that ends up in spam no matter how many hundreds of times you mark it not-spam.
I have Bayesian filtering behind DMARC checks and a few other DNS and RBL checks. Still not perfect -- for all folk talk of the difficulties getting mail accepted by the big providers, a surprising number of companies still don't have matching forward and reverse DNS records :P.
One principle I follow is that I don't drop anything. Every message will either be rejected or delivered, with a narrow band of SpamAssassin scores that delivers to a spam folder.
This idea that self-hosted email is impossible is wildly overblown.
I've helped others set up self hosting much more recently, and haven't had any reputation problems beyond the early period where the IP has no history. (It is important to find an IP that doesn't have recent bad reputation, but that is fairly easy to do. Unless your host is in the business of hosting spammers most IPs will be clean.)
The reality is honestly just that self-hosting mail is not as hard as all the people who don't do it say it is.
Your ip and those of others who have sent mail to gmail have been recorded.
Your reputation score is high.
Try a new ip and see how hard it is.
The other option is to relay through Send in Blue, or Sendgrid or something like that.
The only nag is that Microsoft is EXTREMELY strict for their hosted email. It's the only provider that consistently denies recieving mail from my server when the IP range it's on becomes greylisted in UCE-PROTECT -- which happens every so often...
Easily solved with getting the MX and backup MX IP's whitelisted there, but I haven't bothered cashing out for that yet...
This is a common complaint with Linode specifically, but probably fairly common with low-cost virtual server providers in general. It's worth looking into the history of your IP before you start using it to host mail, and if it's feasible, shelling out for a dedicated server (ideally from a provider that doesn't also offer virtual servers, or has enough network separation between them) makes it much less likely that your neighbour is a spammer. Mine's never been on UCE-PROTECT.
At the very least, getting your server marked as spam/blacklisted is not inevitable. Just make sure you aren't an open relay and that you've got properly configured SPF and DKIM records in your DNS. Once that's set up you can pretty much forget about it. I haven't had to touch any of my configs in years.
Initial setup takes maybe a day or two if you know your way around Linux or one of the BSDs.
This is untrue. If you are the only person using your email server, your volume will be so low that the big providers (Gmail, Outlook, etc.) won't track your reputation. So, ironically, being a low-volume sender means your email will be constantly classified as spam.
I speak from experience: https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...
My email server is used by two people. Reputation is tracked by all the big providers, as evidenced by a) my email not being classified as spam, and b) them showing reputation of my domains in their various reputation dashboards.
"Those who say it cannot be done should not interrupt those that are doing it."
When you make a claim that supposedly applies to all people, a single counterpoint is sufficient to disprove the claim. It's as if you had said "all rabbits are black", then I showed you a white rabbit to counter that not all rabbits are black, and you come back with "look, I have a black rabbit here". How does that make sense to you?
> them showing reputation of my domains in their various reputation dashboards.
I never got access to their dashboards because my email volume was so low. If you somehow did, good for you.
> "Those who say it cannot be done should not interrupt those that are doing it."
I'm not "interrupting you from doing it". I'm interrupting you from giving bad advice to OTHER people.
Well said!
Your claim is "it is not possible to self-host your own mail on a low-volume server and not get consistently marked as spam by GMail / other large operators". The existence of a single person successfully doing exactly that (and there are numerous such people in this very thread) is sufficient to disprove your claim.
I do think that most of the effort/risk is at the beginning. Making sure you're on a reputable provider, checking the history of your IP, setting your mail server & the security features up correctly, monitoring deliverability etc.
After everything is working well, if you got that part right, the ongoing effort should basically just be keeping software up-to-date. You could always get unlucky and e.g. someone starts sending spam on a nearby IP and you have to waste some time dealing with that, but hopefully if you picked your provider well that won't happen. It's yet to happen to me, but my provider only offers dedicated servers, which are probably not so popular with spammers.
Oh, that tiny little detail where it has become impossible for almost anyone to actually send email from home, either because residential IPs are all marked as spam, or because your provider is giving you an incomplete internet connection that blocks outgoing SMTP.
> You could always get unlucky and e.g. someone starts sending spam on a nearby IP
Can someone tell me why providers ever thought it was a good idea to block entire IP ranges?
Well, by "provider" I was referring to the hosting provider. I wouldn't typically recommend hosting from home [0]. Apart from the issues you identified, over the course of 20 years I've moved house multiple times so would have had email downtime and likely had to change IP address, likely leading to reputation starting again from zero (at best) or picking up an IP with bad history (much worse).
> Can someone tell me why providers ever thought it was a good idea to block entire IP ranges?
If you repeatedly receive spam from multiple IPs within a given ASN, and the abuse contact is non-functional or doesn't actually resolve the problem, it is fairly reasonable to consider that the ASN is friendly to spammers. Blocking by prefix rather than entire ASN is a bit less of a sledgehammer, but follows the same logic.
[0] If you own your house, have no plans to move any time soon, and can get a connection from a provider who is willing to give you a stable static IP that isn't categorised as 'residential/dynamic' by the major lists, hosting from home should be fine. But that's quite a few 'if's.
Perhaps that was their claim - but I've generally read advice as: "There's no predictable way to guarantee that any given person can today take over hosting their own mail with predictable and good delivery to Gmail and o365."
So just that a, b and c have, so far, good delivery from their setup is not a guarantee that person x can just "set things up correctly" and somewhat straightforwardly get good delivery.
Last I did it, I had to go via undocumented api/pages for both o365 and Gmail in order to improve delivery - and mail that gmail/o365 smtp servers swore they accepted without problems - still sometimes ended up as spam, or simply vanished after delivery.
This was all individual low-volume. Never found any reason for it.
That said, I'll probably go back to hosting my own mail, and just live with certain parties being bad net citizens, eating the occasional mail without error or bounce. It's not like I really expect them to do better. Although especially in the case of Gmail, it's a little like Disney eating up public domain stories and spitting out copyrighted and trademarked content. Google did a lot to force people away from proper quoting (by hiding the fact of how Gmail quoted things in the "friendly" ui) and they pretty much killed Google groups - after marginalizing alternatives. But those ships have sailed.
"will" is a strong word. I've read that very low volume sending server can sometimes have issues, but never experienced it. My outgoing volume is about as low as it gets since it's just me and some family that don't use it much, but don't experience any problems.
Or maybe they have 10x the experience you do, but it was different experience for reasons beyond their control. Don't over-generalize from a sample of one. That's hubris.
You either get everything right (due diligence of your provider and the IPs they're giving you, configuration of your mail server, not sending spam, monitoring your reputation) and your mail is delivered, or you don't and it isn't. That's it. Getting everything right, and knowing what you have to get right, comes with experience.
You don't even need to set everything right. Up until very recently (months), I was sending emails from a few of my servers, and I had NOTHING set right. As in, I was sending from IP addresses that were never mentioned on my DNS, no PTR no SPF no DKIM no nothing. Just good old "here's an email from this address, trust me I actually own that address and it's legit".
And it worked just fine.
Surely just a reputation thing, as I had been doing this for over a decade, and all emails were very important (password recovery, order details, etc), no newsletter or anything.
I recently replaced all that with zoho because I wanted something a bit more secure and didn't want to configure it myself.
Have you tried setting up a brand new email server with a fresh IP from say, Digital Ocean or AWS?
I'd recommend a smaller, tech-savvy hosting provider that has been around for a long time, who has no tolerance of abuse on their network, who is easy to communicate with directly (no paying for 'Business Support' just to talk to a clueful person), and ideally who mostly rents dedicated servers / colo rather than cheap virtual servers.
Just because you're lucky doesn't mean other people are "unexperienced". I don't know what experience has to do with it tbh. Maybe if you have experience working for Microsoft and you can contact the right people there...
In my experience the difference in outcomes that people have with self-hosting mail are largely down to, from most important to least, (1) the nature of the mail they are sending, (2) their choice of hosting provider, and (3) the correctness of their mail server setup. It sounds like you've got (3) sorted, so maybe (1) or (2) are relevant here.
…except when Hotmail decided my entire IP block was barred from sending emails to them. Happened several times, I always had to wait a few days for my emails to stop bouncing.
In my experience it mostly works, but it’s never reliable.
I host my own email server with Vultr on an OpenBSD VM using OpenSMTPD and Dovecot, relaying all outbound mail through SMTP2Go (their free tier more than meets my needs). I have all of the necessary DNS entries set to mark my mail as legit, and I sign all outgoing mail using strong 2048-bit RSA keys. Thus far, I'm able to send mail and not have it marked as spam (at least to everyone that I've corresponded with thus far). It was a lot of work to get to that point, but not terrible.
So it's not an entirely self-hosted solution, is it?
Even my local ISP refuses mail from them.
[1]: https://discourse.mailinabox.email/t/digital-ocean-ips-being...
If you want guaranteed delivery with proof and tracability, send a registered letter at the post office, FedEx, etc.
If one wants such legal protections, there is the post.
(Now, should there be such a right? That's an interesting question. But a world in which one exists would raise the bar to starting one's own email server even higher).
It's real unlikely any such guarantees were made. To do so would be extremely foolish for several reasons (the false-positive rate of spam identification is known and emails can fail to deliver because of an error at either end of the transaction).
Negligent interference.
https://en.m.wikipedia.org/wiki/Tortious_interference
> I host my own email server with Vultr on an OpenBSD VM using OpenSMTPD and Dovecot
But with outgoing mail being relayed internally to dkimproxy which signs it before being relayed back to OpenSMTPD for delivery to the other email server.
I had to set up SPF and DKIM DNS records, and one time I had to request that my IP be removed from the Abusix blacklist. Other than that, it's pretty rare for my emails to be marked as spam. Outlook 365 seems to do it much more often than Gmail though.
[1] I think this is the one I had I mind, though I didn't realize it was already in ports: https://cvsweb.openbsd.org/ports/mail/opensmtpd-filters/dkim...
* https://en.wikipedia.org/wiki/Authenticated_Received_Chain
I've found ARC to fiddle some to get going than ARC.
This was my main experience, and all I did was try to set up the ability to simply send emails to myself (gmail) (and no-one else). Things like: this script crashed, or btrfs scrub finished + scrub results, and similar.
The first thing I tried was just setting up a VM with postfix running on it locally with my residential ISP. I don't even remember what the error was for this scenario, but it was just totally dead in the water. Absolutely zero mail delivery. I think I eventually figured out it's because google defers to spamhaus, and spamhaus says residential IPs = hard no.
That next thing I tried, and what I ended up doing, was writing a docker container that just runs an SSH port forward to jump from my local network to a digitalocean host, which is where another docker container runs postfix. I had done this bit once before, and I tried to just set up DKIM (since DKIM was, to my reading, basically bulletproof - why bother with SPF when you have real cryptographic identity assurance?). This led to weird error messages from google about my IP having a super low reputation. This was something I'd been worried about so I spent a bit of time trying to cycle my IP. But I eventually figured out it was just a bad error message and setting up SPF suddenly made my emails start delivering.
My main ongoing issue is that I had to add all my sending addresses (things my internalhostnamehere@myrealdomain.com) to my contacts in gmail, otherwise there was like a 50% chance they'd just go to spam. I've been running this setup for about a year and it's still a coin toss whether emails will come through fine, or if they'll say "this would've gone to spam but it's in your contacts". When that happens, I check the DKIM and SPF status in "original message" in gmail, and gmail itself says they both passed.
Absurd tbh.
For my "not self-hosted but better than letting google own my digital identity" solution, since I use apple icloud+ or whatever it's called, I set up the SPF stuff to let me send+receive email from my custom domain, so while icloud could still scan my mail, at least if I get banned, I still own the actual domain and could move somewhere else.
Outlook and Gmail are basically having opaque rules who can receive email and there is no process to get “whitelisted” on these receivers.
If you keep an eye on your logs, when your emails are being blackholed (it accepts them but it does not deliver them!) it does provide a link in one of the 550 status messages, where you can get yourself unblocked. I've elaborated here: https://news.ycombinator.com/item?id=31185297
However this only works temporarily, after a month you're back in the doghouse. Only senders which send a large volume of legit traffic are allowed. It's ridiculous but sadly true.
Edit: I found the message in my old emails:
---
550 SC-001 (BAY004-MCxxx) Unfortunately, messages from XXX.XXX.XXX.XXX weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.
---
In that link the "SC-001" code also refers to that reputation thing. This was the same at outlook.com / hotmail.com and live.com . It did not, however, affect corporate customers using Office 365 / Exchange 365. Only customers of MS' consumer offerings.
My "internet service provider" was a legit colocation service and nothing funny was going on in their network by the way. Microsoft was the only party that had issues with my server. All known blocklists had no issues with it. It was just MS being difficult and making up their own rules.
Anyway going to that link there is a form somewhere to temporarily unblock it. Give it a try.. Perhaps you can create an account at live.com yourself and send a daily test email or something... I thought of doing this but eventually I got so frustrated I gave up on it.
That’s the thing I’ll never understand: why would one have to spam email to be considered not spam?
I wonder at which point it is legitimate to go full conspiracy theory, and suspect they’re just trying to block the little players so they can keep their relative monopoly. Maybe they don’t do it on purpose, but the way their anti-spam measures make life real hard for the little ones sure looks convenient.
There are two ways to run a bot on Telegram, either by running the bot client directly (meh, interesting but extra setup) or by using Telegram's bot hosting system that works over HTTPS. It's the second approach that takes 3 minutes (!) to get to an MVP state for notifications.
- You walk through a flow with a specific account (@Botfather) on Telegram to create a new bot account, which gives you an API key
- Find the new bot using the search function then open a conversation with it and (after sending /start) send a junk message
- Call `curl "https://api.telegram.org/bot$APIKEY/getUpdates"` and fish out the "chat"->"id" value from the JSON representation of the message you just sent to obtain your user ID
- Call something like `curl "https://api.telegram.org/bot$APIKEY/sendMessage" -X POST -H 'Content-Type: application/json' -d '{"chat_id":"1234567890","text":"boop"}'` (set chat_id to your account id) to send a new message - yup, it's literally this simple to send messages
- Go into Telegram's settings and add the bot as a notification exception (assuming you have notifications universally turned off by default)
- If you also set the full-screen popup to "when off" Telegram will (even when your device is locked) show an instant notification containing the sent text
- Because this is a conversation, the message history will be preserved unless you explicitly delete the messages (which you can do on a per-message basis)
- The Telegram bot API supports both polling and push-based I/O, where you can periodically poll /getUpdates or have Telegram call a webhook you configure. IMHO the way easier approach is just running the bot client locally at that point, *but*, for just sending out one-way notifications where replies don't matter, the default polling setting (no webhooks) is ideal as the bot server will delete un-acknowledged messages after IIRC 24 hours or so - so you don't have to worry about queue quotas or whatever, you can just ignore the whole receive side and it just works
Obviously the caveat is that this is 1% of the complexity and equally 1% of the... provenance, for want of a better way to put it. But in terms of "I need realtime notifications now" I am yet to find a better system. It worked perfectly.
At work we've had issues with email delivery due to things like outdated IP block lists at some random ISP four hops away, only impacting deliverability when mail gets routed through that part of the web.
I've run into this with both Sam's Club and Speedway Rewards.
Only thing I can think of is that some outbound mail service they're using is dropping them, or some relay in the middle is dropping them... I can see where the word "spam" would be a keyword you might use, but I've had this email address for 15 years now and it's only been a problem in the last few years.
I think the only way to make distributed social media practical is to have an extremely inexpensive turnkey self-hosting solution for the average person. A Chromecast-like device that they plug into their TV that backs up all their photos, plays music, and also hosts a Mastodon instance. Some kind of very friendly backup solution where you make an "emergency contacts" list, and the device encrypts all of your data and stores it on your emergency contacts' devices as a backup, and vice-versa.
There are types of bulk unsolicited email/web-comments which involve text without links, but they aren't spam per se; they're (the initial bait emails for) scams.
(And the usual solution to scam postings on forums, is to prevent people from sharing any off-forum contact details, except maybe via forum private message. No Nigerian prince is going to run their whole scam through the forum; they want/need to convert you ASAP into having a non-intermediated conversation with them. So they won't bother with any forum where they can't stuff their email address / Telegram handle / etc into each post.)
I remember back in the day having to change your SMTP settings whenever you travelled to whatever the ISP was where you were staying. then you could finally send email from your @homeisp.example email
That would be an open relay. That is simply not something that mail servers do anymore. If one was to deliberately set up an open relay, one would find that their email server was blacklisted pretty much immediately.
I self-host my mail for over 17 years. Most of the spam I'm observing those days comes from hacked/broken websites (sometimes it's probably some stolen SMTP credentials, sometimes sent from the server directly). Legit domain name, SPF and even DKIM present, looks totally legit in this regard - only stopped by RBLs and content filtering.
Store and forward.
Do remember that email was THE great federated protocol.
The goal of a mail server was to get your email "at least one hop" closer to your destination. And that wasn't an easy task.
Servers came online and went offline. Users logged in and out. Connections came up and went down. IP wasn't the only transit. DNS? Oh, the hosts file? Even higher things--thing DECnet and Janet.
Email was barely functional most days. Your best bet if you weren't an Internet God and weren't able to write your own super complicated sendmail.cf was to know a sysadmin at a node who had an Internet God and ask him if you could forward emails that you couldn't handle to their server.
Because the people on the 'net were generally not bad actors back in the day, so we would you need to lock things down?
Unfortunately, greedy people soon jumped in to take advantage of this generosity, resulting in a tragedy of the commons.
John Gilmore used to run an open relay, and I used to get spam from it. He was really stubborn about promoting the freedom of the spammers over the peace and quiet of the poor recipients. He eventually got shut down, still complaining.
http://www.toad.com/gnu/verio-censorship.html
But if this is only a fight between admins, the intuition is that we would end up with the big instances constantly losing users to smaller ones (created by those breaking away from the bad admins) who would then federate among themselves.
The right to peer implies the right to not peer.
(1) People are turning their noses up at Mastodon because all of Twitter isn't already there and because you'll be cut off from instances that aren't federated with yours.
(2) People are worried about "all of Twitter" becoming more people than they would like. There are communities they'd rather be cut off from and words they'd rather not read.
It's not a bug, it's a feature. Unfortunately, very influential companies that have figured out how to game our attention have tricked users into thinking they want something they don't.
No? Even if you don't want to force smaller instances to peer (which generally makes sense) you can apply more strict requirements to huge instances that contain a significant portion of the population.
The experience behind this predated peer-to-peer electronic cash and related developments. You may be right, and it may still be too soon. But problems can be solved.
There is no technical solution for people being assholes.
Well OK there is - turn off computer or server :)
Looking at the 30 years and millions of dollars poured into making email work, the evidence seems to be against this
This also allows authors to seamlessly switch servers without losing audience or at least being able to recreate it very easily.
Perhaps some kind of blockchain would be a solution? (No, I'm not trying to appeal to tech investors, I actually think it might offer just the solution here :P )
I’d love a world where data was truly distributed and federated, but unfortunately, the barrier of entry is too high. Because of this people will start hosting nodes for people. Which isn’t necessarily a bad thing, but network effects will take over, and we’ll be back where we started.
Look at git. It’s distributed in all the right ways, but almost everyone uses github.
The web is decentralized, but the same few websites dominate to the point that people — even people on this very site — think that you can’t post a video except to YouTube.
Not only did Facebook and GChat refuse to peer with little players, they refused to peer among the big players too. We could have had something like IRC for the masses, peered chat servers with bring-your-own-client. Instead, we waited decades for iMessage to get Android support which only happened long after everyone moved on to IG, Messenger, WeChat, etc.
Email is probably one of the last great open[ish] distributed systems we’ll ever see. There are just too many incentives to build walled gardens instead.
Going the self-host route, I'd still want a service of some sort so I didn't have to maintain it myself. Almost like an evergreen program that self-hosts my data and synchronizes and backups and transfers anything and everything.
Everything would be accessible outside of the program as local human readable or viewable files where possible. That'd be the best way to be non-walled garden.
If only one entity does it, as far as I can imagine it is only a marketing statement to appeal to a niche demographic - people who care about it from an ideological standpoint.
If more than one entity does it, it could lower the bar for critical mass. Instead of having to get enough people on your platform to start benefiting from the network effect, you only have to get enough people on your platform and platforms that you have bidirectional integration with.
Really, a format should be created (e.g. file.[short|vine], etc.) that could be then edited by any editor and viewed by any viewer, and all that you'd need to do to copy a YT short to a facebook reel is to copy the file itself to each platform.
It's literally the same exact concept over and over again, just wall gardened instead. So much wasted development time doing the exact same thing.
I've had several email providers die since the 1980s. Each time it was a major disruption in my life. The last time, I coped by mostly dispensing with email whenever possible. Like most people I have a "good" email address for important things, that I check weekly, and a "garbage" email address I only bother to check when I have a need to.
Hosting my own mail server, not subject to some provider's ideas of filtering, or simply vanishing in the night, would make email more attractive. But the festering mess that SMTP email evolved into isn't just something you can set up in an evening. It's not even a hobby. It looks pretty much like at least a part-time job. Weighing the options, I don't really need email that badly.
[1] https://news.ycombinator.com/item?id=11795658
XMPP doesn't even support E2E encryption out of the box. It's as outmoded as IRC.
PoW has been the best solution so far.
The only technical way maybe. You could always legislate that large enough networks MUST peer.
I did some experiments back when I ran my own mail. Sending from my mail server to my Microsoft account it not only marked everything as spam, it continued marking everything as spam after I marked a bunch of them as not spam.
After that, I tried also answering several of them and composing several new mails to send to my non-Microsoft email to see if Microsoft's spam system was smart enough to figure out that if I'm actively corresponding with someone their incoming mail should not be marked as spam. It was not smart enough.
Then I tried whitelisting. Nope, still spam.
I hosted my own mail for more than 20 years. A couple years back I just got tired of trying to solve deliverability puzzles, plus the fears that deliverability issues generate. (E.g., "Did that potential employer get my email about the job?") Especially since some of the puzzles are not solvable, like why GMail does what it does. I even had friends at Google, and I still couldn't find out why GMail occasionally didn't like my server. And arguably, that's the right choice for them, as the more spammers know about how they work, the worse it is for Google staff and GMail users.
For me, switching to Fastmail hosting was a big win. It's not like I'm out of technical challenges to solve, but I get to apply that to things where the upside is greater than, "The thing everybody expects to work still works."
In addition to the iCloud, I have a free ProtonMail account, which I use sparingly, and anything large or important that comes in there I move to local backup and delete. 99% of the messages I get can just be deleted. If you set up pop3 you can auto delete from the server. That’s the old school solution, but it depends on your use case. The pop3 option doesn’t work well with multiple devices.
How many gigs? It appears that the 100GB plan is only $9/mo.
The problem with spam is that there's no real legal recourse for spam. If it's in your own country then maybe. But outside of your country? Well the easiest thing to do is to IP block and the next best thing to do (when IP block isn't an option) is to use some sort of "smart detection" to put spam into a special box labeled "spam". There's no deterrence and literally no criminal prosecution for spam.
Many of us do it. If you have any interest in the topic, either due to the fun of managing the servers and learning something along the way or due to the moral high ground of supporting decentralization above proprietary walled gardens, do it!
Ignore the naysayers, if you're interested you can do it.
Will some emails very occasionally end up in the spam folder of a recipient? I mean, yes, but that is true of everything. You can end up in spam folder sending from Microsoft Office mail to gmail or vice versa. Heck, every now and then an email from my manager will end up in my spam folder in gmail even though he's emailing me from gmail to gmail, both of us in the same corporate gsuite account! So on average, once you set everything up correctly, your deliverability will be as good as gmail to gmail, which is to say not 100% perfect but no worse than any other solution. And you'll be in control of your email infrastructure and address. No longer will google/microsoft/apple/yahoo be able to cut you off all your accounts on the whim an AI gone bad.
The parent post mentions a useful safety valve to know about if you're worried about deliverability and want to take baby steps to get there. You can always, either selectively or wholesale, use a commercial relay for outbound mail from your email server. Some have free tiers that are plenty for personal/family use.
Personally I don't use any third party relay, I deliver to everywhere from my own infrastructure. No issues.
Man, and there's such an easy solution, too - just use Hashcash[1] (invented in 1997) and 90%+ of spam disappears overnight (if not more, depending on how high you set the difficulty).
Well, ok, "easy" in the sense that We Have An Algorithm For This - it'd still be hard to get email clients/servers to agree on a protocol...
[1] https://en.wikipedia.org/wiki/Hashcash
So, this argument is completely invalid.
Not if the machine is a server and was gotten into via (e.g.) a bug in a web app. I don't know about you, but I don't keep my bank account information on the LAMP systems I sysadmin.
You missed the rest:
> and/or mine cryptocurrency directly. Regardless, their spam-sending rate will still be significantly decreased
Your argument remains invalid.
> Requires immediate total cooperation from everybody at once
False. As a silver lining to the Google/Microsoft email oligopoly, those providers could announce that anyone wanting to send email to those services will have to implement this protocol, and it could be done in less than a year.
> Unpopularity of weird new taxes
Irrelevant. No taxes involved - there's no money here, and users won't care if their mail takes an extra few seconds to send, because they don't expect email to be low latency anyway
> Public reluctance to accept weird new forms of money
Irrelevant - no new money involved.
> Huge existing software investment in SMTP
Irrelevant - a small number of server software are used by the majority of users. Also, see earlier point about oligopoly.
> Sending email should be free
Bad idea, and irrelevant, because it still would be.
I suggest you put thought into copypasta before putting it into a comment.
I feel like, if there was such a service, it would be pretty useful to use it to prevent account registrations on other services, from users whose email addresses have domains with bad reputations. After all, they'd very likely just be registering with the intent of using the service to send or post spam in some way.
Contains blacklists on the domain level, also on the ip block and AS level.
https://www.mailop.org/best-practices/
That being said, when you set it up, make sure you set up an SPF record. Also, check the IP Address to make sure it is not already blacklisted.
Cpanel makes it almost effortless to set up an email server, if you have just a little bit of tech know-how.
Curious, how easy it is to set up things like a self-hosted Mastodon? Is it a few minutes, a few hours, or a few days worth of work?
Hosting your own email is a problem because of these things, otherwise a fun exercise (that I gave up long time ago).
The point of DMARC is to protect senders from spoofing. I.e. if I'm PayPal I don't want anyone to be able to spoof email from my domain, so I set a DMARC policy to reject if SFP and DKIM both fail. If I own a domain that I never want to be used for email I can set that same DMARC policy and just neglect to set up SFP and DKIM, therefore ensuring they fail.
So far I've managed to avoid needing to relay my mail out using something like SMTP2Go but eventually I may have to. For now GMail seems to be learning when I email my regulars and Microsoft unbanned me after I joined their Outlook.com Smart Network Data Services (SNDS)
In better news, incoming mail works flawlessly. It's even spam free if you use a catch all address (dodgywebsite.example@yourdomain.example) and drop mail from any company that leaks your address out.
I assume that's because I had no reputation, although I did start on a DigitalOcean VPS before I learnt about their terrible ongoing reputation for ignore abuse reports. [1]
[1]: https://discourse.mailinabox.email/t/digital-ocean-ips-being...
Yes, spam will still make it through and you have to train the filters in either case.
- A handful of instances provided by large companies would probably crop up and end up hosting the majority of users
- Spammers would notice that they could reach a large number of people via Mastodon, and start spamming
- The providers of these large instances would moderate heavily to prevent their own instances being used for spam, and begin blocking / not federating with small instances
I should add that spam is probably _already_ a problem on Mastodon, but perhaps not to the extent that it is for email since the average Mastodon user is (for now!) way less likely to fall for a scam and therefore a much less valuable target.
I suppose it is to their credit that these instances are so transparent about their blocking policies, but I think the world would be a worse place if email or even Twitter made it impossible for people with different politics to message each other.
[0] https://toot.cafe/about/more#blocked-instances
[1] https://im-in.space/about/more#unavailable-content
Not sure what the story is with self hosting spam filtering. I think most people call an external service
What you can't do easily is talk to other large public email servers. Which I don't.
My home email server receives email I send to it, but won't forward email to other domains. This is very useful - basically anything on the iPhone is shareable via email, so now I can capture a copy of things outside of the phone easily and privately from anywhere without having to go through an untrusted third party such as Google.
In addition to having SMTP setup, I have IMAP setup, so I can retrieve email from my phone or other email client that various other self-hosted services on my local network generate, or my desktop PC running Thunderbird.
I do have to monitor logs as I get regular password-guessing attempts on the services. I have a script that checks for repeated failed logins or other bad behaviors and adds malicious IPs to a blocklist ipset.
It would be interesting to federate with others and participate in a private SMTP network. I'm sure these already exist.
Small hosting companies struggle with email deliverability, so you have to ask yourself if it's something you want to deal with. Sometimes if you end up on a blocklist there's no easy way to get off of it, and you may find yourself unable to get an email into someone's inbox until you have the time to redo a lot of work (moving to different servers, IPs, etc).
From my perspective, you're better off using your time to do literally anything else unless email is something you want a very deep understanding of.
But there is one piece of this that's ridiculous, broken and almost cruel: silently dropping messages marked as "spam" with no notification given back to the sender.
Why does this practice exist ? Who believes that this is decent or acceptable behavior ?
If gmail doesn't want my inbound message - for any reason - that is just fine.
If they drop it on the floor without telling me that is totally shitty.
Exactly what a legitimate sender wants and what a provider would not want to give an adversary. Now the adversary has to also incur a cost to determine successful deliver vs open/engagement rates make it just that much harder.
Google tell you why things are spam and often they even return quite detailed error emails when they don't accept stuff it doesn't benefit an attacker that much. Any decent attacker already knows that they should sign stuff and make identifiers align and can do so trivially easily.
People like Yahoo are the opposite and are completely opaque as if they are doing anything that clever. All they can realistically do is check originating IPs, message content, alignment etc. just like everyone else.
Since I can still a lot of very decent SPAM in my inbox, their lack of transparency clearly doesn't work so they might as well help legitimate senders to deliver stuff properly.
Where are you seeing these details ? I have never seen any bounce messages from messages I send into @gmail.com that end up in a spam folder ...
The delivery server doesn't generate the bounce message you were expecting; that's generated by your own mailserver, on seeing a REJECT status code from the delivery server.
Mailservers do spam-filtering after accepting for final delivery because spam filtering can be processor-intensive. Sometimes it's farmed-out to an appliance or whatever. To have the SMTP process suspended while Spamassassin goes through it's contortions multiplies the consumption of server resources on the SMTP server.
The delivery server CAN'T (and shouldn't) send you the desired bounce message, because it doesn't really know who you are. It can't rely on the From: address, because you could be sending on behalf of someone else.
In my view (and the view of the RFCs), if a server says "200 OK Accepted for final delivery", then it MUST deliver the message.
There's an awful lot of the kind of server-side spam-filtering that does actually involve delivering: the kind that filters mail into the recipient's spam folder. That mail hasn't been dropped on the floor. It's been delivered, just not to the inbox.
I'm willing to stipulate that this is correct and would, in my case, be a difficult problem to solve.
But nobody is losing job offers or missing kids' schedules or breaking their summer plans because of my mailserver.
I am talking about gmail. I am talking about MS (whatever it is). I am talking about yahoo.com.
Their spam heuristics are, in many cases, laughably bad - they are demonstrably, clearly broken. If I email my wife twice daily for 15 years and then one of my responses to her emails gets put in the gmail spam folder ... what words to even use for that ?
They need to fix this. I don't care how sticky of a problem it is.
Everything about megacorp spam filtering is broken.
Well, gmail, MS and Yahoo have their own ideas about what "broken" means. Google in particular forces changes to standards by simply implementing them in their own services. Those changes never make it easier for small-fry postmasters; so I conclude that Google would like all small-fry mailservers to disappear.
Discrimation through spam-filtering isn't unthinkable, and it would be hard to prove (especially if they claimed there was "AI" involved in the filters). Google used to have really good spam filtering; I can only suppose that the reason it's got worse is that they want it worse.
there's already some kind of feedback loop mechanism, but mostly available for large senders.
I guess my feeling is: if I want to quarantine suspected spam without telling the sender, that's my prerogative -- why does the sender get any say in this?
If they don’t do that and decide whether checking is needed based on presence of some mail headers, or using heuristics on the subject line, spammers will start faking returned messages.
I think that makes it expensive to return spam. https://99firms.com/blog/spam-statistics/ says about half of all e-mail is spam, so returning that would increase e-mail traffic and spam checking by 50%.
No, it's the ONLY reasonable thing to do when something like 98% of all SMTP traffic on the Internet is spam.
If mail administrators bounce back an explanation for every "bad" message:
1) Their outbound mail volume would go through the roof.
2) The host sending all the bounces would look like a spammer to _other_ automated spam-classification systems.
3) In the unlikely event that a spammer actually reads bounches, they could use the feedback to tweak their systems to avoid the spam filters.
Nobody is talking (I don't thing anyone is) about sending back a bounce message, that would indeed make no sense.
A responsible email server should:
1) Reject the email during the SMTP conversation if it's going to do that. Then the sender knows it didn't go through because it got the error code. There's nothing to bounce back.
2) If it accepts the mail during SMTP conversation, then always deliver to the recipient.
2a) Some disagree, but I think it's totally fine to deliver it to the recipients spam folder if post-processing determines it might be spam. That's not wonderful, but it still got delivered and the recipient can go find it in the spam folder. Most people are used to looking there regularly anyway since many of the larger providers (coughgooglecough) have such terrible false positive rates. The important thing is to never lose email.
What's never ok is to accept the email during SMTP and then silently file it in /dev/null.
0: https://cdn.discordapp.com/attachments/468851579487780905/96...
The blacklisting shouldn't be permanent, though; my self-hosted mailserver took about a year to earn the trust of GMail. I hzave no idea how Hotmail's spam filtering works. I think they just shitcan randomly, unless it's sent from Hotmail.
I'm no longer running a self-hosted server. It's hosted by my (obscure) ISP, but it's still a custom domain. I have no deliverability problems, not even to Hotmail.
You may wish to consider using something like a Synology NAS where a stripped down mail server is a free feature for 5 mailboxes or less. They also support DDNS...
And when spam levels get high, a quick analysis of source IP addresses gives me new entries for a block list at my firewall. I wrote a simplistic visual basic script to harvest the IP addresses, since I still use Outlook as my PIM.
The only question is how do you stop spammers walking away with their stake and creating another domain? There needs to be a way to slash someone's bond, which requires some sort of consensus. With DKIM signatures, it's fortunately possible to cryptographically prove that a given email was sent by a given domain, but ultimately you need to give potential censorship powers to some entity.
My suggestion for doing that is getting the ITU to vote on a set of maybe 7 organisations (e.g. mail providers, universities, non-profits) who share spam reports with each other and can slash a bond if at least 5 of them agree. Of course individual mail providers would be able to override these decisions and continue to accept emails from the blacklisted domains (and they could obviously continue to use other forms of spam filtering), but ideally the bond slashing mechanism would only end up being used once as spammers tested it and then gave up.
This is why I think it makes sense for an email-by-email basis if you also enforce a deposit and withdrawal delay. It's the simplest solution that I think could work. Not perfectly, but who's willing to spend $100 per email account with a 2-week delay between addresses? Add a public blacklist where you can vote for people as spammers and implement on your email services if you so choose, and I think you have a pretty good system. You may well be right that there's an elegant domain-based system with slashing and consensus but I haven't thought about it long enough to think of one myself!
And yes you could do this off of Web3 but then you'd need an escrow account and a centralised party to hold the funds which isn't as decentralised as I was hoping for (at least with Web3 the stake remains in my wallet), but definitely possible!
Edit: Maybe you could do it at an account level - then if you get blacklisted you'd have to open a new wallet meaning not only the 2-week delay and $100 cost but an additional network fee. That means it's unsustainable in the longterm.
Right. In fact I was already thinking that this was a (rare) situation where a smart contract on a blockchain would make sense, because you want transparency and consensus and low-throughput financial transactions between countries that don't necessarily trust each other (or even have banking connections to each other).
I wouldn't necessarily call that Web3, especially as there wouldn't need to be any web servers involved, but I suppose the system would affect people's webmail, so I can't really object to the label.
If you have a static IPv4 in a range that is not actively hostile, and you have proper SFF/DMARC records, things should generally work out?
And otherwise, services like https://www.mailchannels.com/ should help? (Still, you will need proper SPF records.)
I've literally had a 95+% delivery rate from users in actual Lagos Nigeria using the strategy outlined above.
The only problem with email self-hosting is just how many moving parts are involved in a typical setup if you're using tools from unix-land. You need many different programs to work together in a typical setup:
- postfix
- dovecot
- spamassassin
- fail2ban
- kerberos
- ssl+tls
- etc..
And you have to know about how unix account security works because some of the older programs haven't been updated to use modern authentication mechanisms and so they need to be isolated and carefully managed, etc.
The other problem is DNS/verification. You have to set up your DNS records with arcane configuration options that are not well documented in order to play along nicely with the email community and not get blacklisted/blocked.
Some projects have popped up to try and offer containers that have everything pre-configured. ymmv.