I found your link interesting and informative. I'm curious if you have any specific comments on this quote from the story:
> From there, the email services which implement MX2 would publish a public date, on which all messages sent to them by the old MX record, will be automatically sent to Junk. If just Microsoft and Google alone agreed on such a date, that would be 40% of global email traffic.
Do you think this change wouldn't be enough? If so, why not?
MX records don't send messages. Assuming that “sent to them via SMTP” is meant… well, moving all messages to ‘junk’ isn't a good idea: it needs to be restricted to messages sent on or after that time. But why not just respond with “554 No SMTP service here” on opening the connection?
A solution that is 100% interoperable with the existing widely deployed solution but also offers substantial advantages can have good chances to coexist with the old solution for years if not decades, supplanting it only slowly.
Examples of success: monochrome TV -> color TV, landline phones -> mobile phones, the Windows 3.x -> Windows 9x -> Windows NT/2k/XP transition.
Email has a really well-working transport layer. The UX layer can see some innovation while staying compatible.
The above situation is acting like a pseudo-2nd-gen email standard that's evolving in slow motion (without us officially calling it "gen v2.0"). Those email policy changes will be adopted because big cloud email providers like Gmail and Yahoo have a massive influence on the entire email landscape.
Therefore, a new hypothetical "MX2" standard that was more coherent with better authentication, anti-spam, and anti-phishing features could be promoted by a consortium of Google/Microsoft/Yahoo/Apple. The smaller players like Fastmail, ProtonMail, Mailchimp, Sendgrid, etc and most everyone else would all have to follow the cloud email providers' lead because everybody wants to be able to send email to them.
> By simplifying the stack to the above, eliminating SPF, DKIM, and DMARC (and their respective configuration options), and standardizing on one record (MX2) for the future, running your own self-hosted email stack would become much easier. Additionally, the additional authenticity verifications would hopefully allow spam filters to be significantly less aggressive by authenticating against domains instead of IPs.
This assumes that the reason antispam tools make it hard to host your own email is because of spam. That’s only part of it. The other part is that deliverability is a cartel, and two of the biggest players, Google and Facebook/Meta, wish to be the intermediary between you and your audience and sell you access to their eyeballs.
Truly federated email allows you to communicate directly with your audience, bypassing apps and paid ads. This is a threat to their business models.
Additionally, domains are cheap, and moving the trust decision to domain from IP doesn’t get you much. Unknown/fringe domains will still land you in spam by default, same as non-deliverability-cartel IPs do today.
I was wondering about this as well, I think he is suggesting they report the spam to your domain registrar, who either does something about it or there whole cert chain gets blocked by default.
Yeah don't see how domains would be that much better than IPs but it would be easier to understand. It's already quite common for email senders to use multiple domains when they're concerned about deliverability, and domain reputation is already a factor I believe.
Re: deliverability, I do think incumbents are benefiting from the complexity of email, but I'm not sure I follow your argument about Meta.
Nobody would use Gmail if it didn’t deliver messages from Facebook, same as how nobody would use iPhones if you couldn’t install Instagram and WhatsApp on them.
Google and Meta are in the selling-access-to-eyeballs business. They don’t need to explicitly collude to keep others out of your inbox, their interests happen to be aligned here automatically.
Some good ideas in there I think. But I think unless you can shoe-horn them into the existing MX record and get piecemeal buy-in, the results will be quite similar to what we see with IPv6 vs. IPv4. No?
email has already been sufficiently captured by the big guys that if they chose to support this, there would be buy in. and without their support it's dead.
if outlook and gmail announce that emails that aren't MX2 get ranked more harshly in their spam filter, everybody will adopt it. if they don't do that, nobody will.
I don't see the problem with implementing it via the MX2 option. That makes backwards compatibility a lot easier than messing with the existing MX option. It also means if the proposal goes over like a lead balloon it at least won't have made the existing situation worse.
Good luck convincing Microsoft to implement anything you're suggesting in Outlook though. Or for Google to add it to gmail.
Seeing how we got a brand-new and very shiny `HTTPS` RR type instead of using the existing `SRV`, I'd say the hope for improved email is not zero. Still very close to zero, but not zero.
Some great thinking here, and the kind we need. For all the bellyaching about HTML and email, the real problem is that nobody has been bothered to build a standard that everyone agrees to.
The gradual approach is a smart one, too.
I think AMP for Email has some great ideas but bad branding. It could be a useful starting point for this discussion.
Are you familiar with what AMP for Email is about? It’s dynamic content. That’s all. And that’s something that has no place in email.
AMP for Email is dealing with a completely different problem from the one discussed in this article—one that no one asked to be fixed, and which few people even agree is a problem (and they’re all trying to sell you something).
One additional point. This gatekeeping about what email should or should not be is just too much sometimes. It is the very reason why email has been in stasis for so long.
Anyway, you understand why I brought it up, right? It is an attempt at a standard in email that already exists. When we are talking about improving email, highlighting existing work is useful. That means there is something tangible that can be used to improve the weaknesses in the current model.
Additionally, he brought up HTML email in the post. AMP email is an attempt at standardizing HTML email. That is relevant to what he wrote.
It was Google trying to shove through something they’d invented, using their market power as leverage. You don’t make good standards like that.
It was also not at all about standardising HTML email—it didn’t improve anything in that way, except insofar as the AMP part being chosen implies that the client has decent HTML support. AMP for Email is purely about dynamic content in emails.
And the very way that each provider that supports AMP for Email has required whitelisting of each sender shows there’s something extremely rotten about the entire thing.
If they keep the original From: header they would not work, as the sending (mailing list) server would look like a forger, so the ML software would have to do a rewrite that header.
If you wish to keep original From: headers, then ARC would have to be incorporated into this proposal:
> A standardized HTML specification for email; complete with a test suite for conformance. Or, maybe we just declare a version of the HTML5 spec to be officially binding and that’s the end of it.
keels over in laughter
Oh, there's been quite a few attempts at at standardizing HTML rules for email. I was even in one of them, which petered out very quickly.
Functionally speaking, the problem is you have three groups of people with HTML email (well, four groups, if you include the people who wish it died off). You have the marketing folks, who want HTML email to work essentially exactly like regular webpages so they can do all their normal design stuff and get it to work. You have the MUA implementers (particularly webmail), who need to aggressively sandbox and sanitize the HTML because to do otherwise is to risk security leaks. And you have Microsoft, who needs to keep visual compatibility with Word HTML because their userbase would flip out if they broke stuff. These groups want different stuff from HTML, and they're not going to do a good job of reconciling their viewpoints with one another.
1. Yeah the marketing folks would hate this but we need to use a subset of HTML.
2. Ok so these are the people who are going to want to help us and they run the largest email servers at the moment
3. I think MS products already have web views in them (either edge or old edge), if we had a change over like MX 2 maybe that would be an acceptable trade off to break the old.
We don't need a subset of HTML. Actually, we need Markdown emails. You can format stuff that needs structure, but not abusively so (no blinking marquee banners in eyesore colors), it is sufficiently compatible to plaintext that you don't even need the text alternative mime object. It is also more compact than HTML.
And before somebody says "won't fly", all those fancy new "will replace email someday" messengers use markdown or some parts of it's formatting.
You know what? We had that. Markdown was modelled after it.
I would also state that Markdown itself is completely unsuitable for the purpose. You’d need to design something new which might look very similar to Markdown, but which would have basically no shared behaviour with even CommonMark as regards parsing, since you don’t want HTML to be a thing. Markdown itself is seriously compromised by its HTML basis.
I can’t actually think of a single comparatively-mainstream messenger that uses even a variant of Markdown; rather, they use their own lightweight markup languages <https://en.wikipedia.org/wiki/Lightweight_markup_language> that are very clearly incompatible with Markdown. (It’s also often a frontend editing feature that gets turned into something like HTML after that.)
text/enriched has been around for decades, and supports basic font styling (bold/italic/underline, color, font face, font size) and that's basically it.
Actually, text/markdown exists as well. However, the definition of markdown syntax is, um, less than precise: https://daringfireball.net/projects/markdown/syntax (the text/markdown RFC literally has a parameter to indicate which flavor you meant by markdown!). And it incorporates HTML too, FWIW--legal HTML fragments are legal markdown as well. Honestly, markdown's million variants makes the HTML support landscape look uniform.
Microsoft has done more damage to HTML email than everyone else put together. They’ve single-handedly held it back by at least ten years (maybe fifteen), and created tens or hundreds of thousands of jobs.
In Outlook 97, they used the MSO renderer (Microsoft Word) for editing and presentation. It has an incomplete and buggy implementation of HTML 3.2.
In Outlook 2000–2003, they did the obvious sensible thing: ditch that, and use MSHTML (Internet Explorer).
In Outlook 2007, they switched back to MSO, for reasons that never made a skerrick of sense to me (their explanation was vague nonsense that included the word “security”, but the articles that discussed it vanished from the web long ago so I can’t point you to any). I believe they still use the MSO renderer to this day. Windows Mail still embedded MSO. I think that the new Outlook client they released last yearish? was still using MSO, though I’ve heard claims to the contrary as well.
The MSO component has had, I think, approximately two changes in the last 28 years, one of which was supporting high DPI (… which it does imperfectly) and the other I forget.
Since one of the major email clients is still using a dodgy implementation of 1997 web standards, what incentive have other providers had for supporting newer stuff?
We’re slowly getting places, but when you contrast it with the web’s pace, in both backend and frontend (e.g. HTTPS deployment, and new CSS/HTML/JS features)—well, it’s very obviously a completely different environment.
They were embedding IE — I imagine their security concerns were well-founded. But even gmail has crappy html support, and that runs in a friggin’ browser!
MSO probably had security problems at least as large.
And if there were security issues, they needed to fix them for IE’s sake already!
I don’t remember the details of what they wrote, and wasn’t able to find it even five years ago, but I do remember that the reasons claimed just made no sense.
> MSO probably had security problems at least as large.
Down in the parser and such, no doubt about it. But it would have also lacked much of the attack surface of IE, such as, oh, ActiveX. Granted that specific example would be easy enough to disable, but that's just one mine in the whole field. They definitely should have wrestled IE into shape, but the IE team clearly wasn't taking marching orders from the Office team. Organizational dysfunction manifested in sofware.
- no separate body for rich html and plain text, This will just result in "Please enable HTML rendering to view this message"
- maybe we need to define a subset of HTML (no external resources, css etc..)
And require that if any colours are set, both foreground and background are set. (I've seen too much breakage with assumptions about one or the other.)
Not useful proposal. Markdown is both limited and ambiguous. It's okay for writing by hand more naturally than HTML, but the sender should parse it and then explicitly indicate which characters are bold, which ones are underlined, etc, and you're back at a subset of HTML.
The main issue is whatever replaces email MUST be intentionally asynchronous. There are a 1000 email replacements. The problem is they try to be better. No one wants email to work better than it does from a performance or function perspective. Email is literally the last tech bastion keeping us from 24x7x365 work days as matter of default procedure. Everyone knows this deep down and thats why they refuse to learn or adopt the better systems that already exist.
I think it's important to recognize the article does not suggest any obvious user-facing changes to how email operates; only technical ones that would hopefully fix the many pain points there.
All the behind the scenes failures are also a feature. Diffusion of responsibility, plausibly deniability. No one needs "mgmt" to be able to nail their coffin easier than they already can.
99% of the world doesn't work in HN land of high pay for high expectations. Most places are trying to force people to work 100hrs a week for as close to min wage as possible.
Unless countries start enacting actual labor laws with teeth I stand by my statement that no one really wants or needs email to work better, there already are alternatives.
Get a few years on you and the Grind-set will fade as you realize you don't want the only thing in life to be work. You will be glad email still exists.
I don't personally think it sucks. There are bike-shed levels of improvements that probably any person on HN could dream up to make it more effective and work more reliably.
Well, for a start, there is no way of knowing if a message I sent actually get delivered.
Messages sometimes take minutes or even go up a to come through for no apparent reason what so ever.
Massive spam issues. Headers are trivially forged
People can sign up other peoples addresses to mailing lists, and if that 3rd party is a company, good luck not getting re-subscribed to random stuff for the rest of eternity.
> Well, for a start, there is no way of knowing if a message I sent actually get delivered.
Every major email client has supported read receipts for a very long time now. It's just rarely enabled by default outside of corporate environments (I would guess in part because users find it invasive).
Precisely. All actual communication (family, friends, etc) has moved to text, facebook messenger, etc, in large part because of how crappy email is/has gotten.
> large part because of how crappy email is/has gotten
The funny bit here is that email itself hasn't actually changed at all (at least not in ways that affect personal communication). It's just that folks have become used to the intrusive instant gratification of push-based chat systems.
Email sucks, sure. But it sucks less than any of these alternatives:
- Corporate support chats (no way to export/capture a papertrail as a customer, usually horribly brittle)
- Proprietary/company-specific messaging solutions in my account on various sites (no notification channel for responses, often no papertrail for me either)
- Contact forms (unidirectional, not standardized, no papertrail for me as a sender)
- Push notifications (single-device only, no reasonable inbox management, headline-only)
- SMS (just no on so many levels, most importantly that I don't want everything to be tied to my phone/phone plan and that I can't own my phone number in the same way that I can own a TLD)
The EU (or Germany, I still haven't found out) mandates companies to have a support email address, and it's just so much more pleasant than the US pattern of providing only phone support, a horrible support chat experience, or a mailing address.
The person you're responding to is saying that email sucking is a feature for the user, preventing them from being always at work. That making email like Slack would make it less appealing, and that's why nobody wants to replace it with something quote unquote better.
While I understand the unspoken rules and norms of text messages, and chat messages are significantly different from email there is nothing prohibiting one from treating chat messages as asynchronous. You can train recipients the same you train a dog.
Yes, for me (as a recipient) it's about training the senders. Keep their expectations for fast replies low by developing a track record of not responding to texts for variable amounts of time, that sort of thing.
Would be great if Google/Apple/Microsoft could lead some of these Internet upgrade initiatives. I think we have more cross corporate collaboration on Emoji selection.
They did. Google's one is called Gmail, and Microsoft's one is called Outlook 365. They each want you to use their product and offer you improved service if you only use their product and talk to other people who are using their product. But Apple got the best lock-in with iMessage.
I just looked through my Spam folder. I have 48 messages in the month to date. 18 of them are from at least ten domain names matching /^(marketexec|marketexecmail|market-exec)?\d{1,2}\.co\.uk$/. Domains cost money, but not that much, and people do do things like this.
(There’s also the problem of senders aggregating messages from untrusted parties, so that reputation is always a tricky balance. For example, I have four spam messages from gmail.com this month. When I’m not getting bursts like this marketexecmail stuff that started two weeks ago and will probably stop within another couple of weeks, I find that at least a quarter of my spam messages come from gmail.com, outlook.com or hotmail.com; in January, it’s 12⁄46, and at least four more are from other general-public domain names that I recognise.)
Honestly, I wouldn’t consider this trolling. It’s a low-effort dismissal here (probably why it’s been flagged to death), but it’s still insightful and accurate. The first time it came up, certainly over twenty years ago and I’m not sure how much more (but you can find the form from at least that far back), it was honest weariness with claims of having solved spam.
If you’re not familiar with these things, the keyword is FUSSP (Final Ultimate Solution to the Spam Problem).
Proposes to replace SPF, DKIM, and DMARC with a brand new signature scheme that Trust Me, Will Work This Time. And because this scheme is of course innately perfect, hardwires a reject policy.
Mixes up MUA and MTA technologies as if they were even the same ecosystem.
Yeah, email needs replacing, but it needs actually serious proposals.
Yes, please. Every day we don’t fix email is another day organizations will continue shifting to phone numbers as identifiers, which is 10 times worse.
There's a whole world of email mess that the article didn't even touch on: ambiguity, for example by repeating headers with conflicting values.
For example, you could have, in an email:
Content-Type: text/plain
Content-Type: text/html
now, maybe a virus scanner will think that the email is plain text, but the recipient's email reader parses it as HTML.
Same potential confusion with Content-Transfer-Encoding.
And then there's the matter that headers are supposed to be ASCII-only, but what happens when you send header values with 8 bits set?
And then there is how email threads work. Most of the world uses "In-Reply-To" and "References"-headers, but Exchange, in its infinite wisdom, decides to ignore them and has its own proprietary headers.
Basically every aspect of email has very annoying problems, and every attempt at a version 2 needs to either ignore some of them (annoying those particularly invested into this problem), or reinvent the whole wheel, running in Second System Syndrome worries.
"A federated protocol is a protocol (defined next) that makes it possible for servers to communicate with each other, regardless of who is running those servers."
I think it's important to distinguish between application layer and lower layers of the stack here. For the lower layers (firmware updates for protocol changes notwithstanding) it is basically all federated, as you say. But at the application layer there are protocols that you can participate in just by knowing the protocol, and others which require knowing a secret or getting included in trusted peers or otherwise filtered to effectively make them proprietary and not federated.
Federation implies delegation from a central authority, which makes Mastodon and ActivityPub more “confederated” than anything. Inasmuch that MX records stem from the root servers, email is federated that way, but otherwise it’s decentralized, with every MTA being its own authority. The line is fuzzy.
and they are almost more powerful than email. Look how widely Discord is used, compared to email. Email might have more sites, but outside of corporate reminders, more communication happens on Discord.
Meanwhile, if you can't use discord, then You're locked out of all of discord forever. And discord is like the 10th Gen of "hot new chat apps", what do we do when it starts to die (because Discord will probably start dieing in the next 10 years)? Just lose everything in all those discord groups, like we lost ICQ and MSN messenger and AIM and Skype?
A quick google seems to suggest around 350 BILLION emails are sent PER DAY. The statistics I could find show discord has between 0.8 and 4B messages/day, and those will be far shorter in content than an email. It’s two orders of magnitude behind.
Yes a good amount are spam, marketing, scams, etc. But that just comes with the fact that the platform is free and open.
Very strange to me to make such a barrier with corp communication. That’s communication too, and most of these corps now have things like Slack aka Discord anyway.
And if Discord decides to ban you due to some kind of automatic flagging, you're suddenly unable to communicate with everyone who insists on using it as their sole means of communication.
It's drastically different from Gmail. If Gmail bans you, you can go and get a different email address (and if you're not naive, you already have your own domain name for email even if you use Google for it). And you would be able to send messages from that different address, even to Gmail users; and they would be able to send messages to you.
The problem isn't the federated nature, it's the combination of:
- loose standardization, and lack of proper versioning
- Postel's law, which is a recioe for disaster
Takeaway if you want to design a federated protocol use a “the server chose violence” approach and reject any messages that is not perfectly compliant with the expected input for the given version (which must be part of the protocol itself).
Basically every issue with purported email alternatives that various parties are trying to shove onto me (usually companies I need to communicate with for work or customer support purposes) is a direct result of them being non-federated, or that purported solution being SMS.
I missed that little detail, but even then, it doesn't matter. That just becomes a parse error and any half-decent parser can recover from that error. The important bit is that the spec doesn't specify how to recover, unlike html5, where it goes into great length to specify recovery so browsers are somewhat consistent. So, as the OC mentioned, some systems will recover by keeping the first one and some by keeping the last one, making it so various systems show different things.
Replaced a complete mess with a little less complete mess. Nobody calls it Atom feeds as well, leading many people to still offer RSS despite it being the worse protocol.
Replaced a complete mess with a little less complete mess. Nobody calls it Atom feeds as well, leading many people to still offer RSS despite it being the worse protocol.
Trying to wrap my head around how bulk email providers (Salesforce, etc) would operate in this? I guess you could give them a subdomain of your main domain and have a separate MX2 record. This might help silo off newsletter, transactional, and actual person to person messages like many already do in the current environment.
It's a little confusing, but my idea is that there would be multiple MX2 records for every authorized sender's key. One of those MX2 records would have a marker on it for incoming mail.
Ah, gotcha. Feel like I've got the existing MX stuck in my mental model. Honestly with all of the existing DNS record based "authentication" with SPF, DKIM, DMARC its already a mess for bulk senders too lol.
Thanks to the OP for writing this up. I'd love to see mail be rebooted.
A few thoughts:
1. I don't know if HTTPS is the best analogy, since it took 20 years for it to really become widely used (vs. non-secure HTTP). I think a better analogy might be the transition from HTTP/1.1 to HTTP/2 (or HTTP/3?).
2. I don't think we can rely on large providers (e.g. GMail) to quickly adopt a new system and even issue switchover ultimatums. A simpler mail system may not be in their best interest. Aiming for slower, organic growth could be more realistic. Even if a next-generation mail system never becomes dominant, it could still provide value. (Maybe Mastodon/ActivityPub is the analogy here?)
3. In any sort of reboot, I think a modern encryption system (something like Signal protocol?) would be a must.
4. Can we have a system where senders need to hold tokens to authorize them to send a message to a recipient? The whole "anyone on earth should be able to send you an unsolicited message" idea is ultimately the source of the spam problem. Messaging systems that rely on bidirectional agreement have much less of a spam problem. (Obviously this raises a slew of other technical and UX questions...)
I know that making too many changes is a risk. But I think that there is also a risk in making too few changes. (If the value proposition isn't sufficiently bold, this may hurt adoption.)
IMO, the lynchpin to HTTPS accessibility was that we made cutting certificates easy, and most importantly, free. By eschewing the baseline requirement for cutting a certificate to being able to validate that the site on the domain is trusted by virtue of a relationship between the hostname and the certificate requester, and providing an API to automate the renewal process, opting into HTTPS went from "administrative burden that needed a sysadmin to manage" to "service any hosting provider could leverage". LetsEncrypt and the ACME protocol, and Cloudflare before them, did much to radically change the landscape for making HTTPS accessible to everyone. [1] That makes the analogy pretty apt to me, IMO.
2. You make a very valid point here. By virtue of letting SMTP languish so long, bandaiding it as we went, instead of opting for a major refactor of how email works, we made an environment that was rife for consolidation. As a result, it makes sense that we might have to boil the frog to make a change.
3. I'd love to see it, but I'd be interested in learning how key exchange/trust works. I'd be interested to see how you'd envision enabling new senders to send you mail.
4. This feeds back into my comments on 3. With established contacts, this is great, but in a world where, say, you were giving a talk, and wanted to offer folks that listened to your talk a way to contact you afterwards, how would you distribute the token? If someone abuses the permissions, how do you invalidate it? I can't foresee this being an all-or-nothing system, and don't really see a system where creating some sort of one-way or two-way trust between individual senders and recipients is at all feasible.
>In any sort of reboot, I think a modern encryption system (something like Signal protocol?) would be a must.
Signal doesn't strike me as a very suitable candidate. It is quite connection oriented and email is not. For example, it requires a online "prekey" server just to do encryption.
We already have two protocols for encrypted email (OpenPGP, S/MIME). I can't see how another incompatible protocol would help. It seems to me that the first problem to address would be the usability of encrypted email, which is quite poor. While we are at it, we should work on the usability of encrypted instant messaging, which isn't that great either, for most of the same reasons.
Something clicked when he was saying that SendGrid and Mailchimp are clean, and it remained till the end.
Email born as internet, something decentralized. But it is getting harder and more complex to have your own mail server, because you have to comply with a lot of requirements of major mail providers. For big scale servers, and commercial mass sending companies ("legal spam") they can afford to comply with all those requirements. And about ("not legal") spam and malware senders, or they don't care if they reach a limited set of target, or the reward is high enough to try to trick the system.
So the smaller mail servers, without so many users, or without so knowledgeable maintainers, if any, are getting expelled from the game by both bad and big players, some just move their email administration to some of the big providers (and privacy and territorial requirements may be a problem here). What the article proposes is another change to push things in the same direction.
And, for good and bad, some of the present use cases may be harmed by this proposal too, like devices and other simple notification services, or mailing lists.
It's the same on social media and the Internet at large, until the Elmo's tantrum accidentally popularized Mastodon. Sure, you don't have to do anything to be visible in a web browser (except you do, since you need a domain name and a Let's Encrypt certificate) but you have to obey Google's rules to be searchable at all, for example, and everyone's just browsing the same big websites all the time except when they click on an outlink, so you have to go to those websites and post outlinks to yours if you want any traffic, which is likely to get you banned for self-promotion even if it was something people actually wanted to read.
Basically everything's centralized now because bad money drives out good, and I don't have any ideas to fix it. It may be fixing itself to some extent as some CEOs keep banning their anchor users in strange tantrums.
Email is one of the services I would like to self host for small projects, ran into issues immediately for spam. I like the idea of using public keys tied to a domain, but feel the email service providers have little incentive to adopt this because the complexity adds value to their service.
This is an interesting thought experiment, but I really want e-mail to stay mostly as it is today, except with a nicer way for me to make sure SPF/DKIM/etc. are enabled sanely for small domains.
What do you think needs changing in the standards, rather than simplified implementations? That is, implementations that are suitable for small- and medium-scale turnkey deployments.
That is a very good suggestion and it is my highest priority as well, in terms of "thinking about improvements to email".
I do understand the attraction of highly specialized, modular and pluggable tools - so I know why OpenSMTPd does not have a built-in nameserver, spam filter or DKIM service. The architects are thinking about correctness and scalability and modularity, etc.
But Oh My God it would be so much simpler - and comprehensible - if there was just one package with just one config file.
If the only purpose of a particular nameserver is authentication tasks for a mailserver, we should consider moving the nameserver into the mailserver. The same goes double for dkimproxy.
You're describing new solutions like Stalwart and Maddy. Which try to automate away a significant portion of these issues and maintainability concerns.
The best thing you could do to accelerate the rollout of MX2 (or any email replacement) is to have Congress bless it as a secure channel for Private Health Information
There is no reason why this couldn't exist in the current e-mail technology. In fact, as someone who has written HTML, I'd rather not have HTML anywhere if possible! It's a terrible, nonsensical technology. A simple XML format for display text/images would be much easier to deal with, and also easy to transpile to a HTML/CSS subset. I'd like to hope that e-mail will last longer than HTML!
I think the biggest problem with e-mail is that the lack of inter-provider communication protocols, something that distributed social media (the fediverse) has. For example, there is no way for a provider to know whether or not an e-mail sent was opened, so EVERYONE uses tracking pixels! Like, just put this into a protocol already! The problems you're talking about with HTML support are simply because there is no way for a provider to announce what HTML it supports, partly because this depends on the client that opens the e-mail, not the server that stores the e-mail. But even then, seriously, this should really just be a setting per account and if you use Thunderbird it announces to everyone "hey this dude uses Thunderbird, so don't use fancy HTML!" and if you switched to something else you would just start getting different e-mails. You can see why this wouldn't work with HTML, since HTML is a terrible language that has no actual support for being displayed in a zillion different clients despite what it claims.
Return receipts are already a thing aren't they? But most people don't actually want to be tracked, so the correct thing to do there is not allow any external resources to be loaded from an email, and to make user tracking illegal/codify that stalking millions of people doesn't make it okay; it makes it millions of counts of stalking and harassment.
HTML already gracefully degrades. If you don't support a feature, you just ignore it and continue. It works just fine unless you're shooting for exactly matching some figma design on all clients, which is exactly what HTML is not supposed to do.
1) No one changes established tech (software/hardware/services) unless the "new new thing" is 10x better/radically different than the old.
2) "Bullshit talks, code walks."
Here's how something like this becomes an actual standard: Someone codes up a prototype, and gets others to use it. Enthusiasts join in and improve it. Five to 10 years or more passes and if the new standard is actually worthwhile, it will have grown to the point established organizations finally pay attention and - if it's in their best interests - will adopt it.
Take git as a prime example. Launched in 2005, it was still at only 42% adoption by 2014 (according to Stack Overflow yearly survey). A decade after that, however and it's at more than 95%. It took nearly two decades for one of the most quickly adopted technologies in the 21st century to become ubiquitous.
The past/quoted reply should be a separate part too. The main part should only contain the new content.
Potentially even the signature should be its own part.
If we’re really going to mx2 there are so many things that could be done to improve things, but at minimum at the protocol level, it’d be great if there was a two way street to let ppl know you’re treating or reporting them as junk. This not only is great signal for advertisers to stop spamming me, but also let gmail know some account they have is sending out phishing emails (domain reputation is meaningless here).
the spam filter algorithm can be provided with information on the number of attempts, and the explicit user intervention of marking something as spam, so what if the spammer learns they exhausted their attempts or user tolerance?
310 comments
[ 4.4 ms ] story [ 276 ms ] threadhttps://en.m.wikipedia.org/wiki/Lindy_effect
> From there, the email services which implement MX2 would publish a public date, on which all messages sent to them by the old MX record, will be automatically sent to Junk. If just Microsoft and Google alone agreed on such a date, that would be 40% of global email traffic.
Do you think this change wouldn't be enough? If so, why not?
Examples of success: monochrome TV -> color TV, landline phones -> mobile phones, the Windows 3.x -> Windows 9x -> Windows NT/2k/XP transition.
Email has a really well-working transport layer. The UX layer can see some innovation while staying compatible.
The key argument is that email is already getting "2nd-gen'd" -- but it's happening in bite-sized and incoherent steps that don't fully work.
Example of new 2024 DMARC/DKIM/SPF requirements from Google & Yahoo: https://www.google.com/search?q=yahoo+gmail+new+senders+dkim...
The above situation is acting like a pseudo-2nd-gen email standard that's evolving in slow motion (without us officially calling it "gen v2.0"). Those email policy changes will be adopted because big cloud email providers like Gmail and Yahoo have a massive influence on the entire email landscape.
Therefore, a new hypothetical "MX2" standard that was more coherent with better authentication, anti-spam, and anti-phishing features could be promoted by a consortium of Google/Microsoft/Yahoo/Apple. The smaller players like Fastmail, ProtonMail, Mailchimp, Sendgrid, etc and most everyone else would all have to follow the cloud email providers' lead because everybody wants to be able to send email to them.
This assumes that the reason antispam tools make it hard to host your own email is because of spam. That’s only part of it. The other part is that deliverability is a cartel, and two of the biggest players, Google and Facebook/Meta, wish to be the intermediary between you and your audience and sell you access to their eyeballs.
Truly federated email allows you to communicate directly with your audience, bypassing apps and paid ads. This is a threat to their business models.
Additionally, domains are cheap, and moving the trust decision to domain from IP doesn’t get you much. Unknown/fringe domains will still land you in spam by default, same as non-deliverability-cartel IPs do today.
Re: deliverability, I do think incumbents are benefiting from the complexity of email, but I'm not sure I follow your argument about Meta.
Google and Meta are in the selling-access-to-eyeballs business. They don’t need to explicitly collude to keep others out of your inbox, their interests happen to be aligned here automatically.
if outlook and gmail announce that emails that aren't MX2 get ranked more harshly in their spam filter, everybody will adopt it. if they don't do that, nobody will.
Good luck convincing Microsoft to implement anything you're suggesting in Outlook though. Or for Google to add it to gmail.
The gradual approach is a smart one, too.
I think AMP for Email has some great ideas but bad branding. It could be a useful starting point for this discussion.
AMP for Email is dealing with a completely different problem from the one discussed in this article—one that no one asked to be fixed, and which few people even agree is a problem (and they’re all trying to sell you something).
Anyway, you understand why I brought it up, right? It is an attempt at a standard in email that already exists. When we are talking about improving email, highlighting existing work is useful. That means there is something tangible that can be used to improve the weaknesses in the current model.
Additionally, he brought up HTML email in the post. AMP email is an attempt at standardizing HTML email. That is relevant to what he wrote.
It was Google trying to shove through something they’d invented, using their market power as leverage. You don’t make good standards like that.
It was also not at all about standardising HTML email—it didn’t improve anything in that way, except insofar as the AMP part being chosen implies that the client has decent HTML support. AMP for Email is purely about dynamic content in emails.
And the very way that each provider that supports AMP for Email has required whitelisting of each sender shows there’s something extremely rotten about the entire thing.
What's the difference between a VM set up by a real person to handle their email and a VM set up by a spammer to spam you?
If they keep the original From: header they would not work, as the sending (mailing list) server would look like a forger, so the ML software would have to do a rewrite that header.
If you wish to keep original From: headers, then ARC would have to be incorporated into this proposal:
* https://en.wikipedia.org/wiki/Authenticated_Received_Chain
keels over in laughter
Oh, there's been quite a few attempts at at standardizing HTML rules for email. I was even in one of them, which petered out very quickly.
Functionally speaking, the problem is you have three groups of people with HTML email (well, four groups, if you include the people who wish it died off). You have the marketing folks, who want HTML email to work essentially exactly like regular webpages so they can do all their normal design stuff and get it to work. You have the MUA implementers (particularly webmail), who need to aggressively sandbox and sanitize the HTML because to do otherwise is to risk security leaks. And you have Microsoft, who needs to keep visual compatibility with Word HTML because their userbase would flip out if they broke stuff. These groups want different stuff from HTML, and they're not going to do a good job of reconciling their viewpoints with one another.
2. Ok so these are the people who are going to want to help us and they run the largest email servers at the moment
3. I think MS products already have web views in them (either edge or old edge), if we had a change over like MX 2 maybe that would be an acceptable trade off to break the old.
And before somebody says "won't fly", all those fancy new "will replace email someday" messengers use markdown or some parts of it's formatting.
I would also state that Markdown itself is completely unsuitable for the purpose. You’d need to design something new which might look very similar to Markdown, but which would have basically no shared behaviour with even CommonMark as regards parsing, since you don’t want HTML to be a thing. Markdown itself is seriously compromised by its HTML basis.
I can’t actually think of a single comparatively-mainstream messenger that uses even a variant of Markdown; rather, they use their own lightweight markup languages <https://en.wikipedia.org/wiki/Lightweight_markup_language> that are very clearly incompatible with Markdown. (It’s also often a frontend editing feature that gets turned into something like HTML after that.)
Actually, text/markdown exists as well. However, the definition of markdown syntax is, um, less than precise: https://daringfireball.net/projects/markdown/syntax (the text/markdown RFC literally has a parameter to indicate which flavor you meant by markdown!). And it incorporates HTML too, FWIW--legal HTML fragments are legal markdown as well. Honestly, markdown's million variants makes the HTML support landscape look uniform.
In Outlook 97, they used the MSO renderer (Microsoft Word) for editing and presentation. It has an incomplete and buggy implementation of HTML 3.2.
In Outlook 2000–2003, they did the obvious sensible thing: ditch that, and use MSHTML (Internet Explorer).
In Outlook 2007, they switched back to MSO, for reasons that never made a skerrick of sense to me (their explanation was vague nonsense that included the word “security”, but the articles that discussed it vanished from the web long ago so I can’t point you to any). I believe they still use the MSO renderer to this day. Windows Mail still embedded MSO. I think that the new Outlook client they released last yearish? was still using MSO, though I’ve heard claims to the contrary as well.
The MSO component has had, I think, approximately two changes in the last 28 years, one of which was supporting high DPI (… which it does imperfectly) and the other I forget.
Since one of the major email clients is still using a dodgy implementation of 1997 web standards, what incentive have other providers had for supporting newer stuff?
We’re slowly getting places, but when you contrast it with the web’s pace, in both backend and frontend (e.g. HTTPS deployment, and new CSS/HTML/JS features)—well, it’s very obviously a completely different environment.
And if there were security issues, they needed to fix them for IE’s sake already!
I don’t remember the details of what they wrote, and wasn’t able to find it even five years ago, but I do remember that the reasons claimed just made no sense.
Down in the parser and such, no doubt about it. But it would have also lacked much of the attack surface of IE, such as, oh, ActiveX. Granted that specific example would be easy enough to disable, but that's just one mine in the whole field. They definitely should have wrestled IE into shape, but the IE team clearly wasn't taking marching orders from the Office team. Organizational dysfunction manifested in sofware.
- no script tags in HTML5 it's mark up not JS.
- no separate body for rich html and plain text, This will just result in "Please enable HTML rendering to view this message" - maybe we need to define a subset of HTML (no external resources, css etc..)
99% of the world doesn't work in HN land of high pay for high expectations. Most places are trying to force people to work 100hrs a week for as close to min wage as possible.
Unless countries start enacting actual labor laws with teeth I stand by my statement that no one really wants or needs email to work better, there already are alternatives.
Hard disagree. Email sucks in just about every way.
Get a few years on you and the Grind-set will fade as you realize you don't want the only thing in life to be work. You will be glad email still exists.
Messages sometimes take minutes or even go up a to come through for no apparent reason what so ever.
Massive spam issues. Headers are trivially forged
People can sign up other peoples addresses to mailing lists, and if that 3rd party is a company, good luck not getting re-subscribed to random stuff for the rest of eternity.
That's a feature. I don't want senders to know if their message was received without my explicit acknowledgement.
There is, but nobody uses it.
> Massive spam issues.
Verify your recipients, then add them to a white list, then block everybody else, like a typical chat program does.
Every major email client has supported read receipts for a very long time now. It's just rarely enabled by default outside of corporate environments (I would guess in part because users find it invasive).
The funny bit here is that email itself hasn't actually changed at all (at least not in ways that affect personal communication). It's just that folks have become used to the intrusive instant gratification of push-based chat systems.
For personal use cases email is great.
- Corporate support chats (no way to export/capture a papertrail as a customer, usually horribly brittle)
- Proprietary/company-specific messaging solutions in my account on various sites (no notification channel for responses, often no papertrail for me either)
- Contact forms (unidirectional, not standardized, no papertrail for me as a sender)
- Push notifications (single-device only, no reasonable inbox management, headline-only)
- SMS (just no on so many levels, most importantly that I don't want everything to be tied to my phone/phone plan and that I can't own my phone number in the same way that I can own a TLD)
The EU (or Germany, I still haven't found out) mandates companies to have a support email address, and it's just so much more pleasant than the US pattern of providing only phone support, a horrible support chat experience, or a mailing address.
(There’s also the problem of senders aggregating messages from untrusted parties, so that reputation is always a tricky balance. For example, I have four spam messages from gmail.com this month. When I’m not getting bursts like this marketexecmail stuff that started two weeks ago and will probably stop within another couple of weeks, I find that at least a quarter of my spam messages come from gmail.com, outlook.com or hotmail.com; in January, it’s 12⁄46, and at least four more are from other general-public domain names that I recognise.)
If you’re not familiar with these things, the keyword is FUSSP (Final Ultimate Solution to the Spam Problem).
It's a really old meme.
Mixes up MUA and MTA technologies as if they were even the same ecosystem.
Yeah, email needs replacing, but it needs actually serious proposals.
For example, you could have, in an email:
now, maybe a virus scanner will think that the email is plain text, but the recipient's email reader parses it as HTML.Same potential confusion with Content-Transfer-Encoding.
And then there's the matter that headers are supposed to be ASCII-only, but what happens when you send header values with 8 bits set?
And then there is how email threads work. Most of the world uses "In-Reply-To" and "References"-headers, but Exchange, in its infinite wisdom, decides to ignore them and has its own proprietary headers.
Basically every aspect of email has very annoying problems, and every attempt at a version 2 needs to either ignore some of them (annoying those particularly invested into this problem), or reinvent the whole wheel, running in Second System Syndrome worries.
It's not easy...
Don't make a federated protocol. Basically every issue with email can be traced to it being federated.
It is decentralized
"A federated protocol is a protocol (defined next) that makes it possible for servers to communicate with each other, regardless of who is running those servers."
Federated is not a synonym for Mastodon instance.
Wouldn't that definition include almost everything on the internet?
For a start, it wouldn't include virtually every online game, or any protocol that enforces user sign in, like Discord or MS Teams.
Now both are servers, and they're communicating with servers. Server to server communication on http.
(What would "server/server" even mean if you're making that distinction? Two communicating parties that only receive data, never sending it?)
Or an IRC network (the server to server parts of it).
Yahoo Mail
Hotmail
Protonmail
Fastmail
AOL
Etc.
Meanwhile, if you can't use discord, then You're locked out of all of discord forever. And discord is like the 10th Gen of "hot new chat apps", what do we do when it starts to die (because Discord will probably start dieing in the next 10 years)? Just lose everything in all those discord groups, like we lost ICQ and MSN messenger and AIM and Skype?
Yes a good amount are spam, marketing, scams, etc. But that just comes with the fact that the platform is free and open.
Very strange to me to make such a barrier with corp communication. That’s communication too, and most of these corps now have things like Slack aka Discord anyway.
- loose standardization, and lack of proper versioning
- Postel's law, which is a recioe for disaster
Takeaway if you want to design a federated protocol use a “the server chose violence” approach and reject any messages that is not perfectly compliant with the expected input for the given version (which must be part of the protocol itself).
The problem is, neither clients nor servers want to enforce it because users will complain.
Replaced a complete mess with a little less complete mess. Nobody calls it Atom feeds as well, leading many people to still offer RSS despite it being the worse protocol.
Replaced a complete mess with a little less complete mess. Nobody calls it Atom feeds as well, leading many people to still offer RSS despite it being the worse protocol.
Thanks for putting these thoughts together :-)
A few thoughts:
1. I don't know if HTTPS is the best analogy, since it took 20 years for it to really become widely used (vs. non-secure HTTP). I think a better analogy might be the transition from HTTP/1.1 to HTTP/2 (or HTTP/3?).
2. I don't think we can rely on large providers (e.g. GMail) to quickly adopt a new system and even issue switchover ultimatums. A simpler mail system may not be in their best interest. Aiming for slower, organic growth could be more realistic. Even if a next-generation mail system never becomes dominant, it could still provide value. (Maybe Mastodon/ActivityPub is the analogy here?)
3. In any sort of reboot, I think a modern encryption system (something like Signal protocol?) would be a must.
4. Can we have a system where senders need to hold tokens to authorize them to send a message to a recipient? The whole "anyone on earth should be able to send you an unsolicited message" idea is ultimately the source of the spam problem. Messaging systems that rely on bidirectional agreement have much less of a spam problem. (Obviously this raises a slew of other technical and UX questions...)
I know that making too many changes is a risk. But I think that there is also a risk in making too few changes. (If the value proposition isn't sufficiently bold, this may hurt adoption.)
> 1. I don't know if HTTPS is the best analogy...
IMO, the lynchpin to HTTPS accessibility was that we made cutting certificates easy, and most importantly, free. By eschewing the baseline requirement for cutting a certificate to being able to validate that the site on the domain is trusted by virtue of a relationship between the hostname and the certificate requester, and providing an API to automate the renewal process, opting into HTTPS went from "administrative burden that needed a sysadmin to manage" to "service any hosting provider could leverage". LetsEncrypt and the ACME protocol, and Cloudflare before them, did much to radically change the landscape for making HTTPS accessible to everyone. [1] That makes the analogy pretty apt to me, IMO.
2. You make a very valid point here. By virtue of letting SMTP languish so long, bandaiding it as we went, instead of opting for a major refactor of how email works, we made an environment that was rife for consolidation. As a result, it makes sense that we might have to boil the frog to make a change.
3. I'd love to see it, but I'd be interested in learning how key exchange/trust works. I'd be interested to see how you'd envision enabling new senders to send you mail.
4. This feeds back into my comments on 3. With established contacts, this is great, but in a world where, say, you were giving a talk, and wanted to offer folks that listened to your talk a way to contact you afterwards, how would you distribute the token? If someone abuses the permissions, how do you invalidate it? I can't foresee this being an all-or-nothing system, and don't really see a system where creating some sort of one-way or two-way trust between individual senders and recipients is at all feasible.
[1] Let's Encrypt stats: https://letsencrypt.org/stats/
Signal doesn't strike me as a very suitable candidate. It is quite connection oriented and email is not. For example, it requires a online "prekey" server just to do encryption.
We already have two protocols for encrypted email (OpenPGP, S/MIME). I can't see how another incompatible protocol would help. It seems to me that the first problem to address would be the usability of encrypted email, which is quite poor. While we are at it, we should work on the usability of encrypted instant messaging, which isn't that great either, for most of the same reasons.
Email born as internet, something decentralized. But it is getting harder and more complex to have your own mail server, because you have to comply with a lot of requirements of major mail providers. For big scale servers, and commercial mass sending companies ("legal spam") they can afford to comply with all those requirements. And about ("not legal") spam and malware senders, or they don't care if they reach a limited set of target, or the reward is high enough to try to trick the system.
So the smaller mail servers, without so many users, or without so knowledgeable maintainers, if any, are getting expelled from the game by both bad and big players, some just move their email administration to some of the big providers (and privacy and territorial requirements may be a problem here). What the article proposes is another change to push things in the same direction.
And, for good and bad, some of the present use cases may be harmed by this proposal too, like devices and other simple notification services, or mailing lists.
Basically everything's centralized now because bad money drives out good, and I don't have any ideas to fix it. It may be fixing itself to some extent as some CEOs keep banning their anchor users in strange tantrums.
I also think a lot of the ambiguities could be fixed with new RFCs gradually.
I do understand the attraction of highly specialized, modular and pluggable tools - so I know why OpenSMTPd does not have a built-in nameserver, spam filter or DKIM service. The architects are thinking about correctness and scalability and modularity, etc.
But Oh My God it would be so much simpler - and comprehensible - if there was just one package with just one config file.
If the only purpose of a particular nameserver is authentication tasks for a mailserver, we should consider moving the nameserver into the mailserver. The same goes double for dkimproxy.
There is no reason why this couldn't exist in the current e-mail technology. In fact, as someone who has written HTML, I'd rather not have HTML anywhere if possible! It's a terrible, nonsensical technology. A simple XML format for display text/images would be much easier to deal with, and also easy to transpile to a HTML/CSS subset. I'd like to hope that e-mail will last longer than HTML!
I think the biggest problem with e-mail is that the lack of inter-provider communication protocols, something that distributed social media (the fediverse) has. For example, there is no way for a provider to know whether or not an e-mail sent was opened, so EVERYONE uses tracking pixels! Like, just put this into a protocol already! The problems you're talking about with HTML support are simply because there is no way for a provider to announce what HTML it supports, partly because this depends on the client that opens the e-mail, not the server that stores the e-mail. But even then, seriously, this should really just be a setting per account and if you use Thunderbird it announces to everyone "hey this dude uses Thunderbird, so don't use fancy HTML!" and if you switched to something else you would just start getting different e-mails. You can see why this wouldn't work with HTML, since HTML is a terrible language that has no actual support for being displayed in a zillion different clients despite what it claims.
HTML already gracefully degrades. If you don't support a feature, you just ignore it and continue. It works just fine unless you're shooting for exactly matching some figma design on all clients, which is exactly what HTML is not supposed to do.
1) No one changes established tech (software/hardware/services) unless the "new new thing" is 10x better/radically different than the old.
2) "Bullshit talks, code walks."
Here's how something like this becomes an actual standard: Someone codes up a prototype, and gets others to use it. Enthusiasts join in and improve it. Five to 10 years or more passes and if the new standard is actually worthwhile, it will have grown to the point established organizations finally pay attention and - if it's in their best interests - will adopt it.
Take git as a prime example. Launched in 2005, it was still at only 42% adoption by 2014 (according to Stack Overflow yearly survey). A decade after that, however and it's at more than 95%. It took nearly two decades for one of the most quickly adopted technologies in the 21st century to become ubiquitous.
So get on it and we'll see you in 2044.
- make unsubscribes work - make phishing harder