Ask HN: Why XMPP failed and SMTP didn't?

Then we tried Slack but most people didn't come there too much. Today we have a Telegram group and it's so much better UX/UI wise...

If there were an alternative to SMTP I believe it would be NNTP given it is similar in concept and just introduces threaded messaging which SMTP clients try to mimic now. There are several potential rabbit hole discussions of why moving away from SMTP are highly unlikely to ever occur.

As far as any individual user is concerned, loss of the server(Which their identity is tied to) would be a hassle. Just like email.

SMTP got really entrenched really early. It's what gmail uses. They can't really do a proprietary mail protocol, SMTP was around way before them and nobody wanted to switch.

Instant messaging for the masses was proprietary from the start, as far as I can tell.

There was IRC, but it seems like by the time the internet was everywhere, people were already on AOL and ICQ and a bazillion other proprietary ones offering various extras and integrations.

None of them seem that interested in open protocols.

One is that XMPP is an overcomplicated nightmare. The protocol was verbose, flabby, and hard to implement, and the server software was very hard to set up and run especially for novices. A person could not just install a server and start chatting, even if the server was just stand-alone let alone linked to anyone else.

The second and IMHO more fundamental reason is that the Internet is a dark forest. Any open system that becomes sufficiently popular will be destroyed by abuse.

If you offer a chance to make any amount of money whatsoever online millions of hustlers will rush into the void like gas molecules invading a cracked vacuum bell and will scramble all over each other to suck every last fraction of a penny out until nothing of value remains. So far only centralized managed systems have been (somewhat) successful at keeping the barbarians at bay. I'm not saying a decentralized system could never succeed here, but I don't think it's been done yet. (Cryptocurrency isn't an example as it's already been throughly destroyed for its original vision and use case by scammers. It's a great case study in exactly what I mean.)

(Edit: the goal is not always money either. Read money as "value." Political propaganda, cult recruitment, weaponized disinformation, or just trolling for lulz all count as extracting value of some kind at the expense of the commons.)

SMTP along with Usenet was one of the first casualties of this phenomenon, and I would argue that it did fail as an open system. You can run your own SMTP server but it's not for novices or people without time on their hands. You'll have to fight constantly to keep your IP out of blacklists and to keep spam away from your users. Spammers are in an arms race against huge companies with massive training data sets and entire teams dedicated to spam filtering, so this only gets harder over time. An independent SMTP server is easy prey.

99% or more of users use one of several large mail hosts. These are mostly Google, Microsoft, and Apple in the USA. E-mail hosting is cheap to free and the vast majority of people would rather someone else deal with the pain of defending them from spam.

* Onboarding was difficult. There was no obvious choice of server or client to use.

* Adding friends was difficult. You needed to send a subscription request to a contact, and they needed to send one to you. If anything happened during this process, you couldn't chat.

* Popular XMPP clients, like Pidgin, also supported the other chat services (AIM, ICQ, MSN, Yahoo, etc) so people just continued using those.

* Network effect. You need to convince a mass of people its better, otherwise nobody's using it because no-one uses it.

* No obvious benefit to the user. It's decentralized sure, but there weren't many improvements over AIM that people actually used.

* A lack of good iPhone XMPP clients.

In 2005 Google added XMPP support to Google Talk/GMail Chat and they were federated, but nobody federated back and they closed off its successor (Hangouts).

At work (operations for a space mission), we use an XMPP-based chat system for tactical communications along with several other systems, but Slack was introduced and more and more communcations are being handled over Slack. I'm advocating for replacing several systems with Jitsi, which is XMPP-based, because it allows us better control and customizability. It's an uphill battle.

XMPP:

  - is designed for chatting and P2P and VoIP over persistent HTTP connections
  - requires everyone to be online
  - servers have to implement 50 different specs
  - works at the frontend and backend

  - is designed to send one large message in bulk over a regular short-lived TCP session
  - is a store-and-forward messaging system, highly resilient to slow or inconsistent networks and server issues
  - is simple and the ecosystem is layered, so the core server doesn't have to implement a dozen specs
  - is practically backend-only

https://signal.org/blog/the-ecosystem-is-moving/

I'm not entirely certain why this was, but I suspect that valuing extensibility over interoperability (or the perception of it) was part of it. Optional extensions and interop with hosts that used a different set simply gave hosts/operators too much to think about in terms of deploying the service for users.

There are probably multiple reasons and we can't replay history to know which reason contributes the most but one key difference is that email accounts (SMTP) were pushed onto the regular non-techie consumers whereas XMPP was more of a geek tool that users had to pull and opt into.

E.g. New students at a university automatically got a ".edu" email address. Residents got a free email account (ISP) with their cable service. Employees got a corporate email address. In other words, millions effortlessly got the utility of email/SMTP even without installing AOL CDROMS.

To further reinforce email/STMP, if consumers want to order something from Amazon, they needed an email address to create an account instead of an XMPP address.

Other forms of communication like XMPP/IRC don't have that widely disseminated self-reinforcing utility cycle so they stay a niche tool. What critical service in normal life requires an XMPP address? I can't think of any.

Aren't open protocols and interoperability great?

SMTP is an old protocol; it really predates much of what we'd think of as the Internet (e.g., it predates DNS or even IPv4). This means that during the big initial growth phase of the Internet, SMTP was already an established standard that could be used to transfer email. Indeed, for email routing on the internet, you kind of had to support SMTP anyways, which makes it difficult for any other protocol to gain traction.

That's not to say that there weren't alternative protocols. The biggest of these was X.400, which many in the 90s saw as the eventual replacement of SMTP. But this was hampered by the already existing install base of SMTP. Other failures leading to the failure of X.400 was its reliance on the OSI stack, which lagged behind in implementation compared to the TCP/IP stack, and the rather cumbersome addressing model of X.400 compared to SMTP. The value added in supporting X.400 in addition to SMTP wasn't worth the cost of implementing X.400, and it wasn't really feasible to support only X.400 given the already widespread use of SMTP.

In this model, XMPP actually works closer to X.400 than it does to SMTP. It came about after the protocols it aimed to replace were already in existence--and wide use. (See also IRC, which is still alive and active). So implementing XMPP means you need to justify the value-add of XMPP over existing legacy protocols. If you were implementing your own, new chat service, it might make sense to build it on top of XMPP instead of a custom protocol. But replacing existing chat protocols with XMPP was again a costly move with benefits rarely justifying the move. The further federation goal of XMPP would have been an anti-goal to many of the chat implementations.

Same issue with Mastodon, in my opinion. People want to grab something from the Store or Play, sign in, and be where everyone else is.

As developers we look too often for technical reasons for a limitation or failure but usually the social reasons are more significant.

Getting a client installed on a computer was socially difficult like convincing some one to install an app on their phone. Having a browser installed on every computer and phone made it trivial to get a client on a computer the only down side is being forced to use http and websockets as the protocol.

DNS and WHOIS used to be protocols in their own right but both are being replaced by http, I wouldn't be surprised if http came for smtp next.

Eventually, for various reasons, Google decided that an open xmpp based system didn't work for them, and they closed it and extended it. This was effectively the end of xmpp as a distributed tech that many people use today.

Interesting, in a lot of ways SMTP is no longer an open system. It is extremely hard to operate a mail server in such a way that your valid emails get delivered correctly and not marked as spam. So for example in the name of fighting spam, it is extremely difficult to participate in this "open" system.

I tried XMPP early on, and found the software to have a user experience that was utter garbage, so after that I didn't give it a second thought.

At the enterprise level, though, garbage UX is the norm. SMTP is hard to deploy, but you have paid full-time sysadmins whose job it is to figure out the necessary parts of sendmail.cf or whatever else. You tell them "make it so" and they do.

I suspect there was no enterprise level push for XMPP anywhere.

SMTP was early enough and good enough to get widespread adoption. IRC in its original form was never good enough (esp. re: scaling), so services like AIM were able to step in and were good enough and open enough[1] solution while the open source world stuck with IRC for too long because it 'works for us'. So open standards like IRC never got entrenched the way SMTP did in proprietary solutions and by the late 90's the open source world had become pretty weak on developing new Internet standards that would even get widespread adoption in the open source world.

It's worth noting that there's nothing etched in stone to say that SMTP has won indefinitely. Try setting up a purely open source SMTP server on the Internet these days and see who you can talk to. Google has been making it harder and harder[2] to use non-Google clients with their servers. Slowly but surely, business interests seem to be enveloping SMTP with their own walled gardens... it's just taking longer.

[1] Well, sort of. They actually seemed to want to be proprietary about it. But every time they changed the protocol, the apps that worked with it were able to reverse engineer their changes quickly and AOL didn't fight too hard to prevent them from doing so.

[2] Or at least enough of a chore so that most people won't bother trying.

I thought it's what everyone used for messaging and push notifications. I know Google extended it beyond compatibility around when Hangouts came out and they started doing browser notifications. Apple stopped providing sources for ejabberd in the last few years, so I assume they moved off that for their iChat/messaging backend, but isn't iChat still an XMPP client?

Aren't most SMS messages routed via XMPP?

All of these federated protocols: XMPP and even Mastodon to an extent, are smaller groups of humble nerds running instances. They aren't massively advertised, large-scale billion dollars of investment companies like Discord or Facebook, trying to build network effects and/or advertising revenue.

-----------

When competing against profitable companies, its not sufficient to merely exist. You're competing against advertising and eyeballs. Facebook is big because its big, because it advertises, because it is constantly pushing for more-and-more users. Same with Twitter, same with Discord.

This leads to little advantages: a $10-million+ UI overhaul every few years. Designers to simplify the interface and onboard faster. Paying Google/Apple absurd amounts of money to access the push-notification APIs on phones. Etc. etc.

Smaller XMPP instances already fail at the push-notification thing. Who will pay for that? And without push-notifications, do you really have a modern chat platform? Other companies can afford the costs.