While JMAP seems to scratch every itch of a sucker for proper web API design, I’m wondering if the design space for new protocols should really be constrained to layers on top of HTTP. Is there really any new-ish binary protocol these days?
Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO. Then again, a lot of solid thinking went into these things, so there probably are a lot of good reasons that I’m not aware of.
People think serializers are easy, when suddenly this makes a whole raft of choices your problem. You can write your own, and now you have two problems; or you can build on top of capnproto or grpc or, if you really want to get retro, ASN.1 and inherit the problems of your serializer framework.
JSON by comparison has simple, obvious limitations that more people are familiar with dealing with.
There's also the tendency to tie your protocol to implementation. The Microsoft Exchange "protocol" didn't get reverse engineered for so long because it's basically the COM structure of Outlook fed through (if I remember rightly) DCOM-RPC.
It’s such a breeze to self-host your own email server using Stalwart. It has been a new era for email self-hosters like myself since these kind of fully integrated email servers like Stalwart appeared. Another good one but not as actively maintained is Maddy.
For those needing to deal with customers/clients/internal teams with Google Workspace/Outlook and wanting JMAP-style (though not JMAP) modern JSON APIs, Nylas might be a viable option: https://www.nylas.com/
Nylas pricing has gotten better recently, but is still quite high though - at $1.50/connected account/month at scale, it's likely material to your per-user margin if it's part of your SaaS offering.
But if you have a use case where this is a no-brainer (like capturing/analyzing/building custom real-time UI around your internal sales team's emails) then it's remarkably powerful.
Anyone got a link to a better sales job on JMAP & friends?
It sounds awesome but the way it is intro'd here:
Over the past few years, the IETF has been redefining how email, calendars, and contacts are synchronized and shared. Building upon the success of JMAP for Mail, several new protocol extensions have been introduced:
JMAP for Calendars - A modern replacement for CalDAV and CalDAV Scheduling.
JMAP for Contacts – A powerful alternative to CardDAV.
JMAP for File Storage – A replacement for WebDAV-based file storage.
JMAP Sharing – A modern successor to WebDAV ACL.
JSCalendar - A clean, JSON-based evolution of iCalendar.
JSContact – A modernized, JSON-native successor to vCard.
...gave me pause. A protocol I've never heard even though I hang out here for an hour a day, was so successful, that it launched 6 new projects?
Sounds more like the parts of the web dev that give me ick (new and shiny; rush to copy new and shiny in other contexts; give it a year; and all of a sudden only 1 of the 6 actually was successful)
Running Stalwart in production for ~20 heavily used accounts for some company and no problems so far! The simplicity for such a complex stack and flexibility of deployments is off the charts!
We need better client support for JMAP. Apple Mail, Thunderbird, Outlook (as if), and so on. I'm surprised some of the smaller ones like Canary or Spark don't implement it as a product differentiator.
Outlook ? Lol. In 2 years-time Outlook will only connect to MS365 and that's it. You're betting MS will switch to JMAP ? Lol again.
P.S. ("New" Outlook already only connects to MS365 servers and then stores your credentials and data on Azure, while they proxy to your actual IMAP/SMTP server )
I can understand why JMAP instead of IMAP given the latter's antiquated design. I don't see the advantage to clients in replacing WebDAV though, and the others are a bit iffy too. They'll need to make a way better sales pitch than 'JSON vs XML' (serialization ain't tough, XML is supported everywhere).
I guess contacts/calendar follows JMAP naturally when the clients already implement it, but that only applies in the 'already wrote a JMAP email client' case. Virtually any other case would rather stay with widely supported protocols?
I really hope Fastmail implements the JMAP spec for calendars and contacts soon. They’ve had the mail part of the spec implemented for a while, but it still requires CardDAV/CalDAV for contacts and calendar access.
> They are robust, widely adopted, and battle-tested. Yet, their XML-based design is notoriously verbose, inconsistent, and difficult to implement correctly. Information is scattered across HTTP headers, XML payloads, and even embedded iCalendar data, creating endless compatibility and interoperability challenges between clients and servers.
Can others confirm if these problems are widespread? I get that these protocols are probably a pain to develop for but given they are "robust, widely adopted and battle-tested" it seems that is probably a solved problem. It's better to have one standard that is used everywhere than to have to choose between two standards.
I worked on the iCalendar, CalDAV, and CardDAV parsers at Apple in 2010 or so, and I see no reason to believe that today’s Macs, iPhones, and Watches, are using anything more modern.
I haven’t been there in more than a decade. I really am curious what the response in Apple (and Google) is to this spec.
Does it seem like the calendar protocol will be able to replace the VTODO bit of ical so that Todo applications can be built on top of it? I've played around with ics files a bit for the tasks app in nextcloud and it wasn't a pleasant experience so I kinda dropped the project.
What next? Replacing Sieve with something cumbersome, but JSON based?
There is no good desktop implementation of MUA with old technologies (IMAP, Sieve), will all this JMAP help?
I don't think so.
What is profit to have good server with new good (assume it is good, I'm not sure, but lets assume) protocols without good client?
IMAP4 is underused by modern clients: it allows to effectively store client configuration on server, nobody implements it on client side. It allows to configure per-folder Sieve scripts, nobody implements it on client side. Nobody implements good Sieve client (with folder name autocomplition and such) even for global script, not to mention per-folder ones. Heck, there is no good Sieve editor! (I know about Sieve client built on Electron, it is not good, it is incomplete and buggy).
Servers are solved problem (sendmail, exim, postfix, dovecot, cyrus). Clients are not, they stagnated at the moment GMail was announced.
Reading weirdly many “but there are no clients” posts. Of course something has to come first. You can’t really develop a client without a server implementation and Stalwart is essentially the first server implementation of JMAP.
With Stalwart in place, there’s finally a reason to develop a client for JMAP.
I hope y’all are aware that Mozilla’s new mail service will use it, so that is likely going to give JMAP a big push!
Stalwart is, from what I’ve read, an excellent JMAP server.
JMAP is, from what I’ve read, a great protocol for building an E-Mail (and now also others) client on top of.
Since I would like an innovative way to access my E-Mails, but do not want to self-host, I would find it interesting to use Stalwart as the server component of an E-Mail client: Data is somehow synced into Stalwart via the “ugly” protocols and I get a nice API to build an elegant client on top of.
My basic research shows that something like IMAP-IMAP sync seems to be a thing. Has anyone done something like this, perhaps even with Stalwart? (this of course grows in complexity for each new protocol to be proxied).
I believe having this kind of setup easily accessible could jumpstart a new generation of E-Mail clients on top of JMAP because it (relatively elegantly?) circumvents the chicken-egg problem by allowing all existing IMAP mailboxes to be accessed via JMAP.
As a JMAP client implementer, there’s a big plus, beside JMAP, in having an integrated server for email, contacts, and files.
Some setups take days even for email, so an easy, fast setup saves a lot of time.
Also, after code the CalDAV ↔ JSCalendar part, using only IANA time zones instead of scattered ones in CalDAV components makes things much simpler.
Interesting I didn't know that JMAP for File Storage is a replacement for WebDAV. My experience with WebDAV is that it has traditionally been sluggish and clunky; I'm not sure if that's due to the protocol or the backend. I'm hoping this could be a huge improvement.
I tried to set up stalwart, but I didn't understand exactly what it wanted me to do. It's a webserver (for webmail and admin) and mail server in one, but I already run a webserver and I already have cert infra. So I couldn't figure out what dns settings to use and what ports to reverse proxy. And how to get it to play nice and share the certificate. Seems like stalwart has been designed as if it is the only thing running on a machine, with sparse documentation for any other setup. I tried getting help on the discord server but the tone seemed to be of a sort of "it's quite obvious, you should already know this". At that point, it's so much friction to reverse engineer it that it might just be easier to set up dovecot and postfix.
The installation instructions seem to agree with you:
1. curl this shell script
2. Run it as root.
Anytime you see that, you can assume the software wants to take over the whole box and isn't likely to make any attempt to play nicely with other services/software/users already running on the same host.
I have had similarly been unable to get as much help from their Discord as I had hoped.
That said, I'd give this another shot. What I discovered is that Stalwart is incredibly flexible and designed to be used in any configuration you want, and the documentation and examples are incredible, compared to most other projects. BTW, there is no webmail yet, but you are right that there is the admin that is served over HTTP/S, as well as MTA_STS, JMAP. The default listeners get you started and it's kind of out-of-the-scope of Stalwart in terms how you want your server to interact with those listeners. But whether you containerize it or setup a reverse proxy, the documentation has a lot of examples, including how to pick up the certificates, if not managed by Stalwart itself.
I landed on a Caddy for HTTPS and haproxy for proxy protocol on just JMAP, and the remaining TCP services directly binded to the host for the main mail services.
Traefik is another example that I had initially prepped and got working that also handled TCP connections with proxy protocol where needed. The Stalwart documentation was very helpful with that.
In short, I was impressed with how easy it was to integrate Stalwart into whatever setup I wanted, and how open the developer is to different setups. For instance, mox (while awesome in its own right) considers containerized setups to be not-recommended, and generally expects it to live on its own server. Stalwart is flexible, but the side effect is that it is overwhelming at first... but becomes quite elegant once you get the hang of it.
28 comments
[ 2.8 ms ] story [ 54.2 ms ] threadStill, it’s an interesting space, I think.
JSON by comparison has simple, obvious limitations that more people are familiar with dealing with.
There's also the tendency to tie your protocol to implementation. The Microsoft Exchange "protocol" didn't get reverse engineered for so long because it's basically the COM structure of Outlook fed through (if I remember rightly) DCOM-RPC.
Nylas pricing has gotten better recently, but is still quite high though - at $1.50/connected account/month at scale, it's likely material to your per-user margin if it's part of your SaaS offering.
But if you have a use case where this is a no-brainer (like capturing/analyzing/building custom real-time UI around your internal sales team's emails) then it's remarkably powerful.
It sounds awesome but the way it is intro'd here:
...gave me pause. A protocol I've never heard even though I hang out here for an hour a day, was so successful, that it launched 6 new projects?Sounds more like the parts of the web dev that give me ick (new and shiny; rush to copy new and shiny in other contexts; give it a year; and all of a sudden only 1 of the 6 actually was successful)
P.S. ("New" Outlook already only connects to MS365 servers and then stores your credentials and data on Azure, while they proxy to your actual IMAP/SMTP server )
edit: we use it on very resource constrained environments, the container version is too much overhead.
I guess contacts/calendar follows JMAP naturally when the clients already implement it, but that only applies in the 'already wrote a JMAP email client' case. Virtually any other case would rather stay with widely supported protocols?
I think we're about ten years past the point where "newer = better" was a reasonable starting presumption.
Can others confirm if these problems are widespread? I get that these protocols are probably a pain to develop for but given they are "robust, widely adopted and battle-tested" it seems that is probably a solved problem. It's better to have one standard that is used everywhere than to have to choose between two standards.
I haven’t been there in more than a decade. I really am curious what the response in Apple (and Google) is to this spec.
There is no good desktop implementation of MUA with old technologies (IMAP, Sieve), will all this JMAP help?
I don't think so.
What is profit to have good server with new good (assume it is good, I'm not sure, but lets assume) protocols without good client?
IMAP4 is underused by modern clients: it allows to effectively store client configuration on server, nobody implements it on client side. It allows to configure per-folder Sieve scripts, nobody implements it on client side. Nobody implements good Sieve client (with folder name autocomplition and such) even for global script, not to mention per-folder ones. Heck, there is no good Sieve editor! (I know about Sieve client built on Electron, it is not good, it is incomplete and buggy).
Servers are solved problem (sendmail, exim, postfix, dovecot, cyrus). Clients are not, they stagnated at the moment GMail was announced.
With Stalwart in place, there’s finally a reason to develop a client for JMAP.
I hope y’all are aware that Mozilla’s new mail service will use it, so that is likely going to give JMAP a big push!
JMAP is, from what I’ve read, a great protocol for building an E-Mail (and now also others) client on top of.
Since I would like an innovative way to access my E-Mails, but do not want to self-host, I would find it interesting to use Stalwart as the server component of an E-Mail client: Data is somehow synced into Stalwart via the “ugly” protocols and I get a nice API to build an elegant client on top of.
My basic research shows that something like IMAP-IMAP sync seems to be a thing. Has anyone done something like this, perhaps even with Stalwart? (this of course grows in complexity for each new protocol to be proxied).
I believe having this kind of setup easily accessible could jumpstart a new generation of E-Mail clients on top of JMAP because it (relatively elegantly?) circumvents the chicken-egg problem by allowing all existing IMAP mailboxes to be accessed via JMAP.
Also, after code the CalDAV ↔ JSCalendar part, using only IANA time zones instead of scattered ones in CalDAV components makes things much simpler.
1. curl this shell script
2. Run it as root.
Anytime you see that, you can assume the software wants to take over the whole box and isn't likely to make any attempt to play nicely with other services/software/users already running on the same host.
To their credit, they do have a docker image (https://stalw.art/docs/install/platform/docker).
That said, I'd give this another shot. What I discovered is that Stalwart is incredibly flexible and designed to be used in any configuration you want, and the documentation and examples are incredible, compared to most other projects. BTW, there is no webmail yet, but you are right that there is the admin that is served over HTTP/S, as well as MTA_STS, JMAP. The default listeners get you started and it's kind of out-of-the-scope of Stalwart in terms how you want your server to interact with those listeners. But whether you containerize it or setup a reverse proxy, the documentation has a lot of examples, including how to pick up the certificates, if not managed by Stalwart itself.
I landed on a Caddy for HTTPS and haproxy for proxy protocol on just JMAP, and the remaining TCP services directly binded to the host for the main mail services.
Traefik is another example that I had initially prepped and got working that also handled TCP connections with proxy protocol where needed. The Stalwart documentation was very helpful with that.
In short, I was impressed with how easy it was to integrate Stalwart into whatever setup I wanted, and how open the developer is to different setups. For instance, mox (while awesome in its own right) considers containerized setups to be not-recommended, and generally expects it to live on its own server. Stalwart is flexible, but the side effect is that it is overwhelming at first... but becomes quite elegant once you get the hang of it.