JSON is an incredibly inefficient format for shareable data: it is annoying to write, unsafe to parse and it even comes with a lot of overhead (colons, quotes, brackets and the like).
CBOR is a better wire format in all aspects except readability when all you have is a text viewer. It's a superset replacement when it comes to supported datatypes
It looks it performs way better than msgpack in following benchmarks [0]. However, it's true that it's not very popular - personally, I rarely see any software that utilize CBOR.
What about all those java, ruby, python and C# services that expose JSON based REST services. JSON is widely used for network API, config files and data stores.
But you have yet to say why JSON is worse than your preferred alternative. I agree that it is not perfect (for many use cases I prefer TOML personally), but I cannot see how it is worse that S-expressions (except for the trailing comma issue, that is a sin in itself)
It is not the wrong thing. You were arguing that it was wrong because it was rarely used; you have yet to prove your point that it is in any way wrong.
And, perhaps, all those people know better than you.
> The json1 extension does not (currently) support a binary encoding of JSON. Experiments have been unable to find a binary encoding that is significantly smaller or faster than a plain text encoding. (The present implementation parses JSON text at over 1 GB/s.)
Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. There are too many "yes buts" in the living standard.
> that overhead is not much worse
It _is_ much worse than the two-byte "overhead" of one s-exp node.
> It _is_ much worse than the two-byte "overhead" of one s-exp node.
I don't believe you. Atoms are roughly the same, and compounds, well:
[1,2,3] vs
(1 2 3)
{"a":1,"b":2} vs
(("a"1)("b"2))
I suppose if you have a lot of booleans you might save a little by using T/NIL or #t/#f, but otherwise it's more or less a wash.
(You might suggest plists, ("a"1"b"2), for dictionaries, in which case you save a whopping two bytes per entry. But I'd choose the nested-list format over plists, if I were encoding a dictionary into an S-expression.)
In specific contexts such as this one you don't need arbitrarily complex JSON, you just need to validate that the JSON structure respects not just JSON itself but also the protocol specification at hand while you parse, and bail out when it doesn't, so you can have a dedicated, fast parser tailored for the use case, instead of first parsing general JSON then validate its content.
Anyway, for being a Fasmail user, JMAP rips IMAP to shreds in terms of performance. I've been following JMAP from afar and waiting for Fastmail to publish this for a long time.
Not for each use case of everything, anywhere, of course. This specific use case is important enough to warrant a dedicated parser though, and has benefits both in performance and in security.
People have ben actively doing this for XML with streaming parsers that emit events, where you don't just validate XML itself but the semantics of the payload you're receiving, and bail out early instead of parsing the whole XML into a DOM then validating the DOM structure and values.
>There are too many "yes buts" in the living standard.
Could you give an example? The only deviation from the standard I've run into is JSONL being called JSON, which isn't actually meaningfully harder to parse.
Is nesting in JSON more complex than nesting in sexpr?
I feel like it should be possible to translate from valid JSON to valid sexpr with a regexp parser (just match parenthesis and stuff)
> There are too many "yes buts" in the living standard.
I am not an expert and do not want to sound dismissive, but I feel like most of those "yes buts" are a solution to otherwise incompatible implementations. I fail to see where S-expressions have an advantage here. Is there a widely recognized standard that most implementations respect?
> Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision.
Do you mean parsing the JSON bytes itself into maps and lists and atoms, or turning that into an application-useful object?
If the former, are you claiming that Python's standard json library, libjansson, JSON::XS, etc. have bugs? (I don't know why you would roll your own JSON parser in production code.)
If the latter, how are S-expressions any different?
(And there are non-buggy libraries that do both, e.g., serde.)
Related to this there is something I wonder, how does compressed size of a binary file compare to the compressed Hex/Base64 encoding of the original file?
I’m a developer at Fastmail, and I’m going to bite.
Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust.
Professionally, I happen to write mostly JavaScript, and no Rust at all, but we do still focus on performance at Fastmail. (It’s in the name!)
If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible.
But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON.
When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
All in all, JSON is a robust choice for just about any API.
Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. Neil Jenkins gave a talk about this and related matters last month, https://www.webdirections.org/code/speakers/neil-jenkins.php, no public video sadly. I’ve thought about writing a blog post about it all. An unfortunate side-effect of this compression is that you can’t inspect traffic in the network panel of the dev tools; you can add ?compress=0 to disable it, if you’re an interested Fastmail user.
MessagePack and CBOR are equivalent. I haven’t ever had much to do with them (never used them, only known about them), but I’m inclined to think that simple compression achieves between “roughly as much” and “more” in general, and leaves you with the industry standard of JSON underneath. Do not consider me an expert by any means, and I’m fairly ambivalent on your question; but I don’t see a particularly useful niche for them between plain JSON and a binary representation matching a predefined static schema, all potentially with compression.
MessagePack has an ugly history with text data that's still reflected in some of the libraries interfaces. E.g., the Python library doesn't round-trip strings by default. (They'll turn into bytes objects.) There's a flag to do the right thing, but not everyone is aware of it.
Add to that that CBOR has an RFC, and I'll pick it over MessagePack every time.
Binary formats are mostly nicer from a parsing perspective: it's much easier to unpack. You're right, I think, that simple compression will mostly close any size gap.
> When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
I don't think that's a deal-breaker. protobuf has an extension mechanism. [1] You could have a registry for an org to claim an extension number to a given message (and then use it to hold their own message with as much stuff in it as they like). I'm not as familiar with cap'n proto but assume it has something similar.
That said, I think JSON was a fine choice for this purpose. The efficiency is good enough with standard implementations; and there are much faster-than-standard implementations available if you have a special reason to want one. [2]
> When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
What do you mean? Both Protobuf and Cap'n Proto are designed to be extensible. You can add new fields without breaking compatibility.
(Disclosure: I wrote Protobuf v2 and Cap'n Proto.)
Further, like, protobuf now supports a Map<str, str>, although IDK about cap'n or flatbuffers.
That lets you store the the common things in highly compressed protobuf, and extensions can either be done via extension protos, or with stringly-typed map stuff for testing/weird stuff.
[I am replying to scottlamb and joshuamorton as well as to you in this.]
Forgive me; I am not speaking from a strong position of grokking Cap’n Proto or Protobuf (I’m casually familiar with them but have really only once used them, maybe three years ago), and I believe I erred in what I said.
For Cap’n Proto, I don’t believe I’m far off the mark: to pick a simple example, extensions may add fields anywhere, and I believe you’re not allowed to skip field numbers, so you can’t just establish a registry of used field numbers and pick and choose the extensions.
proto3 looks like it probably can’t handle this either.
On proto2, though, it looks like I was approximately wrong, and you could have finite extensibility, and maintain a registry of assigned IDs so that extensions can be compatible. I’m not sure how suitable that would be in practice—the examples I see only allot a hundred numbers for extensions, which seems rather tiny; I’d picture allowing at least a million.
Anything like Map<str, str or any or whatever> I’m going to disqualify from consideration because either you use structs in some places and thereby limit the places that extension can apply, or you use maps right through and then get no benefit over JSON.
So I think they’d still be mostly technically unsuitable, but maybe Protobuf v2 would work. And maybe I’m completely wrong still—you’re the expert! But really, unless you get a major advantage out of such a thing, an important consideration is social suitability. Protobufs and the likes are not widely-used, require more effort to work with, and require more involved library support than JSON, which is much more universally understood and handled. Just look at how I’ve been fumbling around with them!
I see, the point is that you want to extend someone else's protocol, without their cooperation, and without conflicting with any future protocol changes they make.
That is indeed tricky. With protobuf you could claim some high-numbered fields and hope that no one else does the same thing -- with some randomization you can probably be pretty confident. With Cap'n Proto you can't really do it at all since fields are identified by offset, and making structs arbitrarily large is probably not what you want. With JSON, because fields are identified by name, it's possible to choose a name that almost certainly will not collide with anyone else.
I'd argue that the right answer here would be to ask the spec owners to add extension points, which hopefully would not be controversial. Conceptually this could take the form of something like map<string, any> (there are cleaner idiomatic ways to do it in each of Protobuf and Cap'n Proto but the details are beside the point of this conversation).
It's imperfect, but it does get the job done. Among its advantages is that it's both pure text and binary: there's a canonical representation which is binary, and an advanced representation which is textual, and they map 1:1 to one another.
> I think it's pretty obvious which one is cleaner …
Yes, clearly it's the JSON. How do you tell at a glance in the s-expr formatted data what's an object and what's an array? What's a keyword and what's a literal? I can barely even tell the heirarchy where it's very visually obvious with the JSON.
I have no strong love for s-expressions, but objects and arrays are programming language constructs, not data format constructs. There's also no way in JSON to distinguish between an int and a float and a decimal type, but that's fine because it's a data format, and not a programming language literal.
The JSON and s-expr have a different indent level, which is why the hierarchy is less obvious to you.
> objects and arrays are programming language constructs, not data format constructs.
Debatable. To me there is a clear difference between a list of things of the same type, and a mapping of names to values of potentially varying types. I don't think the specific programming language really matters, they are two totally different concepts with much different applications.
> There's also no way in JSON to distinguish between an int and a float and a decimal type
True, but they are just small literals which are easy to differentiate at a glance, not big multi-line structures like objects or lists. Plus s-expressions have the same issue.
There are no objects — it's all lists (or arrays, if you prefer). This is a good thing, because ordering is always the same.
> What's a keyword and what's a literal?
Everything's a byte string: any interpretation above that is a matter for the protocol. This is a good thing, because it means you neatly sidestep issues like JSON's numbers actually being floats rather than integers, as well as issues like numeric precision.
'But I want types!' Of course, they are good. But the type of a number isn't just integer or float: it's integer-which-is-a-known-version or float-which-is-between-zero-and-one-exclusive. You need to check those types at ingestion anyway; it's not a big leap from integer-which-is-a-known-version to string-which-parses-as-an-integer-which-is-a-known-version.
> I can barely even tell the heirarchy where it's very visually obvious with the JSON.
They are both indicated with indentation. Probably I should have used two spaces for the JSON indentation, but I was lazy.
Having seen all three examples, I am now convinced that JSON was the correct choice. Thank you for posting all three examples, it's much appreciated. I hadn't seen s-expressions before and I was considering them a plausible alternative, but now that I've seen it, I don't believe that prefix-operator functions in Lisp-compatible syntax are the appropriate way to construct and transmit data from APIs.
> I don't believe that prefix-operator functions in Lisp-compatible syntax are the appropriate way to construct and transmit data from APIs.
Why not? I'm honestly curious, because I cannot comprehend preferring that mess of JSON to the s-expressions. The JSON just looks like a dog's breakfast, while the s-expressions look clean & easy to write, read, parse and use.
Also, I don't think they are functions: they're just lists with a convention. Those lists can be used to expression function calls, as in Lisp, but in a data format I think that the lists more often express structure.
One thing I love about s-expressions is that one indicates the type as the first item in the list, e.g.:
(foo …)
whereas with JSON one has to invent pseudo-properties:
JSON was created as the lowest common denominator between multiple ancient javascript engines. At one time, at least one browser couldn't handle keywords being used as unquoted object keys, so JSON just has all keys quoted for simplicity. It's the same deal with trailing commas.
The S-expressions look much cleaner to me: parentheses and characters, with a few quotes where needed. Meanwhile, the JSON is a mess of curly brackets, square brackets, commas, quotes &c. All the field names are strings, not symbols or tokens. To my eyes, JSON is an embarrassingly ugly format, while S-expressions are clean & readable.
Funny, to me the commas, culry brackets, and square brackets actually make it more readable. They provide context like "this is a list of stuff" or "this is a key:value mapping" which make it much faster to mentally parse it. The S-Expressions on the either hand are a sea of parentheses with no way to quickly tell what's a list or mapping or whatever without closely looking at each item.
> The S-Expressions on the either hand are a sea of parentheses with no way to quickly tell what's a list or mapping or whatever without closely looking at each item.
I guess that's part of the point from my point of view: it's not about hash tables or objects or even lists, but about what they represent. In my view focusing on the implementation (a hash table mapping strings to hash tables mapping strings to numbers or strings) is looking at the trees and missing the forest.
I agree. I'm in the process of writing an efficient JSON parser, since using the simple approach is too slow, I find, and because a database I want to introspect on uses it. To process JSON, you must inspect every character. Douglas Crockford brags about how smart he is for disallowing comments, but there's whitespace allowed everywhere.
I actually have a (I think) rather novel approach and there's some potential for little optimizations here and there, but it disgusts me that I'm doing this at all. The optimizations popular, shitty formats like JSON receives are purely wasteful.
It's suitable, though. The WWW wasn't made right from the start and there's always an idiot somewhere looking to add more shit to the pile. Not only does JavaScript exist, but it morphed into this ugly, inefficient, very poor ''format''. We don't have Lisp machines, but new processors are getting instructions to cope with JavaScript idiocy.
A numerical (binary) format is clearly best. It's small, random-access when properly designed, and efficient, but hey, everything's being pushed into port 80 so why not push all data into JSON?
Now, the rest of this message is in response to @chrismorgan from an organization pushing this filth.
>Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust.
We agree on that, then, but I prefer to write that Rust is just the effeminate man's Ada.
>If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible.
This is wrong. Firstly, we should clarify what is meant by S-expressions. Let's assume the S-expressions have lists, strings, symbols, and numbers. That's already simpler than JSON. You're not hampered by JavaScript's floating point nonsense, either. There's fewer edge cases and less escaping. JSON is, at best, a worse-is-better form of S-expressions.
>But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON.
So, this ties into what I wrote before I even read your post, exactly. Cretins continue to push everything into the WWW protocols and act like their garbage is standard, simple, acceptable, or wanted; it's only technically standard, but it's a bad standard. I'd bet my foot you also like ''DNS over HTTPS''. A numerical (binary) format should be used, because it's the best option and people can be expected to bother writing an acceptor for such important data. There are plenty of languages that can easily accept a numerical format, but not JSON; think of, say, Forth. Oh, but everyone writes in Rust and JavaScript nowadays, right.
>When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
All of this is unnecessary. Email doesn't need this drivel.
>All in all, JSON is a robust choice for just about any API.
Of course; forty years ago, UNIX was the solution for every problem. The WWW and JavaScript are nothing more than a continuation of this worse-is-better filth.
>Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLAT...
Aside from the ad hominem attacks on JSON, WWW etc. The fact is that a binary format needs exactly the same level of specification as a text based one. ASN.1 is a binary TLV encoding mechanism.
The fact that Fastmail, a well known business that is entirely based on successful deployment of email to paying customers has defined a protocol for object exchange etc that is based on Internet RFCs and standards is a Good Thing.
"Cretins" don't decide to use JSON, JSON is an object encoding that is supported by all languages and environments. It compresses to the same level as binary based encodings. It allows for arbitrary extensions that can be migrated in production without having to update all clients/servers simultaneously.
Binary formats tend to be brittle and require simultaneous end-to-end upgrades to handle change. Protobufs and other TLV based encodings help with this, but they still require recompilation and synchronization of IDL specifications.
Compression and caching are easy to insert into a network pipeline (and/or a process-to-process pipeline) easily and transparently. If you read Fielding's thesis on REST, one of the advantages of the HTTP "stack" (not HTML) is that caching and other "dependency injection" can occur transparently between the endpoints.
Efficiency is not by putting all your eggs in one basket. Efficiency comes from a) designing the protocol well, b) keeping the encoding decisions well away from the protocol's content so that decisions like caching and compression etc can all be added or removed without changing the underlying implementation.
All languages "accept" a "numerical format" (not sure what you mean here, but I assume you mean binary encoding. Unless you mean arbitrary length binary numbers. Do you think binary should continue to be on octet boundaries? Or are you talking about encoding an arbitrary bit length? How will you deal with word endianess?
JSON has some warts, the quoting of keys is a pain, the fact that comments cannot be included (but ignored) in the same way as whitespace makes it harder to document configurations or other static documents. The use of floating point is another wart, but if you are sending any numeric values that rely on floating point accuracy, your protocol is wrong. Define a fixed point integer definition instead (eg send monetary values in cents, not dollars).
I don't know if this addresses everything, but from the spec:
1.5. JSON as the Data Encoding Format
JSON is a text-based data interchange format as specified in
[RFC8259]. The Internet JSON (I-JSON) format defined in [RFC7493] is
a strict subset of this, adding restrictions to avoid potentially
confusing scenarios (for example, it mandates that an object MUST NOT
have two members with the same name).
All data sent from the client to the server or from the server to the
client (except binary file upload/download) MUST be valid I-JSON
according to the RFC and is therefore case sensitive and encoded in
UTF-8 [RFC3629].
A quick scan of it says to me it doesn't get all the issues but it does get the biggest one, which IMHO are the ambiguities in what constitutes a "number" in JSON. I-JSON explicitly limits it to 64-bit floats.
I'm really not sure what I feel about the naming. It's one letter off (both on the keyboard and alphabetically) from IMAP. Is this an attempt to coattail the widespread recognition of IMAP or just an extremely unfortunate name collision?
The original authors of JMAP were IMAP experts who were increasingly frustrated with IMAP. So not it’s not a coincidence. It just started as IMAP over JSON and then ask they fixed issues and added features it became its own thing.
It has been jested in the Fastmail office that one of the requirements for a spec to be adopted by the IETF is that it have a clever name. Have a look around the mail and calendar specs, for example; there are many amusing names.
JMAP is a generic protocol, but its primary motivating example and the thing that has influenced the design most is email, as a replacement for SMTP (between the user and server, not between servers) and IMAP.
The name JMAP was chosen quite early on. I + 1 = J. The words the acronym stands for have changed a couple of times along the way!
IMAP was originally conceived essentially to allow a very thin client that proxies every user command back to the user store, and it also embeds some reliance on old implementation artifacts (UIDVALIDITY, anyone?). There are a lot of "fun" corner cases--I once found a message that was interpreted 5 different ways by 4 different implementations. (It doesn't help that MIME structure is actually pretty loopy and maps very poorly to how all modern clients view mail messages, resulting in many interesting interpretations).
Modern email clients are all structured very differently from the "thin client" perspective--everyone keeps their own copy of the email (metadata) database locally, and use IMAP essentially as a database synchronization protocol, but IMAP is pretty bad at synchronization, at least until the LEMONADE additions came in ~a decade ago. IMAP is especially bad at handling multiple folders correctly, since it assumes that every folder is independent of the other ones, and you can only sit and wait for messages on one folder per connection at a time.
The introduction, for those not wanting to click too deeply into the link:
The JSON Meta Application Protocol (JMAP) is used for synchronising
data, such as mail, calendars, or contacts, between a client and a
server. It is optimised for mobile and web environments and aims to
provide a consistent interface to different data types.
This specification is for the generic mechanism of data
synchronisation. Further specifications define the data models for
different data types that may be synchronised via JMAP.
JMAP is designed to make efficient use of limited network resources.
Multiple API calls may be batched in a single request to the server,
reducing round trips and improving battery life on mobile devices.
Push connections remove the need for polling, and an efficient delta
update mechanism ensures a minimum amount of data is transferred.
JMAP is designed to be horizontally scalable to a very large number
of users. This is facilitated by separate endpoints for users after
login, the separation of binary and structured data, and a data model
for sharing that does not allow data dependencies between accounts.
For context: RFC 8620 is the JMAP core, just published. It defines an RPC framework, an object synchronisation protocol, and miscellaneous other things like push channels. It has no data model associated with it, beyond handling blobs.
The JMAP mail spec, which is what most people will be more interested in, is awaiting final sign-off before publication as RFC 8621.
We’re using JMAP in production at Fastmail on our Fastmail and Topicbox products, right through the stack. Both of their web interfaces speak JMAP to the server. Unfortunately at this time it’s not ready for public use: definition of authentication technique is one thing that is deliberately left out of JMAP, and we’re currently using our own system which has no public documentation (and we don’t expose /.well-known/jmap yet either). The intrepid user may be able to figure it out, but we won’t be providing any support for it yet. We use the JMAP mail spec, the draft calendar and contacts specs, and a handful of our own specs for our own data types, also not publicly documented at this time.
JMAP is much more complex than the standard sort of simple REST API, but there is cause for all aspects of its complexity; the most notable complexity in it is its record state management, part of it being an object synchronisation protocol; an advantage of that is that robust offline support is feasible, even fairly straightforward. I’m looking forward to working on offline support in the Fastmail web app.
I personally think the increased complexity is well worth-while, and I have several products I’ve been sketching out and working on personally (completely unrelated to the world of email or to my employer Fastmail), and all the ones with any kind of client-server API I intend to build atop JMAP, because they’ll benefit heavily from it with respect to features like real-time interface updates and offline support (for web, desktop and command line programs).
I've hacked IMAP long enough to want it thoroughly dead, so I've been following JMAP from afar and waiting for Fastmail to publish this for a long time. Props to the team, and I hope the (publicly) missing parts will come soon enough.
Are there open-source JMAP implementations you recommend? I understand not wanting to commit to the protocol for customers but I suspect it's been dissuading the creation of third-party clients since there's nothing to use it with.
(Also, a little bit off-topic, but as a Pobox customer, can I use JMAP / am I already using it from the webmail interface?)
Our own implementation sits atop the open-source Cyrus mail server. I’m not sure if JMAP is enabled by default in stable releases yet, or whether you have to go for the bleeding edge to get it.
I personally have no basis upon which to make any recommendations for JMAP clients, servers or libraries. The JavaScript one we use in our webmail is not particularly suitable for others to use, despite being open source, as it’s quite tied to our Overture web framework which pretty much only we use.
If you say 'object sync protocol', would it work to build a dropbox alternative? I'd love for you guys to make one, I'd pay a premium over the 120/yr I now pay Dropbox for a sync-only product that you would just leave alone (well, as long as it's in a working state) after that, ie none of the fucking around dropbox is doing.
I've lately toyed with using Avro in place of JSON and found it shows some fairly substantial savings. It has a JSON encoding as a backstop for places where the binary encoding isn't kosher.
Section 1.5 of the RFC nails the colours to I-JSON with a "MUST". Do you envisage any amendment to allow equivalent encodings? I know this is indirectly discussed elsewhere, I'm just going through an Avro infatuation.
All JMAP servers must support that format, but that doesn’t preclude supporting other formats via extensions. We use custom compression, for example; and a WebSocket transfer layer for the API parts, instead of HTTPS, is being standardised. In such cases, the server declares its support for such capabilities in the session object (which will be in the baseline format, JSON), and clients that know about them can opt into their behaviours, knowing that the server supports it.
Do you know why in particular 8621 is still in AUTH48? Usually when there's a family of new standards they all get issued essentially simultaneously that I've seen.
Yeah, a couple of last minute editorial things - "headers" -> "header fields", an example to show the use of Content-* headers for single-part messages because it was considered a little unclear, and an update to the contact address for the IANA registrations. They should be cleared up this week with everyone (except me, boohoo, couldn't make it due to local commitments at home) in Montreal for IETF105.
IMAP's killer design flaw was that it imposed a strict UID consistency requirement between replica servers if client sync was to work correctly, i.e. you needed a master-slave setup or Paxos, you couldn't use a true multi-master setup or CRDTs with eventual consistency because of the possibility that clients might miss UIDs.
There have been attempts to solve this with even-odd UIDs etc. but those broke down on further analysis. There have also been papers claiming to implement multi-master IMAP servers over CRDTs but these never addressed the strict UID consistency requirement and never tested with actual client implementations for sync correctness.
How does JMAP address the distributed multi-master server sync problem, assuming eventual consistency?
> IMAP's killer design flaw was that it imposed a strict UID consistency requirement between replica servers if client sync was to work correctly
That may be a problem for folks running things at the proverbial web scale, but if it's just a Dovecot server handling a couple of dozen or hundred accounts, is it that big of an issue?
No, unless you're designing a protocol to replace IMAP and you want all the proverbial web scale systems to adopt your new protocol. And then again, interoperability and universal adoption is probably something the smaller Dovecot installations would want.
Master-master conflict-free eventual consistency systems are easier and safer to operate, there's no nightmare damage from split-brain and things just work, even in the worst scenario. Master-slave failover is notoriously prone to error, even for small deployments.
This isn't on topic of GP, but you seem to have a handle on what this is, so I thought I'd ask...
The conversation seems to focus on JMAP as a replacement for IMAP and primarily as an email protocol. The RFC identifies itself as -
> a protocol for clients to efficiently query,
> fetch, and modify JSON-based data objects, with support for push
> notification of changes and fast resynchronisation and for out-of-
> band binary data upload/download.
It reads like it applies to APIs in general, or is this only for mail/messaging services?
edit: I see now, this is the core spec, not specifically the mail spec as referenced in another comment. Thanks.
Note that you can retrieve arbitrary files over IMAP too - there's not that much "email" about it. And as one quickly realize: it makes sense to solve the general problem first (implement a document database, essentially) and then add "special sauce" for email (eg: a notion of sender/receivers, threads etc).
I've wondered for a while how hard it would be to wrap dbmails[db] (postgres) schema/db in graphql and call it a day, for a jmap work-a-like, for example.
It does describe APIs in general, and that is one of the reasons I like it so much. All of our internal APIs are JMAP: when you need to do something with a user, you call User/set; with a rule, call Rule/set, and so on.
JMAP makes it super easy to build new APIs because you can reuse most of the code; the only things you need to define are the data types (and any custom implementation for access controls, etc.).
Although Fastmail itself has much more legacy code (we’ve been around a while!), new things can start with a framework called Ix, which is available at https://github.com/fastmail/Ix. Ix is the basis for Topicbox (www.topicbox.com), which is built ground-up on top of JMAP.
> IMAP's killer design flaw was that it imposed a strict UID consistency requirement between replica servers if client sync was to work correctly, i.e. you needed a master-slave setup or Paxos, you couldn't use a true multi-master setup or CRDTs with eventual consistency because of the possibility that clients might miss UIDs.
+1, and it's been a while since I've dealt with this, so my memory's fuzzy, but I think it's even worse than that in a couple ways:
* The failure mode here is not just that the client misses UIDs but that a given mailbox/uidvalidity/uid means something different to the client than to this replica. Then when the client says to delete one message, they delete a different one. This would be a lot less likely if the server could use a uuid for a message (so collisions are unlikely) instead of following a sequence and/or if it the identifier didn't need to change whenever it moves to another folder. (My memory is even fuzzier here, but iirc in gmail's case, its emulation of the folder model also means a message can have multiple uids at a time, one per label. That mismatch between the folder setup clients expect and the label model is another problem with IMAP. Strictly speaking, I think you can do it all with one folder and an IMAP flag for each label; it's just that the clients don't expect it to work this way.)
* Even with a synchronous replication design, there's the possibility that due to a bug or bad machine the server replicas won't match entirely, or just the client state doesn't match. And if you discover and fix such a problem on the server side, your only option for addressing the bad client state is to bump uidvalidity, which is super annoying to the client. It'd be nicer if you could do something that kicks off some softer sync that doesn't use as much bandwidth, lets the client be more usable while it's happening, and/or lets the server know what mismatches were actually found for diagnostic purposes.
I haven't looked into JMAP before, but I hope it materially improves this situation...
Author of JMAP here.
* JMAP uses opaque ids rather than sequential ids, which makes a lot of this easier (the servers can ensure they have their own id namespace to avoid collisions).
* The resync mechanism (see the "/changes" method) just indicates something may have changed and then you fetch those records. It's definitely possible to make your master-master replication work so that /changes works regardless of which master you were previously talking to, at the expense of some loss of efficiency (you may end up refetching some data you actually already had).
They were built with reference to how Google do theirs so it will be relatively easy for them to support the standard names, similar to how special-use was built to give similar behaviour to XLIST.
The JMAP Calendar spec is the next one being standardised. There's a first draft at https://tools.ietf.org/html/draft-ietf-jmap-calendars-00 but quite a lot more to be added still. Expect a final standard probably first half of next year.
Is there an extensible JMAP core (what this RFC presents) server framework available? I know there are JMAP mail/contacts/calendar server implementations, but it'd be interesting to have a fairly clean starting point for other potential applications...
101 comments
[ 1.6 ms ] story [ 159 ms ] threadI'd prefer s-expressions.
[0]: http://tutorials.jenkov.com/ion/ion-performance-benchmarks.h...
Not sure there is a decent standard spec and a good Java (say) parser around. Is there?
There are no RFCs for s-expressions, but that's fixable, I guess. And yes - there are a few Java parsers for them.
Whether it's the “wrong thing” is another issue, though I've yet to see a standard of “wrong” proposed in the thread for which that is accurate.
And, perhaps, all those people know better than you.
Wide adoption and popularity trumps "being wrong" everyday.
And not just for web stuff: the way you describe a custom architecture to rustc is via JSON.
Here's a Java parser: https://github.com/bpsm/edn-java
> it even comes with a lot of overhead (colons, quotes, brackets and the like).
according to Sqlite devs the that overhead is not much worse that for a binary encoding.
quoting: https://www.sqlite.org/json1.html
> The json1 extension does not (currently) support a binary encoding of JSON. Experiments have been unable to find a binary encoding that is significantly smaller or faster than a plain text encoding. (The present implementation parses JSON text at over 1 GB/s.)
Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. There are too many "yes buts" in the living standard.
> that overhead is not much worse
It _is_ much worse than the two-byte "overhead" of one s-exp node.
I don't believe you. Atoms are roughly the same, and compounds, well:
I suppose if you have a lot of booleans you might save a little by using T/NIL or #t/#f, but otherwise it's more or less a wash.(You might suggest plists, ("a"1"b"2), for dictionaries, in which case you save a whopping two bytes per entry. But I'd choose the nested-list format over plists, if I were encoding a dictionary into an S-expression.)
Which pretty much disappears when compressed anyway.
Anyway, for being a Fasmail user, JMAP rips IMAP to shreds in terms of performance. I've been following JMAP from afar and waiting for Fastmail to publish this for a long time.
People have ben actively doing this for XML with streaming parsers that emit events, where you don't just validate XML itself but the semantics of the payload you're receiving, and bail out early instead of parsing the whole XML into a DOM then validating the DOM structure and values.
Could you give an example? The only deviation from the standard I've run into is JSONL being called JSON, which isn't actually meaningfully harder to parse.
I feel like it should be possible to translate from valid JSON to valid sexpr with a regexp parser (just match parenthesis and stuff)
> There are too many "yes buts" in the living standard.
I am not an expert and do not want to sound dismissive, but I feel like most of those "yes buts" are a solution to otherwise incompatible implementations. I fail to see where S-expressions have an advantage here. Is there a widely recognized standard that most implementations respect?
Do you mean parsing the JSON bytes itself into maps and lists and atoms, or turning that into an application-useful object?
If the former, are you claiming that Python's standard json library, libjansson, JSON::XS, etc. have bugs? (I don't know why you would roll your own JSON parser in production code.)
If the latter, how are S-expressions any different?
(And there are non-buggy libraries that do both, e.g., serde.)
Edit: I see, JMAP moves attachments out of band
Base64 mixes the byte, but Hexadecimal does not.
Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust.
Professionally, I happen to write mostly JavaScript, and no Rust at all, but we do still focus on performance at Fastmail. (It’s in the name!)
If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible.
But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON.
When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
All in all, JSON is a robust choice for just about any API.
Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. Neil Jenkins gave a talk about this and related matters last month, https://www.webdirections.org/code/speakers/neil-jenkins.php, no public video sadly. I’ve thought about writing a blog post about it all. An unfortunate side-effect of this compression is that you can’t inspect traffic in the network panel of the dev tools; you can add ?compress=0 to disable it, if you’re an interested Fastmail user.
MessagePack has an ugly history with text data that's still reflected in some of the libraries interfaces. E.g., the Python library doesn't round-trip strings by default. (They'll turn into bytes objects.) There's a flag to do the right thing, but not everyone is aware of it.
Add to that that CBOR has an RFC, and I'll pick it over MessagePack every time.
Binary formats are mostly nicer from a parsing perspective: it's much easier to unpack. You're right, I think, that simple compression will mostly close any size gap.
I don't think that's a deal-breaker. protobuf has an extension mechanism. [1] You could have a registry for an org to claim an extension number to a given message (and then use it to hold their own message with as much stuff in it as they like). I'm not as familiar with cap'n proto but assume it has something similar.
That said, I think JSON was a fine choice for this purpose. The efficiency is good enough with standard implementations; and there are much faster-than-standard implementations available if you have a special reason to want one. [2]
[1] https://developers.google.com/protocol-buffers/docs/proto#ex...
[2] https://github.com/lemire/simdjson
What do you mean? Both Protobuf and Cap'n Proto are designed to be extensible. You can add new fields without breaking compatibility.
(Disclosure: I wrote Protobuf v2 and Cap'n Proto.)
That lets you store the the common things in highly compressed protobuf, and extensions can either be done via extension protos, or with stringly-typed map stuff for testing/weird stuff.
Forgive me; I am not speaking from a strong position of grokking Cap’n Proto or Protobuf (I’m casually familiar with them but have really only once used them, maybe three years ago), and I believe I erred in what I said.
For Cap’n Proto, I don’t believe I’m far off the mark: to pick a simple example, extensions may add fields anywhere, and I believe you’re not allowed to skip field numbers, so you can’t just establish a registry of used field numbers and pick and choose the extensions.
proto3 looks like it probably can’t handle this either.
On proto2, though, it looks like I was approximately wrong, and you could have finite extensibility, and maintain a registry of assigned IDs so that extensions can be compatible. I’m not sure how suitable that would be in practice—the examples I see only allot a hundred numbers for extensions, which seems rather tiny; I’d picture allowing at least a million.
Anything like Map<str, str or any or whatever> I’m going to disqualify from consideration because either you use structs in some places and thereby limit the places that extension can apply, or you use maps right through and then get no benefit over JSON.
So I think they’d still be mostly technically unsuitable, but maybe Protobuf v2 would work. And maybe I’m completely wrong still—you’re the expert! But really, unless you get a major advantage out of such a thing, an important consideration is social suitability. Protobufs and the likes are not widely-used, require more effort to work with, and require more involved library support than JSON, which is much more universally understood and handled. Just look at how I’ve been fumbling around with them!
That is indeed tricky. With protobuf you could claim some high-numbered fields and hope that no one else does the same thing -- with some randomization you can probably be pretty confident. With Cap'n Proto you can't really do it at all since fields are identified by offset, and making structs arbitrarily large is probably not what you want. With JSON, because fields are identified by name, it's possible to choose a name that almost certainly will not collide with anyone else.
I'd argue that the right answer here would be to ask the spec owners to add extension points, which hopefully would not be controversial. Conceptually this could take the form of something like map<string, any> (there are cleaner idiomatic ways to do it in each of Protobuf and Cap'n Proto but the details are beside the point of this conversation).
There is: https://people.csail.mit.edu/rivest/Sexp.txt
It's imperfect, but it does get the job done. Among its advantages is that it's both pure text and binary: there's a canonical representation which is binary, and an advanced representation which is textual, and they map 1:1 to one another.
Examples:
Or the binary encoding: {KDc6c2Vzc2lvbigxMjpjYXBhYmlsaXRpZXMoMjU6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29y ZSgxNTptYXgtc2l6ZS11cGxvYWQ4OjUwMDAwMDAwKSgyMTptYXgtY29uY3VycmVudC11cGxv YWQxOjgpKDE2Om1heC1zaXplLXJlcXVlc3Q4OjEwMDAwMDAwKSgyMDptYXgtY2FsbHMtaW4t cmVxdWVzdDI6MzIpKDE4Om1heC1vYmplY3RzLWluLWdldDM6MjU2KSgxODptYXgtb2JqZWN0 cy1pbi1zZXQzOjEyOCkoMjA6Y29sbGF0aW9uLWFsZ29yaXRobXMoMTU6aTthc2NpaS1udW1l cmljMTU6aTthc2NpaS1jYXNlbWFwMTc6aTt1bmljb2RlLWNhc2VtYXApKSkoMjU6dXJuOmll dGY6cGFyYW1zOmptYXA6bWFpbCkoMjk6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29udGFjdHMp KSgzMTpodHRwczovL2V4YW1wbGUuY29tL2FwaXMvZm9vYmFyKDE3Om1heC1mb29zLWZpbmFn bGVkMjo0MikpKDg6YWNjb3VudHMoNjpBMTM4MjQoNDpuYW1lMTY6am9obkBleGFtcGxlLmNv bSkoMTE6aXMtcGVyc29uYWw0OnRydWUpKDEyOmlzLXJlYWQtb25seTU6ZmFsc2UpKDIwOmFj Y291bnQtY2FwYWJpbGl0aWVzKDI1OnVybjppZXRmOnBhcmFtczpqbWFwOm1haWwoMjM6bWF4 LW1haWxib3hlcy1wZXItZW1haWwzOm5pbCkoMTc6bWF4LW1haWxib3gtZGVwdGgyOjEwKSko Mjk6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29udGFjdHMzWiYpKSkoNjpBOTc4MTMoNDpuYW1l MTY6amFuZUBleGFtcGxlLmNvbSkoMTE6aXMtcGVyc29uYWw1OmZhbHNlKSgxMjppcy1yZWFk LW9ubHk0OnRydWUpKDIwOmFjY291bnQtY2FwYWJpbGl0aWVzKDI1dXJuOmlldGY6cGFyYW1z OmptYXA6bWFpbCgyMzptYXgtbWFpbGJveGVzLXBlci1lbWFpbDE6MSkoMTc6bWF4LW1haWxi b3gtZGVwdGgyOjEwKSkpKSkoMTY6cHJpbWFyeS1hY2NvdW50cygyNTp1cm46aWV0ZjpwYXJh bXM6am1hcDptYWlsNjpBMTM4MjQpKDI5OnVybjppZXRmOnBhcmFtczpqbWFwOmNvbnRhY3Rz NjpBMTM4MjQpKSg4OnVzZXJuYW1lMTY6am9obgo=}This is the equivalent of the JSON:
Yes, clearly it's the JSON. How do you tell at a glance in the s-expr formatted data what's an object and what's an array? What's a keyword and what's a literal? I can barely even tell the heirarchy where it's very visually obvious with the JSON.
The JSON and s-expr have a different indent level, which is why the hierarchy is less obvious to you.
Debatable. To me there is a clear difference between a list of things of the same type, and a mapping of names to values of potentially varying types. I don't think the specific programming language really matters, they are two totally different concepts with much different applications.
> There's also no way in JSON to distinguish between an int and a float and a decimal type
True, but they are just small literals which are easy to differentiate at a glance, not big multi-line structures like objects or lists. Plus s-expressions have the same issue.
There are no objects — it's all lists (or arrays, if you prefer). This is a good thing, because ordering is always the same.
> What's a keyword and what's a literal?
Everything's a byte string: any interpretation above that is a matter for the protocol. This is a good thing, because it means you neatly sidestep issues like JSON's numbers actually being floats rather than integers, as well as issues like numeric precision.
'But I want types!' Of course, they are good. But the type of a number isn't just integer or float: it's integer-which-is-a-known-version or float-which-is-between-zero-and-one-exclusive. You need to check those types at ingestion anyway; it's not a big leap from integer-which-is-a-known-version to string-which-parses-as-an-integer-which-is-a-known-version.
> I can barely even tell the heirarchy where it's very visually obvious with the JSON.
They are both indicated with indentation. Probably I should have used two spaces for the JSON indentation, but I was lazy.
Why not? I'm honestly curious, because I cannot comprehend preferring that mess of JSON to the s-expressions. The JSON just looks like a dog's breakfast, while the s-expressions look clean & easy to write, read, parse and use.
Also, I don't think they are functions: they're just lists with a convention. Those lists can be used to expression function calls, as in Lisp, but in a data format I think that the lists more often express structure.
One thing I love about s-expressions is that one indicates the type as the first item in the list, e.g.:
whereas with JSON one has to invent pseudo-properties: (yikes, that verbosity is ugly)Why are the quotes needed? It would be a simply change in the spec if names without spaces in them could be expressed without quotes.
Having to quote the symbols is terrible though.
I guess that's part of the point from my point of view: it's not about hash tables or objects or even lists, but about what they represent. In my view focusing on the implementation (a hash table mapping strings to hash tables mapping strings to numbers or strings) is looking at the trees and missing the forest.
I actually have a (I think) rather novel approach and there's some potential for little optimizations here and there, but it disgusts me that I'm doing this at all. The optimizations popular, shitty formats like JSON receives are purely wasteful.
It's suitable, though. The WWW wasn't made right from the start and there's always an idiot somewhere looking to add more shit to the pile. Not only does JavaScript exist, but it morphed into this ugly, inefficient, very poor ''format''. We don't have Lisp machines, but new processors are getting instructions to cope with JavaScript idiocy.
A numerical (binary) format is clearly best. It's small, random-access when properly designed, and efficient, but hey, everything's being pushed into port 80 so why not push all data into JSON?
Now, the rest of this message is in response to @chrismorgan from an organization pushing this filth.
>Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust.
We agree on that, then, but I prefer to write that Rust is just the effeminate man's Ada.
>If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible.
This is wrong. Firstly, we should clarify what is meant by S-expressions. Let's assume the S-expressions have lists, strings, symbols, and numbers. That's already simpler than JSON. You're not hampered by JavaScript's floating point nonsense, either. There's fewer edge cases and less escaping. JSON is, at best, a worse-is-better form of S-expressions.
>But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON.
So, this ties into what I wrote before I even read your post, exactly. Cretins continue to push everything into the WWW protocols and act like their garbage is standard, simple, acceptable, or wanted; it's only technically standard, but it's a bad standard. I'd bet my foot you also like ''DNS over HTTPS''. A numerical (binary) format should be used, because it's the best option and people can be expected to bother writing an acceptor for such important data. There are plenty of languages that can easily accept a numerical format, but not JSON; think of, say, Forth. Oh, but everyone writes in Rust and JavaScript nowadays, right.
>When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
All of this is unnecessary. Email doesn't need this drivel.
>All in all, JSON is a robust choice for just about any API.
Of course; forty years ago, UNIX was the solution for every problem. The WWW and JavaScript are nothing more than a continuation of this worse-is-better filth.
>Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLAT...
The fact that Fastmail, a well known business that is entirely based on successful deployment of email to paying customers has defined a protocol for object exchange etc that is based on Internet RFCs and standards is a Good Thing.
"Cretins" don't decide to use JSON, JSON is an object encoding that is supported by all languages and environments. It compresses to the same level as binary based encodings. It allows for arbitrary extensions that can be migrated in production without having to update all clients/servers simultaneously.
Binary formats tend to be brittle and require simultaneous end-to-end upgrades to handle change. Protobufs and other TLV based encodings help with this, but they still require recompilation and synchronization of IDL specifications.
Compression and caching are easy to insert into a network pipeline (and/or a process-to-process pipeline) easily and transparently. If you read Fielding's thesis on REST, one of the advantages of the HTTP "stack" (not HTML) is that caching and other "dependency injection" can occur transparently between the endpoints.
Efficiency is not by putting all your eggs in one basket. Efficiency comes from a) designing the protocol well, b) keeping the encoding decisions well away from the protocol's content so that decisions like caching and compression etc can all be added or removed without changing the underlying implementation.
All languages "accept" a "numerical format" (not sure what you mean here, but I assume you mean binary encoding. Unless you mean arbitrary length binary numbers. Do you think binary should continue to be on octet boundaries? Or are you talking about encoding an arbitrary bit length? How will you deal with word endianess?
JSON has some warts, the quoting of keys is a pain, the fact that comments cannot be included (but ignored) in the same way as whitespace makes it harder to document configurations or other static documents. The use of floating point is another wart, but if you are sending any numeric values that rely on floating point accuracy, your protocol is wrong. Define a fixed point integer definition instead (eg send monetary values in cents, not dollars).
Email obviously does need this "drivel".
A quick scan of it says to me it doesn't get all the issues but it does get the biggest one, which IMHO are the ambiguities in what constitutes a "number" in JSON. I-JSON explicitly limits it to 64-bit floats.
That's subjective and it seems plenty of people have the opposite opinion.
> unsafe to parse
Is this based on theoretical concerns or practical experience?
> comes with a lot of overhead
Keep in mind the alternatives it is largely replacing. XML, text-based email protocols each with their own unique grammars, etc.
> I'd prefer s-expressions.
Ah yes, the plaintive cry of the lisper in the desert. ;-)
I'm afraid in the 60+ years that s-exps have had to catch on in the mainstream, they haven't.
I can't explain this result but nonetheless it seems irrational to expect it to suddenly change.
s-expressions seem like an out-of-left-field choice.
JMAP is a generic protocol, but its primary motivating example and the thing that has influenced the design most is email, as a replacement for SMTP (between the user and server, not between servers) and IMAP.
The name JMAP was chosen quite early on. I + 1 = J. The words the acronym stands for have changed a couple of times along the way!
IMAP was originally conceived essentially to allow a very thin client that proxies every user command back to the user store, and it also embeds some reliance on old implementation artifacts (UIDVALIDITY, anyone?). There are a lot of "fun" corner cases--I once found a message that was interpreted 5 different ways by 4 different implementations. (It doesn't help that MIME structure is actually pretty loopy and maps very poorly to how all modern clients view mail messages, resulting in many interesting interpretations).
Modern email clients are all structured very differently from the "thin client" perspective--everyone keeps their own copy of the email (metadata) database locally, and use IMAP essentially as a database synchronization protocol, but IMAP is pretty bad at synchronization, at least until the LEMONADE additions came in ~a decade ago. IMAP is especially bad at handling multiple folders correctly, since it assumes that every folder is independent of the other ones, and you can only sit and wait for messages on one folder per connection at a time.
The JMAP mail spec, which is what most people will be more interested in, is awaiting final sign-off before publication as RFC 8621.
We’re using JMAP in production at Fastmail on our Fastmail and Topicbox products, right through the stack. Both of their web interfaces speak JMAP to the server. Unfortunately at this time it’s not ready for public use: definition of authentication technique is one thing that is deliberately left out of JMAP, and we’re currently using our own system which has no public documentation (and we don’t expose /.well-known/jmap yet either). The intrepid user may be able to figure it out, but we won’t be providing any support for it yet. We use the JMAP mail spec, the draft calendar and contacts specs, and a handful of our own specs for our own data types, also not publicly documented at this time.
JMAP is much more complex than the standard sort of simple REST API, but there is cause for all aspects of its complexity; the most notable complexity in it is its record state management, part of it being an object synchronisation protocol; an advantage of that is that robust offline support is feasible, even fairly straightforward. I’m looking forward to working on offline support in the Fastmail web app.
I personally think the increased complexity is well worth-while, and I have several products I’ve been sketching out and working on personally (completely unrelated to the world of email or to my employer Fastmail), and all the ones with any kind of client-server API I intend to build atop JMAP, because they’ll benefit heavily from it with respect to features like real-time interface updates and offline support (for web, desktop and command line programs).
(Also, a little bit off-topic, but as a Pobox customer, can I use JMAP / am I already using it from the webmail interface?)
I personally have no basis upon which to make any recommendations for JMAP clients, servers or libraries. The JavaScript one we use in our webmail is not particularly suitable for others to use, despite being open source, as it’s quite tied to our Overture web framework which pretty much only we use.
There is, however, a list at https://jmap.io/software.html of software that supports JMAP.
On your Pobox question: when reading your email at fastmail.com, you’re using the Fastmail webmail, which speaks JMAP.
Section 1.5 of the RFC nails the colours to I-JSON with a "MUST". Do you envisage any amendment to allow equivalent encodings? I know this is indirectly discussed elsewhere, I'm just going through an Avro infatuation.
There have been attempts to solve this with even-odd UIDs etc. but those broke down on further analysis. There have also been papers claiming to implement multi-master IMAP servers over CRDTs but these never addressed the strict UID consistency requirement and never tested with actual client implementations for sync correctness.
How does JMAP address the distributed multi-master server sync problem, assuming eventual consistency?
That may be a problem for folks running things at the proverbial web scale, but if it's just a Dovecot server handling a couple of dozen or hundred accounts, is it that big of an issue?
No, unless you're designing a protocol to replace IMAP and you want all the proverbial web scale systems to adopt your new protocol. And then again, interoperability and universal adoption is probably something the smaller Dovecot installations would want.
Master-master conflict-free eventual consistency systems are easier and safer to operate, there's no nightmare damage from split-brain and things just work, even in the worst scenario. Master-slave failover is notoriously prone to error, even for small deployments.
The conversation seems to focus on JMAP as a replacement for IMAP and primarily as an email protocol. The RFC identifies itself as - > a protocol for clients to efficiently query, > fetch, and modify JSON-based data objects, with support for push > notification of changes and fast resynchronisation and for out-of- > band binary data upload/download.
It reads like it applies to APIs in general, or is this only for mail/messaging services?
edit: I see now, this is the core spec, not specifically the mail spec as referenced in another comment. Thanks.
source: https://news.ycombinator.com/item?id=20478227
I've wondered for a while how hard it would be to wrap dbmails[db] (postgres) schema/db in graphql and call it a day, for a jmap work-a-like, for example.
[db] http://dbmail.org/
It does describe APIs in general, and that is one of the reasons I like it so much. All of our internal APIs are JMAP: when you need to do something with a user, you call User/set; with a rule, call Rule/set, and so on.
JMAP makes it super easy to build new APIs because you can reuse most of the code; the only things you need to define are the data types (and any custom implementation for access controls, etc.).
Although Fastmail itself has much more legacy code (we’ve been around a while!), new things can start with a framework called Ix, which is available at https://github.com/fastmail/Ix. Ix is the basis for Topicbox (www.topicbox.com), which is built ground-up on top of JMAP.
+1, and it's been a while since I've dealt with this, so my memory's fuzzy, but I think it's even worse than that in a couple ways:
* The failure mode here is not just that the client misses UIDs but that a given mailbox/uidvalidity/uid means something different to the client than to this replica. Then when the client says to delete one message, they delete a different one. This would be a lot less likely if the server could use a uuid for a message (so collisions are unlikely) instead of following a sequence and/or if it the identifier didn't need to change whenever it moves to another folder. (My memory is even fuzzier here, but iirc in gmail's case, its emulation of the folder model also means a message can have multiple uids at a time, one per label. That mismatch between the folder setup clients expect and the label model is another problem with IMAP. Strictly speaking, I think you can do it all with one folder and an IMAP flag for each label; it's just that the clients don't expect it to work this way.)
* Even with a synchronous replication design, there's the possibility that due to a bug or bad machine the server replicas won't match entirely, or just the client state doesn't match. And if you discover and fix such a problem on the server side, your only option for addressing the bad client state is to bump uidvalidity, which is super annoying to the client. It'd be nicer if you could do something that kicks off some softer sync that doesn't use as much bandwidth, lets the client be more usable while it's happening, and/or lets the server know what mismatches were actually found for diagnostic purposes.
I haven't looked into JMAP before, but I hope it materially improves this situation...
https://datatracker.ietf.org/doc/rfc8474/
They were built with reference to how Google do theirs so it will be relatively easy for them to support the standard names, similar to how special-use was built to give similar behaviour to XLIST.
https://news.ycombinator.com/item?id=19839104
https://news.ycombinator.com/item?id=18996200
https://news.ycombinator.com/item?id=18766709