Protobufs and similar projects like it (gRPC, Cap'n Proto) seem really interesting, but I haven't come across a time at work yet where it's made sense to the team to adopt it. Maybe that's just my own inexperience, but the serialization scheme is low on the list compared to optimizing DB queries, getting rid of bloat in the app, etc. I've been waiting for an excuse to adopt this stuff at work because it seems really cool!
I guess I'll need to do some research and propose to the team that we test it out at some point. Things like Swagger just seem...clunky in comparison to protobufs.
Dealing with data as my day job I've become highly sensitive to schema's. And I hate those "schemaless" (aka schema-on-read) serialization formats more and more. No, there is no schemaless, there is a schema, but it is buried in your code in a convoluted way on each line where you read and interpret your deserialised data and all tests and assumptions you have there are a horrible representation of your schema. That is an important dimension to compare serialization on, and JSON is a complete loser in that sense. Not to mention how it flawed its numeric type is.
I really wish we had a JSON 2 format which could fix JSON's obvious shortcomings. I would like to see:
- Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys)
- A standard Date format
- Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding.
- Arbitrary precision integer support. This is particularly useful for cryptocurrencies and for interoperability with 64 bit integers in other languages. And bigints are coming to javascript - https://github.com/tc39/proposal-bigint
I've heard of JSON implementations in other languages making bigger JSON numbers than javascript supports; but I didn't realise JSON.parse would quietly parse them and throw away precision in the process. I wonder what the plan is for JSON serialization support of bigints. I assume there is no plan - Chrome stable 67 supports bigints, but JSON.serialize(2n) throws a TypeError.
In theory, that is true. In practice, if the consuming client is javascript, it is really hard to map the numbers into an arbitrary precision number upon decoding (which would come from some library like big.js), rather than a pure double.
Which means, in practice, you are stuck passing numbers as strings (and on the javascript side, pass that string into the arbitrary precision number library)
Maps and Sets are entirely irrelevant when it comes to serialization. You may as well store or transmit that information as an array (of pairs, in case of Map). The point of a Map or Set (O(1) insertion / removal / contains) don't matter when you're talking about serialization.
Well, unlees you're using something like capnp, where O(1) contains is meaningful when O(n) deserialise is not acceptable.
And really, if you have consumers/producers in different languages/codebases, built-in support for these things is a great convenience. You wouldn't say that objects/dicts in JSON are irrelevant, would you?
> You wouldn't say that objects/dicts in JSON are irrelevant, would you?
I kind of would. Coming from Erlang, I don't see a point to there being a whole separate syntax for encoding object/map types. Just encode objects/maps as arrays of pairs (2-Tuples, or just length-2 arrays).
Then, if you want ser-des type fidelity, stick an annotation onto the array (like in YAML) to say what type it should decode to.
E.g., something like:
[1, 2, 3] # array
[['a', 1], ['b', 2], ['c', 3]] # array of pairs
[@object, ['a', 1], ['b', 2], ['c', 3]] # array of pairs hinted so it should decode to an Object
[@map, ['a', 1], ['b', 2], ['c', 3]] # array of pairs hinted so it should decode to a Map
The nice thing about this approach is that JSON libraries could do as much or as little work in parsing out the annotations as they want: they could recognize the annotations and construct the referred-to type; or they could just pass back the array with the annotation expressed as an Annotation value.
Map and set values are important for many applications. The fact that JSON doesn't have a way of denoting a map or set value (or anything else, but that's another issue) is a problem: it means there's no understanding common to all JSON consumers about what syntax denotes a map or a set. Shortcuts are taken, consumers disagree on the details, and the linked CVE is the result. Being able to reliably convey "this is intended to be a map", or "this is intended to be a set" is crucial for secure and robust interoperability.
That's really weird - why didn't they parse it into a map in the first place, immediately removing duplicate keys? This seems like more of an argument for automatic deserializer generation than it does for adding undue complexity to a serialization format.
Even that's not enough to save you in general. JSON deliberately (!) permits duplicate keys. It leaves it up to the implementation to decide how to handle them.
And there are two (sensible) ways to add key/value pairs to a map: either keep the first occurrence of key X, or the last occurrence of key X.
Some JSON libraries pick one way; others pick the other.
Getting the two to interoperate is, as we see, not easy.
It would have been better for JSON to forbid duplicate keys, or to specify a mandatory first-wins or last-wins policy. Then there'd be no room for error.
> Do you want remote code execution? Because that's how you get remote code execution.
It's fundamentally not the case that any data serialization format without support for map and set values allows for remote code execution. This CVE was CouchDB's auth code not handling some edge cases in the JSON standard.
Your argument that edge cases like this lead to problems is definitely a good one though. I'm more inclined to say that a lot of these kinds of issues are due to people thinking JSON is a simple, straightforward format when it definitely isn't -- and that's due to mostly two things:
- It fits on a business card!
- You can serialize/deserialize in one line in most implementations
Serialization/Deserialization is something you should always pay careful attention to. Making it a one-liner and advertising it as such was pretty irresponsible.
Yaml is well readable by a human but it's much to tightly defined to be "for humans". It's for machines but readable by humans I'd say. Plus it's got sets and you can embed csv's (just 2 things of the top of my head).
No need to extend json itself. But the libraries would need to be extended.
maps can be implemented with 1 array and 1 map, with the keys being the hash of the object. the hash function should probably be written in the Json itself for completeness.
embedded beinary blobs already work. lookup GLB for an example on how to embed binary blobs in Json.
as others said, json already supports arbitrary precision.
> embedded beinary blobs already work. lookup GLB for an example on how to embed binary blobs in Json.
Those aren't embedded binary blobs, those are string representations of base64-encoded binary blobs. Unless they come out of the JSON decoder as a byte array, they're not "working" as part of JSON; they're another standard on top of JSON.
(Also, the GP commenter probably wants them to be transmitted with 0 encoding overhead. Which can't really work while JSON is still JSON. But you can always use an alternative format which handles a superset of JSON's types, and which is binary, e.g. BSON or CBOR.)
> maps can be implemented with 1 array and 1 map, with the keys being the hash of the object. the hash function should probably be written in the Json itself for completeness.
I think you're fundamentally misunderstanding the thrust of the GP poster's request, here? They don't want to serialize a map in a way that is cheap to deserialize—different languages have different hashtable implementations so there's no way it could really work. What the GP poster (and many other people) want, is just to have something that encodes similarly to existing "object maps" (the ones with curly braces), but with a slight syntactic alteration so that they come out of the decoder as Maps, rather than as Objects.
I'm not talking about data URIs. You can write binary data in the same file as the json. Most parsers don't care what comes after the last curly brace.
I'm assuming you are using a weakly typed language. My schema is defined by my data objects and I use the built in validation attributes in C#. If the request doesn't pass the validation part of the pipeline, my controller never gets the request and a Bad Request message is sent back to the client.
Both ASN.1 and XML are standardized (with extensive, lengthy and detailed specifications!). The term "S-expression" covers a very broad family of vaguely related not-very-interoperable surface syntaxes.
S-expressions are standardized in both Common Lisp and Scheme. The two standards are not identical, but are mostly compatible, with the CL standard being mostly a superset of the Scheme standard.
Yes, that's exactly what I mean. It's like saying "we use JSON as our file format", but worse.
> "So this program saves its data as 'S-expressions'."
> "Oh, cool. Which dialect?"
> "Uh, I don't know. It doesn't say in the README. It just says 'S-expressions'."
The answer can be CL, Scheme (lots of variations and dialects even here), the never-finished SPKI Sexps, OCaml sexps, something that the new dev on the team cooked up last Tuesday that vaguely resembles what they learned as "lisp" in college, or something else entirely.
On the other hand, if one were to say "this program uses R4RS S-expressions" (or, presumably, "CL S-expressions", but I haven't read the relevant bits of CLtL), you'd immediately be in a much nicer place than JSON can offer. Not only would you have a well-specified syntax for a reasonably broad range of data types, you'd have a useful equational theory as well. [ETA: Unless you want unicode. Doh. R6RS, maybe.] Ah, the impossible dream.
In the most general sense an XML document can reference schema(s) not locally available; a client wishing to validate against those schemas has to download them.
For most use cases, this isn't relevant, because the application expects a document of schema X and will provide said schema as well.
> I hate those "schemaless" (aka schema-on-read) serialization formats more and more
This, and also I think a lot of people who preach the flexibility of schemaless anything really just wants a more flexible schema definition language, with a lot of "maybe" options and similar stuff.
I removed performance piece mostly because it's meaningless discussing performance without in the context of designated implementation and benchmarks.
Above catch is my point so I tweaked the words to `Performance is not good generally when dataset is huge unless you use a library support streaming parsing or writing.`
I think it's point is that JSON isn't a great streaming format; which it isn't.
That's not to say there aren't streaming decoders available or workarounds to allow non-streaming decoders to emulate streaming behaviour. However generally JSON parsers require the full JSON object before they can decode.
I read somewhere the problem with MsgPack is, that JSON has a rather fast parser build into JavaScript that beats the MsgPack parser. So you would have to check if the saved bandwidth would be enough to justify the slower parsing.
Would be interesting if this still holds true with a WASM implementation.
I used to think binary formats for network protocols were a really good idea until I found out how big the TCP header itself can be. Saving a few bytes from your payload by using binary representations of integers doesn't make a huge difference when the TCP header is 60 bytes.
Just to add another dimension to your analysis, if you're sending large binaries the 33% overhead of Base64 adds up pretty quickly. It may not apply to your use case but if you are delivering images or video over a websocket it can make a big difference.
Here’s my old article about Microsoft’s binary XML format: http://const.me/articles/net-tcp/ As you see, for some payloads it compresses XML to 9% of the original size.
The data contract serializer included in .NET framework (and even in .NET Core) can read & write objects directly from/to this binary serialization format, without intermediate text XML anywhere.
Since WASM can't create or access JavaScript objects directly, I bet it's still slower. MessagePack (or my preference, CBOR) is still useful and faster than JSON if you need to exchange raw bytes from typed arrays (for JSON you'd need to encode them, e.g. in base64).
That's the case with Perl at least. Both JOSN::XS and Cpanel::JSON::XS are faster than Data::MessagePack and Sereal on an iMac 2009 with Yosemite. On Linux Sereal is 53% faster than Data::MessagePack, but the rest of the benchmark is quite similar. CBOR::XS beats just about any data serializer. Sereal is quite sophisticated, it can compress data using snappy, zlib and zstd.
> It requires the program doing data parsing work to have the generated library as well. This would generally cause problem when schema modified.
That's not the case, and is the exact reason why you need to specify tag numbers in protos - so that you can make your schema forward and backward compatible when decoding/encoding from/to the wire format.
I'm working on a product where I had to serialize 100KB+ BLOBs in two places. The first was over stdout from a c program to an elixir application and the second was over a binary websocket, specifically a Pheonix channel.
In both cases I'm using MsgPack. It was easy enough to implement and I like how it has real numeric types.
Don't we do CSV/TSV anymore? It's certainly the most basic schema-less data format imaginable, and you can even have field names using the convention that the first row contains field names.
It might be considered out of fashion today in our staged this-vs-that culture, but I don't see anything wrong with it. Tab-separated-values are just data fields separated by a distinguished character, with rows separated by another distinguished character; as simple as it gets and no API required.
CSV standards are flawed and it has a lot of weird edge cases.
Moreover, byte encoding formats are more concise, so faster to transport over the wire and store and the schema seldom proves to be an hindrance, as it assists in data validation and acts as documentation.
"It might be considered out of fashion today in our staged this-vs-that culture, but I don't see anything wrong with it.... no API required."
Well, it has no type system, not even "string vs. number", no hierarchy of any kind (no objects, no lists, nothing going deeper), and CSV/TSV is actually a meta-specification requiring a correct use of CSV to provide a complete specification (since you can trust literally nothing about a CSV/TSV implementation otherwise) so you can handle the distinguished characters in the data values. Whatever it is you are about to say is the answer to that problem is not; it is an answer and there are others in common use, and correct usage needs to specify them.
Unlike the formats listed, it's completely unclear how to serialize arbitrary objects into it in a way that other people would agree to sight-unseen. That is, yes, I'm aware you can jam anything you want into CSV. For instance, you could make one of the columns be JSON. But that defeats the purpose of claiming it's good enough on its own, no? Same for any other scheme you might propose; unless you can show it to be a standard, it's not an answer, just an ad-hoc solution.
I'm not saying it's useless. Literally used it last Friday to dump some data into something that could be loaded by a spreadsheet program so someone can mix & match the data without having to ask for every individual possible view to come straight from me. Very useful format at times. But as a generalized serialization format? It's not even in the running, which is precisely why it doesn't come up in these discussions.
If you want to get concrete, show me the canonical way you'd lossly serialize the DOM tree nodes for the HN home page into CSV, in a way that you are reasonably sure is the same way everybody else would tend to if given the task, then show me the emitter and generator code that is more convenient than the alternatives.
I'd rather not :) That's what SGML and XML are designed for.
I agree with your points, but I think the pursuit of the one generalized serialization format to rule them all is misguided and scratching an intellectual itch rather solving a practical problem anyone is having.
In my experience, either you want an upfront interface-first design. When you constrain yourself to mainstream techniques, chances are you'll be using XML/XSD, though it's not ideal to describe co-inductive data structures (it's a grammar-based formalism for text after all).
Or you're not working on system boundaries, and your project doesn't benefit from interface-first design (such as in a Web app with a dedicated back-end). Again if you don't feel like introducing non-mainstream formats, you'll be using JSON.
Everything else seems to be prone to falling into an https://xkcd.com/927/ trap at this point.
Sure there are other use cases (binary data, mass data, streaming, whatever). But these are at least as niche as those for TSV/CSV.
It's complicated enough that you need a library to properly use it and simultaneously simple enough that people think they don't need a library. Then there are the billions of possible variations. Nobody can agree on a common column (is it ',' ';' or '\t'), newline seperator, whether the first row is a header and you can only store data in the first normal form. CSV is so terrible that even excel spreadsheets are a better data exchange format.
Serialization format has no relation with schemas. Also, Schema validation can be as strict or lax as you want it to be.
I wrote Spyne (in Python) mainly to abstract the schema from the serialization format. The protocols are totally pluggable and your code cares only about the models and not the serialization format. The latest alpha is out not long ago. Check it out if this sounds interesting.
S-Expressions are missing. Language support is poor (except if you're programming in a Lisp dialect in which case it's built-in), but I do believe they are the best serialization format out there:
As far as I understand S-expressions are completely code-as-data, so how do you protect yourself from malicious code execution when loading S-expressions?
Simply not passing any of these parsed expressions to your eval function.
ANSI Common Lisp presents a pitfall here in that it features read-time evaluation via the #. (hash dot) syntax. For instance #.(+ 2 2) produces the object 4. After seeing #., he reader scans the (+ 2 2) expression, evaluates it immediately, and substitutes the result. When reading untrusted data in Common Lisp, the * read-eval * variable must be set to nil to disable hash-dot.
Lisps that don't have a read-time-eval escape mechanism don't require anything.
One topic not brought up in this is versioning/change management. While I have my criticisms of Json, if I decide to add a new field to my object, most JSON parsers have the advantage that they'll just consider my old data without that field to have a full value there. XML, same thing.
How does protobuf or MsgPack handle that? Aren't they both trying to align data by byte number at some deep level? Or do I not understand them?
Protobuf has supported this for as long as I’ve used it. You add an optional field for the new data. And there’s a couple other considerations. (Docs: [0])
Fields have a number, which you can/should mark as reserved after you take it out. Then future developers won’t use a field that was something else in the past. [1]
Since version all fields are optional ( they removed the required field feature). So your app has to check that the fields you want (in that version of your app) are present.
From memory msgpack is semantically very similar to JSON. It unpacks into arrays and strings and hash tables in more or less the same way, though I think hash table keys are more flexible. It does not come with a schema for object structure or anything, the unpack routines are generic (and pretty "branchy" as a result.)
> Performance is not good when dataset is huge. Program usually needs to load all data into memory first.
You could use a streaming parser, or, more easily, NDJSON (assuming your data is huge, because it's a list of stuff). Just save each JSON as a line in a file and then stream the file line by line, only parsing one line at a time. That's fairly fast and allocates very little memory.
Could you expand on that a bit? I'm interested in promoting gRPC over thrift at work, but so far the only benefits I see are the HTTP/2 transport which allows for better load balancing and request tracing on the transport level.
gRPC with Protobuf has been faster in our usage and also has more development these days compared to Thrift. It's also simpler and better designed by just sticking to a single well-tested serialization format.
HTTP/2 is also a big advantage since it's standardized and easily integrates into many existing load balancers and proxies like Envoy and nginx, both of which now natively support gRPC directly too.
I would also recommend CBOR (http://cbor.io/); one can think of it like a binary form of JSON. It has a few advantages:
* datetime objects
* binary blobs
It is very similar to MsgPack in nature. However, MsgPack, in particular on Python, poorly handles the text/bytes separation, and CBOR is backed by RFC.
BSON is another format that's "binary JSON with blobs and datetimes". It's notably used a lot in MongoDB, but I'm not sure how these three formats compare.
CBOR is MessagePack. At least cbor-ruby started with the MessagePack sources. The story is that Carsten took MessagePack, wrote a standard and added some things he wanted, and called it something else.
I wrote [1] a pretty comprehensive (and admittedly biased) critique of the CBOR standard last year.
I wonder why FlatBuffers aren't a more popular option[0]. The format isn't even mentioned in the article. I have never used it (since I my needs were not that advanced), but it seem to combine quite a few amazing qualities of other serialization formats, while adding a few of its own. It even has a version without a schema called "FlexBuffers"[1]. I would expect anyone considering ProtoBuffers and MsgPack to give FlatBuffers a look.
Two five minute lightning talks by Van Oortmerssen, who created the format (another reason to take a closer look IMO), that shows off some of the powerful features:
I've used it in production, can confirm it is awesome.
Flatbuffers destroys ProtoBuf both in terms of allocations(1 vs N where N is usually huge) and deserialization speed.
You also get explicit control over your data layout in memory when you write the message. So if you know what your read patterns are up-front you can get amazing cache usage even in managed languages.
Our company implemented two rpc protocols(don't ask me why..)
One is based on json, one is modified thrift. The schemaless feature is among the many problems of json, but it's not the most annoying one. That would be binary support.
Imagine we need to make a upload/download storage service with json rpc...So as a rule of thumb, I would consider:
1. Web facing? json would do.
2. RPC? Pick protobuf or thrift or any protocol supporting native types including binary.
90 comments
[ 3.4 ms ] story [ 163 ms ] threadOTOH schemas seem less necessary if you are working on a completely self-contained app.
- Support for Maps and Sets (unlike objects, maps allow arbitrary types to be used as keys)
- A standard Date format
- Embedded binary blobs. No idea how to do this and keep it human readable, but when you need this its super useful. Maybe something similar to WS's binary message encoding.
- Arbitrary precision integer support. This is particularly useful for cryptocurrencies and for interoperability with 64 bit integers in other languages. And bigints are coming to javascript - https://github.com/tc39/proposal-bigint
- Maybe even fix JSON's weird unicode encoding: http://timelessrepo.com/json-isnt-a-javascript-subset
Unfortunately it seems like nobody 'owns' JSON enough to give JSON 2.0 the political weight it would need for cross-language support.
Which means, in practice, you are stuck passing numbers as strings (and on the javascript side, pass that string into the arbitrary precision number library)
And really, if you have consumers/producers in different languages/codebases, built-in support for these things is a great convenience. You wouldn't say that objects/dicts in JSON are irrelevant, would you?
I kind of would. Coming from Erlang, I don't see a point to there being a whole separate syntax for encoding object/map types. Just encode objects/maps as arrays of pairs (2-Tuples, or just length-2 arrays).
Then, if you want ser-des type fidelity, stick an annotation onto the array (like in YAML) to say what type it should decode to.
E.g., something like:
The nice thing about this approach is that JSON libraries could do as much or as little work in parsing out the annotations as they want: they could recognize the annotations and construct the referred-to type; or they could just pass back the array with the annotation expressed as an Annotation value.Map and set values are important for many applications. The fact that JSON doesn't have a way of denoting a map or set value (or anything else, but that's another issue) is a problem: it means there's no understanding common to all JSON consumers about what syntax denotes a map or a set. Shortcuts are taken, consumers disagree on the details, and the linked CVE is the result. Being able to reliably convey "this is intended to be a map", or "this is intended to be a set" is crucial for secure and robust interoperability.
And there are two (sensible) ways to add key/value pairs to a map: either keep the first occurrence of key X, or the last occurrence of key X.
Some JSON libraries pick one way; others pick the other.
Getting the two to interoperate is, as we see, not easy.
It would have been better for JSON to forbid duplicate keys, or to specify a mandatory first-wins or last-wins policy. Then there'd be no room for error.
It's fundamentally not the case that any data serialization format without support for map and set values allows for remote code execution. This CVE was CouchDB's auth code not handling some edge cases in the JSON standard.
Your argument that edge cases like this lead to problems is definitely a good one though. I'm more inclined to say that a lot of these kinds of issues are due to people thinking JSON is a simple, straightforward format when it definitely isn't -- and that's due to mostly two things:
- It fits on a business card!
- You can serialize/deserialize in one line in most implementations
Serialization/Deserialization is something you should always pay careful attention to. Making it a one-liner and advertising it as such was pretty irresponsible.
https://github.com/cblp/yaml-sucks
If you ignore the flame-inducing title, it's just a table showing how different implementations parse YAML input in very different ways.
maps can be implemented with 1 array and 1 map, with the keys being the hash of the object. the hash function should probably be written in the Json itself for completeness.
embedded beinary blobs already work. lookup GLB for an example on how to embed binary blobs in Json.
as others said, json already supports arbitrary precision.
Those aren't embedded binary blobs, those are string representations of base64-encoded binary blobs. Unless they come out of the JSON decoder as a byte array, they're not "working" as part of JSON; they're another standard on top of JSON.
(Also, the GP commenter probably wants them to be transmitted with 0 encoding overhead. Which can't really work while JSON is still JSON. But you can always use an alternative format which handles a superset of JSON's types, and which is binary, e.g. BSON or CBOR.)
> maps can be implemented with 1 array and 1 map, with the keys being the hash of the object. the hash function should probably be written in the Json itself for completeness.
I think you're fundamentally misunderstanding the thrust of the GP poster's request, here? They don't want to serialize a map in a way that is cheap to deserialize—different languages have different hashtable implementations so there's no way it could really work. What the GP poster (and many other people) want, is just to have something that encodes similarly to existing "object maps" (the ones with curly braces), but with a slight syntactic alteration so that they come out of the decoder as Maps, rather than as Objects.
- NaN/inf floats
- Comments
Schemas are defined somewhere, even if it's a poor and bug-ridden definition in the code. Ditto for the point that JSON doesn't count.
I wrote an article, On Schemas, to try and elaborate on this topic a bit.
https://adamdrake.com/on-schemas.html
> "So this program saves its data as 'S-expressions'." > "Oh, cool. Which dialect?" > "Uh, I don't know. It doesn't say in the README. It just says 'S-expressions'."
The answer can be CL, Scheme (lots of variations and dialects even here), the never-finished SPKI Sexps, OCaml sexps, something that the new dev on the team cooked up last Tuesday that vaguely resembles what they learned as "lisp" in college, or something else entirely.
On the other hand, if one were to say "this program uses R4RS S-expressions" (or, presumably, "CL S-expressions", but I haven't read the relevant bits of CLtL), you'd immediately be in a much nicer place than JSON can offer. Not only would you have a well-specified syntax for a reasonably broad range of data types, you'd have a useful equational theory as well. [ETA: Unless you want unicode. Doh. R6RS, maybe.] Ah, the impossible dream.
For most use cases, this isn't relevant, because the application expects a document of schema X and will provide said schema as well.
This, and also I think a lot of people who preach the flexibility of schemaless anything really just wants a more flexible schema definition language, with a lot of "maybe" options and similar stuff.
> Performance is not good when dataset is huge. Program usually needs to load all data into memory first.
This is just downright false. There are plenty of SAX-style JSON parsers.
Above catch is my point so I tweaked the words to `Performance is not good generally when dataset is huge unless you use a library support streaming parsing or writing.`
That's not to say there aren't streaming decoders available or workarounds to allow non-streaming decoders to emulate streaming behaviour. However generally JSON parsers require the full JSON object before they can decode.
Would be interesting if this still holds true with a WASM implementation.
Here’s my old article about Microsoft’s binary XML format: http://const.me/articles/net-tcp/ As you see, for some payloads it compresses XML to 9% of the original size.
The data contract serializer included in .NET framework (and even in .NET Core) can read & write objects directly from/to this binary serialization format, without intermediate text XML anywhere.
https://hastebin.com/wazuqovexo.pl
> It requires the program doing data parsing work to have the generated library as well. This would generally cause problem when schema modified.
That's not the case, and is the exact reason why you need to specify tag numbers in protos - so that you can make your schema forward and backward compatible when decoding/encoding from/to the wire format.
In both cases I'm using MsgPack. It was easy enough to implement and I like how it has real numeric types.
https://github.com/real-logic/simple-binary-encoding
It might be considered out of fashion today in our staged this-vs-that culture, but I don't see anything wrong with it. Tab-separated-values are just data fields separated by a distinguished character, with rows separated by another distinguished character; as simple as it gets and no API required.
Moreover, byte encoding formats are more concise, so faster to transport over the wire and store and the schema seldom proves to be an hindrance, as it assists in data validation and acts as documentation.
Well, it has no type system, not even "string vs. number", no hierarchy of any kind (no objects, no lists, nothing going deeper), and CSV/TSV is actually a meta-specification requiring a correct use of CSV to provide a complete specification (since you can trust literally nothing about a CSV/TSV implementation otherwise) so you can handle the distinguished characters in the data values. Whatever it is you are about to say is the answer to that problem is not; it is an answer and there are others in common use, and correct usage needs to specify them.
Unlike the formats listed, it's completely unclear how to serialize arbitrary objects into it in a way that other people would agree to sight-unseen. That is, yes, I'm aware you can jam anything you want into CSV. For instance, you could make one of the columns be JSON. But that defeats the purpose of claiming it's good enough on its own, no? Same for any other scheme you might propose; unless you can show it to be a standard, it's not an answer, just an ad-hoc solution.
I'm not saying it's useless. Literally used it last Friday to dump some data into something that could be loaded by a spreadsheet program so someone can mix & match the data without having to ask for every individual possible view to come straight from me. Very useful format at times. But as a generalized serialization format? It's not even in the running, which is precisely why it doesn't come up in these discussions.
If you want to get concrete, show me the canonical way you'd lossly serialize the DOM tree nodes for the HN home page into CSV, in a way that you are reasonably sure is the same way everybody else would tend to if given the task, then show me the emitter and generator code that is more convenient than the alternatives.
I'd rather not :) That's what SGML and XML are designed for.
I agree with your points, but I think the pursuit of the one generalized serialization format to rule them all is misguided and scratching an intellectual itch rather solving a practical problem anyone is having.
In my experience, either you want an upfront interface-first design. When you constrain yourself to mainstream techniques, chances are you'll be using XML/XSD, though it's not ideal to describe co-inductive data structures (it's a grammar-based formalism for text after all).
Or you're not working on system boundaries, and your project doesn't benefit from interface-first design (such as in a Web app with a dedicated back-end). Again if you don't feel like introducing non-mainstream formats, you'll be using JSON.
Everything else seems to be prone to falling into an https://xkcd.com/927/ trap at this point.
Sure there are other use cases (binary data, mass data, streaming, whatever). But these are at least as niche as those for TSV/CSV.
I wrote Spyne (in Python) mainly to abstract the schema from the serialization format. The protocols are totally pluggable and your code cares only about the models and not the serialization format. The latest alpha is out not long ago. Check it out if this sounds interesting.
https://github.com/arskom/spyne
Code generator: http://spyne.io
http://wiki.c2.com/?XmlIsaPoorCopyOfEssExpressions
They have a canonical representation
https://en.wikipedia.org/wiki/Canonical_S-expressions
I swear I have seen a proposal for an efficient binary representation somewhere but I can't find it.
ANSI Common Lisp presents a pitfall here in that it features read-time evaluation via the #. (hash dot) syntax. For instance #.(+ 2 2) produces the object 4. After seeing #., he reader scans the (+ 2 2) expression, evaluates it immediately, and substitutes the result. When reading untrusted data in Common Lisp, the * read-eval * variable must be set to nil to disable hash-dot.
Lisps that don't have a read-time-eval escape mechanism don't require anything.
How does protobuf or MsgPack handle that? Aren't they both trying to align data by byte number at some deep level? Or do I not understand them?
Fields have a number, which you can/should mark as reserved after you take it out. Then future developers won’t use a field that was something else in the past. [1]
Since version all fields are optional ( they removed the required field feature). So your app has to check that the fields you want (in that version of your app) are present.
[0] https://developers.google.com/protocol-buffers/docs/proto3#u...
You could use a streaming parser, or, more easily, NDJSON (assuming your data is huge, because it's a list of stuff). Just save each JSON as a line in a file and then stream the file line by line, only parsing one line at a time. That's fairly fast and allocates very little memory.
There's gRPC now, which uses Protobuf messages and is much better than Thrift.
https://grpc.io
HTTP/2 is also a big advantage since it's standardized and easily integrates into many existing load balancers and proxies like Envoy and nginx, both of which now natively support gRPC directly too.
I wrote [1] a pretty comprehensive (and admittedly biased) critique of the CBOR standard last year.
[1] https://news.ycombinator.com/item?id=14072598
Disclaimer: I wrote and maintain a MessagePack implementation.
Two five minute lightning talks by Van Oortmerssen, who created the format (another reason to take a closer look IMO), that shows off some of the powerful features:
Lightning Talk: FlatBuffers (2015)
https://www.youtube.com/watch?v=olmL1fUnQAQ
Going Further with FlatBuffers
https://www.youtube.com/watch?v=90ND0yQVYg8
[0] https://google.github.io/flatbuffers/
[1] https://google.github.io/flatbuffers/flexbuffers.html
Flatbuffers destroys ProtoBuf both in terms of allocations(1 vs N where N is usually huge) and deserialization speed.
You also get explicit control over your data layout in memory when you write the message. So if you know what your read patterns are up-front you can get amazing cache usage even in managed languages.
One is based on json, one is modified thrift. The schemaless feature is among the many problems of json, but it's not the most annoying one. That would be binary support.
Imagine we need to make a upload/download storage service with json rpc...So as a rule of thumb, I would consider:
1. Web facing? json would do.
2. RPC? Pick protobuf or thrift or any protocol supporting native types including binary.