> Our evaluation of other solutions found poor client-side serialization performance, large run-time overhead, poor browser support, and different trade-offs that drove us to create Bebop.
I guess he's a bit like the programmer equivalent of the underground musician that's not that well known by the mainstream but who inspired a lot of other musicians. For example, he wrote the language that inspired Brainfuck[0].
As I evolve my schema (say add a new field and rename an old one) how is that handled? In my experience with GRPC+Protobuf schema changes over time and various service versions all being deployed this can become a real problem when one schema version starts talking to another.
I wish there was a half way house between something like this and JSON as I really find is useful to be able to debug over the wire with Postman or Charles for example.
Speaking of which, I’ve never seen a marshaling protocol with as sophisticated a message versioning scheme as ASN.1. The amount of thought that went into specifying how to allow “implementations from the future” to interoperate with those of the past is impressive.
And yet outclassed on any meaningful metric by generic serialization languages? ASN.1 is a programming language in addition to a data transport format opening up many opportunities for security vulnerabilities. Performance is shit because it’s using text to encode everything so that it’s human readable. Similarly parsing is extra complicated and slow for that reason. Serialization/deserialization requires a dedicated library rather than one that can be reused across all tasks. Finally the data representation itself is extremely bloated vs the binary data. It’s impressive in the same way that cassette technology and the neat things people did with them. I really wish some reasonable IDL would just get adopted for standards purposes rather than each standard developing their own.
All other serialization formats using an IDL seem strictly better.
> Performance is shit because it’s using text to encode everything so that it’s human readable. Similarly parsing is extra complicated and slow for that reason. [...] Finally the data representation itself is extremely bloated vs the binary data.
A weird criticism considering the encoding rules usually used for ASN.1 are all binary and some of them are bit-packed (like PER), which is very uncommon in newer protocols (for good reason).
Oh and there is OER now, which is actually a very reasonable binary encoding.
I've never done a proper read up on ASN.1 to know when and if to apply it. But i think its biggest obstacle apart from a flashy website is its wikipedia page (https://en.wikipedia.org/wiki/ASN.1). It is just overloaded with concepts that don't seem to be solving my problem, and its not clear what is a must-know and what is not.
I wrote an ASN.1 parser once (the platform we were using didn't have one)... and been wondering exactly why there are so many new binary formats when it all goes down to something just like ASN.1 :D, mostly (which is what, 30 years old now?).
This would be a fun blog post series: pick a couple of basic serialization/marshalling tasks, and a couple of common formats, and compare their performance characteristics, storage size and bandwidth requirements, implementation availability, etc.
The article doesn't mention at all how those speedups have been achieved, what tradeoff the format makes and what it is optimized for - which would be the actually interesting part.
A link to the benchmark code and description of the data would be nice.
It also doesn't show data for FlatBuffer, which is often a lot faster and leaner than ProtoBuf, or for Capt'n Proto.
ProtoBuf is not exactly known for amazing performance or very optimal client implementations across the various languages.
By all means, create a new serialization format, why not. But with so many options to chose from, I would require really strong justification internally.
Designing a serialisation format has a lot similarities with designing a programming language: data types, declarative/ease-of-use versus full control, abstractions, static versus dynamic memory areas, optimising representations etc. There is definitely room for a variety of formats depending on the use cases; and many are unhappy with the current incumbent: https://reasonablypolymorphic.com/blog/protos-are-wrong/
But yes, it's a lot of work and they probably should document what the trade-offs are that they have made.
It does seem more similar to Capt'n Proto than the things it does compare itself to.
I would guess the high speed is the triad of
length encoding with a header vs searching for delimiters
using what they call structs for benchmarks (no repeatedly sending the field name)
how much you trade off safety/sanity checks for performance
- The benchmark code is present in the laboratory directory of the repository.
- We don’t compare to Capt’n Proto because it does not have a stable web-based implementation, at least not one that has the features that make it so fast natively, so there is nothing to compare.
- Flatbuffers are fast but have a notoriously awful API to work with while also creating their own non-standard data structures in languages like C++. Bebop generates standard type-safe code.
- Bebop doesn’t try to compress data other than strings. This is because we don’t want to be responsible for compressing trailing zeroes when faster compression algorithms exist that can be down after encoding. Also most data is tiny.
- Bebop supports discriminated unions and has a much more robust type system than Flatbuffers.
- We’re not convincing anyone to use our stuff. It was made for us and open sourced because it was useful; we don’t need people ripping out their current serializers if there’s no pressure to do so.
Even with that I think flatbuffer, is still the main competition for something like this, not including it might make this look more impressive, but including it and mentioning that bebop has some advantages over flatbuffers (although with less language support) would be more fair.
They are impossible to benchmark against each other without making an assumption about how often you want to access the data, and which parts of it you want to access.
but this means Bebop and FlatBuffers can exist side-by-side / solve different problems: what FlatBuffers does makes sense if you want to access only parts of your objects in limited specific ways, what we do is better if you're always interested in the whole packet.
I only opened this article to see a comparison with FlatBuffers and got disappointed. :(
To be fair though, I am open to the idea of having a separate schema definition language, at least. (And please don't say DDL, it doesn't even come close.)
FlatBuffers creates its own demented and completely incompatible data structures when deserializing into C++. Which means you then need to copy the FlatBuffers structures into normal ones, defeating the entire point of "zero-copy" in the first place.
This thing it looks like uses normal C++ structures under the hood, and if so that's a huge plus.
I am not a JS dev, don't project your frustration on me. ;)
So okay, FlatBuffers doesn't map its zero-copy philosophy perfectly everywhere -- fact of life. What would you offer then? Which other format and/or library?
Some feedback - maybe I’m being picky but I feel you’ve chosen poorly with regard to the “readonly” keyword in your IDL. Or least I’d prefer if you used the normal type/interface naming conventions. “readonly” and “immutable” are different concepts although often confused. I’m pretty sure “immutable” better describes the behavior in your case.
Noob question:
How does performance of state of the art Binary formats compare versus state of the art Json serializers such as SIMDjson?
(both in throughput and file size)
Good binary formats with separate schema are always gonna be smaller and faster than JSON, because it doesn't have to encode field labels with every messages. JSON field names wastes so much space if your values are mostly numbers.
I "audited" the generated c++ code (looked at it for 10 seconds) and immediately noticed that the decode functions all create a copy of the input buffer..
59 comments
[ 3.3 ms ] story [ 129 ms ] thread* Arrow is columnar, batch-oriented, geared toward high throughput.
* Bebob is record-oriented, similar to Avro, Protobuf or JSON, geared toward low latency.
https://arrow.apache.org/docs/format/Columnar.html
Soo... cap'n proto?
http://strlen.com/
[0] http://strlen.com/false-language/
1: https://cs.stackexchange.com/questions/129904/does-there-exi...
I wish there was a half way house between something like this and JSON as I really find is useful to be able to debug over the wire with Postman or Charles for example.
All other serialization formats using an IDL seem strictly better.
A weird criticism considering the encoding rules usually used for ASN.1 are all binary and some of them are bit-packed (like PER), which is very uncommon in newer protocols (for good reason).
Oh and there is OER now, which is actually a very reasonable binary encoding.
What would be helpful is a concrete example showing what was tried with an existing approach that fell short. I mean code, benchmarks, theory.
I miss "tagged unions" or enums with values a.k.a. sumtypes.
A link to the benchmark code and description of the data would be nice.
It also doesn't show data for FlatBuffer, which is often a lot faster and leaner than ProtoBuf, or for Capt'n Proto.
ProtoBuf is not exactly known for amazing performance or very optimal client implementations across the various languages.
By all means, create a new serialization format, why not. But with so many options to chose from, I would require really strong justification internally.
But yes, it's a lot of work and they probably should document what the trade-offs are that they have made.
I would guess the high speed is the triad of
Oh, and keeping ints little endian.- The benchmark code is present in the laboratory directory of the repository.
- We don’t compare to Capt’n Proto because it does not have a stable web-based implementation, at least not one that has the features that make it so fast natively, so there is nothing to compare.
- Flatbuffers are fast but have a notoriously awful API to work with while also creating their own non-standard data structures in languages like C++. Bebop generates standard type-safe code.
- Bebop doesn’t try to compress data other than strings. This is because we don’t want to be responsible for compressing trailing zeroes when faster compression algorithms exist that can be down after encoding. Also most data is tiny.
- Bebop supports discriminated unions and has a much more robust type system than Flatbuffers.
- We’re not convincing anyone to use our stuff. It was made for us and open sourced because it was useful; we don’t need people ripping out their current serializers if there’s no pressure to do so.
To be fair though, I am open to the idea of having a separate schema definition language, at least. (And please don't say DDL, it doesn't even come close.)
This thing it looks like uses normal C++ structures under the hood, and if so that's a huge plus.
So okay, FlatBuffers doesn't map its zero-copy philosophy perfectly everywhere -- fact of life. What would you offer then? Which other format and/or library?