The benchmarks are in the repo + provided as screenshots in the README, but to clarify this is speaking to performance improvements relative to v2 of Bebop.
We don’t benchmark against FlatBuffers or Cap'n Proto for a few reasons:
- They are impossible to benchmark against each other without making an assumption about how often you want to access the data, and which parts of it you want to access. But this means Bebop, Cap'n Proto, FlatBuffers can exist side-by-side / solve different problems: what FlatBuffers and Cap'n Proto do makes sense if you want to access only parts of your objects in limited specific ways, what we do is better if you're always interested in the whole packet.
- We don’t benchmark against Capt’n Proto because it does not have a stable web-based implementation, at least not one that has the features that make it so fast natively, so there is nothing to compare.
Basically, Bebop is great for message oriented applications or where you need the entirety of your packet deserialized in a single step. So we can only benchmark against similar formats (JSON, Protobufs, MsgPack, etc.)
cap'n'proto is ancient, unmaintained, and its C# library is one of the most poorly written binary serialization implementations I have seen in my life.
You will be better served by gRPC and, if you need absolute maximum performance, MemoryPack (it is C#-first but can generate TypeScript clients).
If you need MQ, NATS is a good choice (special kudos to them adopting Yoshifumi Kawai's AlterNATS into NATS v2 C# client implementation).
If we’re talking specifically about C# binary serialization I’d recommend checking out this independent benchmark another developer did comparing Bebop to everything else
The comparison uses old and/or deprecated/EOL serializers which no one in their sane mind should use today. Consider referring to my previous comment, thanks!
(and by deprecated I mean deprecated 5+ years ago like in the case of Utf8Json)
Sadly a lot of videos discussing performance do only limited research and do not approach the matter comprehensively (likely not worth it for YT algorithm favoring quick and easy content).
Nothing wrong with offering your own first-party use cases that attempt to simulate realistic workloads on top of suggested patterns for each respective contestant, but it is important to include more performance-oriented frameworks/libraries (which is why I mentioned MemoryPack) that are used e.g. in gamedev sphere which cares a lot about realtime communication.
With all that said, it's actually awesome to see a framework that offers first-class C# support, so I probably need to shut up and appreciate the fact :)
The C++ implementation is well-maintained and heavily used by its own maintainers (e.g. me). Other implementations are maintained to varying degrees, since each implementation has a different maintainer. Agreed the C# implementation isn't maintained.
> You will be better served by gRPC
Well, this really depends, considering that Cap'n Proto's RPC system is quite different and far more expressive than gRPC.
Does Bebop support use as a file format by dumping directly to disk, as some serialization libraries do? Or perhaps a better way to phrase it, since ultimately you can send whatever you want to a binary file, is it intended to be used in such a way?
Very cool! I'd definitely be interested in that, especially if it could spit out class definitions for supported languages. Bebop seems to have native support for pretty much every language I care about too.
They were never particularly all that fast, especially not in the browser where support is far from acceptable. Most implementations rely on runtime magic, reflection, and other hacks.
So in that sense they aren’t all that modern as they’ve failed to evolve and keep up with platform evolutions.
Bebop just aims to generate code that modern and current with its target language, works everywhere, and avoids pitfalls protobuf suffers from.
I see. So far protobufs have met my needs and I've been happy with them, but I've used them more in an RPC context and less as a browser based data unit, so that makes sense.
Have you seen protobuf.js? I've dug quite deep into its generated code and have not seen any runtime magic. It's just directly working with the protobuf wire format and transferring the data to/from fields with very explicit code.
I can't speak for Google's official implementation though.
Not sure if it has improved in the last few years or if other target languages are better, but the protobuf code generation for C++ was pretty much the worst C++ code in terms of bloat I have ever seen. And I've seen a lot of bad C++ code ;)
In what ways did that meaningfully matter though? I use generated C++ protobuf code, and the API is clean even if the implementation looks like it wasn't hand written? If you think the C++ code is bad, you should see the Python... But again, the APIs are clean and robust so it hasn't affected me.
The "bloat" part is probably going to impact your compile times.
Also I tend to want generated code to be at least ish comprehensible in case I need to debug into it - from what you're saying, everything has gone sufficiently well in terms of 'clean and robust' that that hasn't been an issue for you in practice ... but justified or not, it's going to worry me anyway.
Out of curiosity, what tooling are you using around ASN.1 for language interoperability? I haven't found a "gold standard" that I'm happy with yet, personally.
Same, to be honest. I mainly use it with both Erlang and Java / Kotlin. While Erlang has excellent support for ASN.1, I have found that Java / Kotlin lacks adequate free tooling for working with it, which is quite surprising.
Sure they are. Cap'n Proto addresses a superset of Protobuf's use cases. Anywhere where you might use Protobuf, Cap'n Proto will work fine. In certain use cases (like mmaping large files or exporting complex APIs over RPC) Cap'n Proto is much better than Protobuf, but it's not bad at the other use cases.
The main limitation of Cap'n Proto compared to Protobuf is the ecosystem -- missing or poor-quality implementations in many languages, limited tooling, etc. Admittedly this is probably a showstopper for most users. It's also the hardest thing for any new contender to solve.
With all that said I would tend to agree that benchmarks are probably pointless. I've spent a lot of time benchmarking serialization and one thing I know is that benchmark results will vary wildly depending on the use case. A benchmark of an example/toy use case isn't really indicative of performance in a real use case.
(I'm the author of Cap'n Proto. I don't know much about Flatbuffers so can't comment there.)
Apologies if my initial comment came off the wrong way; when I say “not alternatives” I was injecting bias into that statement since I normally use Cap’n Proto on embedded and memory constrained systems (amazing library btw.)
When I created Bebop years ago I tried to benchmark it against Cap’n Proto, but the lack of a semi-decent web or C# implementation mentally made me recategorize.
So I suppose I’m just comparing what’s closest in terms of ecosystem support rather than a purely functional level.
Yeah that's fair, it does seem like Bebop and capnp are targeted at different use cases, even if they're both Protobuf alternatives.
It obviously doesn't make sense to benchmark a JavaScript Bebop implementation against a C++ Cap'n Proto implementation, nor does it make sense to compare against a JS Cap'n Proto implementation written by a third-party contributor that is incomplete and unmaintained... so there isn't really a sensible comparison to make.
I'm not sure how you can say they aren't alternatives, they directly market themselves as alternatives. The first graphic on the capnproto site compares it to protobufs and was itself created by one of the original protobuf authors as I understand it (Kenton).
Flatbuffers immediately compares itself to protobufs as well on the front page.
What kind of compatibility guarantees does this make? That's one of the key benefits of protos: I can update clients to add new fields and they can still talk to servers with previous versions of the schema.
you get forward and backwards compatibility out of the box so long as you adhere to the schema guidelines.
The docs go into more detail on how different types are handled. We also have some special features like “binary schema” which let you ship your schema with data so clients are always up to date.
(it may be less of an issue today since the DPGO will devirtualize those, when profiling average rpc framework, more time is spent on 1. deep async call stacks, 2. inefficient serde that copies data frequently and heap allocates transient objects and 3. working against the language by not using its features made for performance (Memory/Span slices, using manually rolled copy and search routines over span.CopyTo, .IndexOf))
And overall observation is the less code and the simpler it is, the better, notably gRPC suffers from its pipeline/interceptors-support style implementations and insisting on usings its own types replacing T[] and List<T>, putting irreducible mandatory cost in places where it may matter most (unless you partially bypass those with reflection/unsafe accessor like I had to do once))
Never understand the hype for protocol buffers, it may works well for languages like java and kotlin, but is a mess on many other languages.
And then there are faster and better replacements.
And all the time I use JSON, there don't exists multiple versions and any JSON file that is somewhere can get read by almost every language. (And I can read the file even with every editor)
I’ve been hoping to see someone eventually build a protobuf style serialisation solution on some of the emerging IETF standards like CBOR and it’s corresponding data definition language.
I’m not really in the market for another proprietary binary data format that doesn’t integrate with anything outside its immediate ecosystem.
42 comments
[ 5.4 ms ] story [ 102 ms ] threadSorry, but I would like some benchmark data to these claims....
- They are impossible to benchmark against each other without making an assumption about how often you want to access the data, and which parts of it you want to access. But this means Bebop, Cap'n Proto, FlatBuffers can exist side-by-side / solve different problems: what FlatBuffers and Cap'n Proto do makes sense if you want to access only parts of your objects in limited specific ways, what we do is better if you're always interested in the whole packet.
- We don’t benchmark against Capt’n Proto because it does not have a stable web-based implementation, at least not one that has the features that make it so fast natively, so there is nothing to compare.
Basically, Bebop is great for message oriented applications or where you need the entirety of your packet deserialized in a single step. So we can only benchmark against similar formats (JSON, Protobufs, MsgPack, etc.)
You will be better served by gRPC and, if you need absolute maximum performance, MemoryPack (it is C#-first but can generate TypeScript clients).
If you need MQ, NATS is a good choice (special kudos to them adopting Yoshifumi Kawai's AlterNATS into NATS v2 C# client implementation).
https://youtu.be/XMoNYQPi2k8?feature=shared
(and by deprecated I mean deprecated 5+ years ago like in the case of Utf8Json)
Sadly a lot of videos discussing performance do only limited research and do not approach the matter comprehensively (likely not worth it for YT algorithm favoring quick and easy content).
Nothing wrong with offering your own first-party use cases that attempt to simulate realistic workloads on top of suggested patterns for each respective contestant, but it is important to include more performance-oriented frameworks/libraries (which is why I mentioned MemoryPack) that are used e.g. in gamedev sphere which cares a lot about realtime communication.
With all that said, it's actually awesome to see a framework that offers first-class C# support, so I probably need to shut up and appreciate the fact :)
I mean... not compared to protobuf.
> unmaintained,
The C++ implementation is well-maintained and heavily used by its own maintainers (e.g. me). Other implementations are maintained to varying degrees, since each implementation has a different maintainer. Agreed the C# implementation isn't maintained.
> You will be better served by gRPC
Well, this really depends, considering that Cap'n Proto's RPC system is quite different and far more expressive than gRPC.
I’m also working on a textual configuration language that allowa you to import a schema and define data like JSON.
So in that sense they aren’t all that modern as they’ve failed to evolve and keep up with platform evolutions.
Bebop just aims to generate code that modern and current with its target language, works everywhere, and avoids pitfalls protobuf suffers from.
Even though our schema was huge, it still did serialisation in a time that was surprisingly small for our underpowered chip.
I can't speak for Google's official implementation though.
Also I tend to want generated code to be at least ish comprehensible in case I need to debug into it - from what you're saying, everything has gone sufficiently well in terms of 'clean and robust' that that hasn't been an issue for you in practice ... but justified or not, it's going to worry me anyway.
Which programming languages do you have in mind?
They aren’t relevant to what Bebop is doing and thus no benchmark or comparison can be made between them.
The main limitation of Cap'n Proto compared to Protobuf is the ecosystem -- missing or poor-quality implementations in many languages, limited tooling, etc. Admittedly this is probably a showstopper for most users. It's also the hardest thing for any new contender to solve.
With all that said I would tend to agree that benchmarks are probably pointless. I've spent a lot of time benchmarking serialization and one thing I know is that benchmark results will vary wildly depending on the use case. A benchmark of an example/toy use case isn't really indicative of performance in a real use case.
(I'm the author of Cap'n Proto. I don't know much about Flatbuffers so can't comment there.)
When I created Bebop years ago I tried to benchmark it against Cap’n Proto, but the lack of a semi-decent web or C# implementation mentally made me recategorize.
So I suppose I’m just comparing what’s closest in terms of ecosystem support rather than a purely functional level.
It obviously doesn't make sense to benchmark a JavaScript Bebop implementation against a C++ Cap'n Proto implementation, nor does it make sense to compare against a JS Cap'n Proto implementation written by a third-party contributor that is incomplete and unmaintained... so there isn't really a sensible comparison to make.
Flatbuffers immediately compares itself to protobufs as well on the front page.
The docs go into more detail on how different types are handled. We also have some special features like “binary schema” which let you ship your schema with data so clients are always up to date.
https://docs.icerpc.dev/slice2
And I can tell the C# code it generates is going to be plagued by vtable lookups. Some decent ideas though.
And overall observation is the less code and the simpler it is, the better, notably gRPC suffers from its pipeline/interceptors-support style implementations and insisting on usings its own types replacing T[] and List<T>, putting irreducible mandatory cost in places where it may matter most (unless you partially bypass those with reflection/unsafe accessor like I had to do once))
It's nice to finally see some quality alternatives to Protobuf.
One idea I like from Slice are custom types. Is there anything similar in Bebop?
https://docs.bebop.sh/tempo/
Update: Coming soon!
https://twitter.com/andrewmd5/status/1767612314730336353
And all the time I use JSON, there don't exists multiple versions and any JSON file that is somewhere can get read by almost every language. (And I can read the file even with every editor)
I’m not really in the market for another proprietary binary data format that doesn’t integrate with anything outside its immediate ecosystem.