HN Explorer

Ask HN: Protobuf as a Database Wire Protocol?

1 points by fearthetelomere ↗ HN
Why is Protobuf a bad choice for a distributed database wire protocol?

I understand it's a popular choice for service development through the use gRPC, Twirp, etc. I assume there are good performance reasons why Protobuf is not used in database node-to-node communication, but I'm curious to know what they are.

As an aside, if I don't want to commit to one protocol like Postgres or MySQL or Memcached, is there an extensible wire protocol out there that can accommodate these different protocols under one umbrella? I'm considering generically pre-pending each frame with a set of bits to identify which protocol the frame uses. This seems like something that's probably already been solved, and I'm just not aware of it. Does anyone have any ideas?

3 comments

[ 3.2 ms ] story [ 16.6 ms ] thread
aq9
So, a lot would depend on what you plan on passing the protobufs over. Plain TCP? gRPC? gRPC kinda sucks for this, the serialization/deserialization can be moderately expensive (CPU); and gRPC's multiplexing of multiple flows onto the same connection can be hard to reason about (i.e. debug); and the performance characteristics are not great either.
fearthetelomere
I didn't consider that gRPC ser/deser was somewhat expensive. Then that would also be true for the TCP case too, right?

Assuming I'm sending data over plain TCP, would Protobuf be comparable in performance to another DB wire protocol? Or am I better off selecting a proven protocol like Postgres and using that?