After gogoproto I'm hesitant to depend on another non-standard implementation, getting off gogo was a pain. This thing may be better than the one from Google (gogo definitely was), but can we be sure that it will still be around in 10 years?
Can you be sure Google’s ${anything} will be around in 6 months? They have a decades long habit of dumping things that have major use but aren’t novel internally. Worrying about the next 10 years isn’t something you can honestly do when you your argument includes Google owning a thing.
Bias: I fought for things like Reader from the inside. If it doesn’t move needles, it goes away.
This is incredible news! I’ve used protobuf in Python, Go, Kotlin and Dart and the Python implementation is totally unusable. I don’t know what black magic Google uses for the Python implementation, but the classes it generates are totally opaque and impossible to inspect. I’ve been waiting for a proper python implementation for years now!
When I was a novice in protobuf I felt tempted to inspect the generated code, but soon I learned that the documentation is good enough that I don’t need to read the generated code.
I wish LaunchDarkly and other feature flag providers supported protocol buffers to MN define the feature flag schema. It would be a game changer when you have complex variations and end up reaching for untyped JSON.
Engineer who works on Google Protobuf here, commenting as myself and not as an official statement.
It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable with any compatible thing that can understand Protobuf encoding including this one.
Kudos to Buf for making something that I'm sure a ton of people will find useful and which takes conformance so seriously.
Just to chime in with some context about Google's own implementations here though (since that's a lot of the discussion otherwise).
Google definitely takes Protobuf seriously including for the long term: you can't really understand how engrained it is within the Google stack without seeing it for yourself. It's not just RPC layer, it's storage, logging, FFI. Html templating is driven off Protobuf messages. Systems which interact with bank XML based systems uses Protobuf schemas. Internally it's widely used for in-memory library api types even without any direct/obvious connection to serialization just because it makes internal details like logging easier. This extremely large surface does create constraints and use-cases to balance. You can see Buf's reported numbers reflect that it is faster for a usecase they expect is typical, but at scale users do fall into the other buckets shown, affecting the performance of preexisting code is a major concern for our implementations that a greenfield implementation doesn't have.
Wide exposure in critical paths alongside long term support directly causes some quirks: for example some of our APIs followed PEP8 when it was created but PEP8 changed. It looks stupid that we have wrong style APIs but also it would be stupider to break compatibility for style reasons. JavaProto as another example still supports Java8 and the runtime is compatible with 2014 gencode which is a pretty major constraint.
Google Py Proto implementation has one extra interesting choice of the same gencode is reused with 3 different implementations (upb, a complete pure python one, and one that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++), which is why design the way that it is with runtime created classes, the pyi files are readable but the .py files not.
This definitely has pros and cons, and the direct approach taken by Buf here really makes a ton of sense. It's just that Google's maintained implementation falls into a different spot in a larger technical tradeoff space.
If you see things that appear to make no sense with the official implementations, feel free to file an issue on GitHub and we can look, sometimes there is no reason and we can fix it, and sometimes there's a reason which we can explain.
Kudos again to Buf here, I'm fully sure this will solve some set of real business needs better than Google's (but not because Google isn't maintaining our offerings too).
Slightly off topic, but is anyone aware of a compile free method to convert protobuf messages to json representation based on a provided .proto file (and the other way around)? All protobuf implementations seem to require a compilation step, which makes it hard to support en-/decoding untrusted content using user provided schemas
Buf.build, never again. Had a proto evangelist get us ingratiated into their system a few years ago and then they introduced a bunch of rent seeking behavior just to be able to to the code generation, and getting it ripped out was a huge PITA. https://github.com/betterproto/python-betterproto2 was a decent (async, type hinted) python implementation last I checked.
inexperienced protobuf user here -- why do non-googlers generally use protobuf? is it for the wire format, or for the schema language, or for the codegen?
i have a feeling there is an unmet need for something without all the google-cruft but still gives you a nice schema language and convenient codegen. just a simple stripped-down service and types definition with some support for codegen plugins would cover most crud-like applications in the wild without requiring much fuss. even just supporting only JSON would be acceptable -- maybe an optional buyin for msgpack.
i'm thinking twirp, but abandoning protobuf and using a new, simpler schema format.
These days I am of the opinion that we don’t need a real IDL. Instead one should choose a “primary language” and define the schema in that language. All other language bindings are generated from that canonical language. Then for wire format you’d just pick something workable from pb json msgpack etc. This replaces all the Google codegen intermediate layer, and for the primary language it naturally becomes very ergonomic.
Whats the case for protobuf these days? I loved it for a while. Don't dislike it particularly now (but I've rolled off).
- Performance? Fastest JSON lib in most languages are as fast if not faster
- Cross language schema generation? So many tools for this these days, they do one thing do them right (depending on your choice of 'right' for things like unions/enums/etc)
- The wire protocol? Seems to get in the way vs http2/3. Need special considerations for your proxies (be it nginx or cloudflare). Forced certs, etc are annoying too.
I feel like to most shops these days its mostly a schema manager? Protobuf is super bloated for that use case.
17 comments
[ 3.1 ms ] story [ 56.9 ms ] threadEdit: I see now. You are spamming links to your owm site.
Bias: I fought for things like Reader from the inside. If it doesn’t move needles, it goes away.
It's great to have a healthy ecosystem in the world around Protobuf. Google can't possibly fill all use cases, there's many tools and Buf makes good tooling. The Protobuf team at Google intentionally tries to enable an ecosystem around Protobuf including examples like this. Google Cloud APIs are intentionally usable with any compatible thing that can understand Protobuf encoding including this one.
Kudos to Buf for making something that I'm sure a ton of people will find useful and which takes conformance so seriously.
Just to chime in with some context about Google's own implementations here though (since that's a lot of the discussion otherwise).
Google definitely takes Protobuf seriously including for the long term: you can't really understand how engrained it is within the Google stack without seeing it for yourself. It's not just RPC layer, it's storage, logging, FFI. Html templating is driven off Protobuf messages. Systems which interact with bank XML based systems uses Protobuf schemas. Internally it's widely used for in-memory library api types even without any direct/obvious connection to serialization just because it makes internal details like logging easier. This extremely large surface does create constraints and use-cases to balance. You can see Buf's reported numbers reflect that it is faster for a usecase they expect is typical, but at scale users do fall into the other buckets shown, affecting the performance of preexisting code is a major concern for our implementations that a greenfield implementation doesn't have.
Wide exposure in critical paths alongside long term support directly causes some quirks: for example some of our APIs followed PEP8 when it was created but PEP8 changed. It looks stupid that we have wrong style APIs but also it would be stupider to break compatibility for style reasons. JavaProto as another example still supports Java8 and the runtime is compatible with 2014 gencode which is a pretty major constraint.
Google Py Proto implementation has one extra interesting choice of the same gencode is reused with 3 different implementations (upb, a complete pure python one, and one that uses C++Proto as the in memory representation which libraries like TensorFlow can use to share memory between Python and C++), which is why design the way that it is with runtime created classes, the pyi files are readable but the .py files not.
This definitely has pros and cons, and the direct approach taken by Buf here really makes a ton of sense. It's just that Google's maintained implementation falls into a different spot in a larger technical tradeoff space.
If you see things that appear to make no sense with the official implementations, feel free to file an issue on GitHub and we can look, sometimes there is no reason and we can fix it, and sometimes there's a reason which we can explain.
Kudos again to Buf here, I'm fully sure this will solve some set of real business needs better than Google's (but not because Google isn't maintaining our offerings too).
Why?
This is where I feel there are better alternatives. If you go with protobuf, you're picking 2 out of 3: simplicity, fast, idiomatic.
Please consider "uvx tsc-py --help" for things that don't fit.
i have a feeling there is an unmet need for something without all the google-cruft but still gives you a nice schema language and convenient codegen. just a simple stripped-down service and types definition with some support for codegen plugins would cover most crud-like applications in the wild without requiring much fuss. even just supporting only JSON would be acceptable -- maybe an optional buyin for msgpack.
i'm thinking twirp, but abandoning protobuf and using a new, simpler schema format.
- Performance? Fastest JSON lib in most languages are as fast if not faster
- Cross language schema generation? So many tools for this these days, they do one thing do them right (depending on your choice of 'right' for things like unions/enums/etc)
- The wire protocol? Seems to get in the way vs http2/3. Need special considerations for your proxies (be it nginx or cloudflare). Forced certs, etc are annoying too.
I feel like to most shops these days its mostly a schema manager? Protobuf is super bloated for that use case.