schema-driven development is great, but I wish people spent more time on JSON-based API schema rather than switch to protobuf. Because protobufs immediately require some minimal level of complexity (at least a build system!) and thus effectively block small one-off scripts. And the debuggability is also not great.
Can we (as an industry) pick a standard (openapi? jsonschema? something else?) and add the tooling to match protobuf ecosystem functionality?
I don't think a large group can pick a standard. What works well for one programming language does not work for another, and we devolve into tribes. Protobuf gained popularity when it dropped features to better support Go more naturally (no more null/nil for example), and rode in on a wave of Go adoption. And made things worse from a Python developer perspective. Pretty much the same as toml adoption, which fits more naturally with Go and more static languages, unlike the much more dynamic and flexible yaml.
> I don't think a large group can pick a standard.
There are plenty of cases where this has happened. JSON has already become the defacto standard. It's not a the step to formalize a ubiquitous format has been taken many times. Every major platform supports it or uses it exclusively. Some frameworks and tooling, only uses JSON (true of other formats to an extent, ie json more than yaml).
> What works well for one programming language does not work for another,
I'm not sure why this roundabout whataboutism is compelling. A standard protocol is used in absence of other restrictions or benefits of another This doesn't snap away the other protocols from existence.
TOML doesn't "fit static languages" better and is just as dynamically typed as YAML. It's used extensively in the Python ecosystem, which has TOML in stdlib (not YAML).
I'm less sure about protobufs in Python (I never tried), but in general relying too much on Python's dynamic nature is not a good thing IMHO.
GRPC <=> Rest transcoding works fine, you can do this with envoy for example.
And if you use the connect protocol (also from buf) then you can also use json.
So it's quite achievable to get all the benefits you want from the gRPC ecosystem for developing your service while also allowing quick and dirty client usage (i.e. scripting with cURL).
I fail to see how running something like protoc before running or compiling code is a show stopper. There are even npm packages that will install protoc for you [1]. That means you can use it in a custom npm build script action, or invoke it before staring the dev server? Put in a git pre-commit to run it to assert there are no changes? Run via a file-watcher?
IMO writing a schema in something like JSON for OpenAPI is way more verbose and time consuming than writing a proto file and a script or Makefile to invoke protoc. There also is the mismatch in capabilities. The JS-adjacent schema standards mostly do not directly support native types like uint64, let alone directly specify how to encode numbers like fixed64. I mean, look... the work around in JavaScript still is use "string" [2].
On the other hand, there is a standard in OpenAPI for validating values. But that doesn't make sense in the Protobuf world since usually the API is expected to be forward and backward compatible, within reason, so something like a min and max value needs to be done at runtime, and not fixed within the schema. And don't get me started on trying to define cross-field validation in a declarative schema. It is easier to just write the code. :-)
So, IMO, no, a single standard won't happen in the industry. The many dimensions like logical vs physical models, validation, or error handling that make it impossible.
Protobuf has a JSON canonicalization [0]. All our gRPC servers automatically accept HTTP/JSON traffic with the procedure name specified in a header. That's what I use when I'm writing a script.
If only JSON-RPC[1] were more popular for JSON-based APIs. With JSON-RPC specifically, the trade-off is that it would not work well with binary data; but if that's a problem then instead of JSON it could be BSON, bencode, EBML, or whatever (at that point it wouldn't be JSON-RPC though), the point is that you can nest stuff without switching syntax.
But compared to HTTP, if what you want is just JSON, then JSON-RPC saves you from having 5 different places to pass data, each with its own special syntax (one for method name; one for path; one for query params in the path; one for headers; one for body with the actual data).
We only not notice this because of just how many libraries are doing the hard work for us and making it seem like HTTP requests are clean and nice stuff; but no, HTTP is certainly not a nice protocol. Its main advantage is its ubiquity.
People may ask, "what about caching and stuff?". Well, with HTTP we made that work by passing a special key in the part of the request that is delimited by CRLF and where each line contains a key-value pair separated by a colon-and-space; and the clients know that a response was cached because they can read the first line of the response, which is a space-separated list of keywords that has a specific numeric literal in its second position (yes, I'm oversimplifying).
So I'm pretty sure we can do something better with fewer different syntaxes. And reverse proxies would presumably have it harder to be vulnerable to things like request smuggling[2].
> pick a standard (openapi? jsonschema? something else?)
This is a false dichotomy, as OpenAPI and JSON Schema are complementary: as of version 3.1, OpenAPI use JSON Schema to describe request and response payload content (prior to that version it was almost compatible).
They describe different things: OpenAPI describes the request-response API model, while JSON Schema can be used to specify contents of any JSON-encoded data -- for example in streaming messages or document databases.
There is also AsyncAPI, which is designed to document and specify the event-based API, for example in a system using Apache Kafka or a similar queue to communicate.
While certainly not as ubiquitous as Protobuf, Cap'n Proto[1] is an extremely excellent alternative worth checking out. The quality of the generated C++ code is enough reason for me to prefer it.
Does protobuf or capnproto have good tooling support out in the world?
Proto was very convenient at Google since everything was proto, your debug tools were proto, all your data pipeline tools spoke proto, you had command line utilities for protos, etc, etc.
It seems like the rest of the world is on JSON unless they actually need performance, so if you don't get performance, should you use something like a code generator for JSON types like Conjure so that you don't actually have to deal with these binary formats everywhere?
Would avro make more sense in a big data world? As I understand it, avro has all the same capabilities of protobuff but it is much more ubiquotous in the big data world, being natively supported by all the big frameworks (spark, flink)
Protobuf is great, but wait till you hear about JSON!
Jokes aside, while JSON is bulkier, binary protocols like protobuf, avro, thrift have one big penalty: they are hard to read/understand by humans without some tooling. The value of seeing the network call and being able to actually understand what's going on without having to translate the format into something human readable is hard to overstate.
The bigger payload size of JSON is not enough for me not to see the main benefit : I can check what's going on with any tool: from curl, to postman, to my browser's damn network window. With some compression, JSON is not that bad size wise either.
To read and generate protobuf, you mostly have to rely on some tooling/library provided by Google, and those, like most of their software (looking at you, Guava), is the reverse of backwards compatible, despite what they're advertising. Just search Google for "protobuf version mismatch".
Maybe I am speaking heresy here, but software by Google (like k8s, protobuf, guava, etc) for me has been notoriously backwards incompatible and overly complex, and I try to steer clear from it as much as possible. I gotta give the crown to K8s here, it specifically is an absolute clusterfuck, the pinnacle of what happens when 1000s of good ideas are combined together in a sloppy, quick execution and without regard for previous versions.
You can achieve schema with any other binary format like avro (which doesn't tie you to using google libraries, which i hate with a passion), and you can achieve a schema with xml and json, so after reading the article, there is still 0 reason for me to use protobuf, if not for performance and only within a google ecosystem.
For structured binary data, protobuf is, of course, probably a better fit than JSON.
However, I don't think structured BINARY data is encountered as often on the interwebs as JUST structured data, and simpler textual formats like JSON, XML (and even csv) shine here, despite their shortcomings in terms of message size.
As for unstructured (or loosely structured) binary data like videos or large binaries (binary blobs is what I am talkin about), protobuf is as much as a poor choice as any other structured/schema based data protocols.
Protobuf has no tremendous benefit to any of the other binary formats like avro either, so I don't see why I should be excited about it. I wish google and the google ecosystem people stop trying to make it sound like THEIR simple schema based binary protocol is a big deal and try to shove it down our throats. For simple message passing around, I intend to keep choosing compressed JSON for the foreseeable future, regardless of what narrative some fan article is peddling.
22 comments
[ 3.2 ms ] story [ 63.0 ms ] threadCan we (as an industry) pick a standard (openapi? jsonschema? something else?) and add the tooling to match protobuf ecosystem functionality?
There are plenty of cases where this has happened. JSON has already become the defacto standard. It's not a the step to formalize a ubiquitous format has been taken many times. Every major platform supports it or uses it exclusively. Some frameworks and tooling, only uses JSON (true of other formats to an extent, ie json more than yaml).
> What works well for one programming language does not work for another,
I'm not sure why this roundabout whataboutism is compelling. A standard protocol is used in absence of other restrictions or benefits of another This doesn't snap away the other protocols from existence.
I'm less sure about protobufs in Python (I never tried), but in general relying too much on Python's dynamic nature is not a good thing IMHO.
And if you use the connect protocol (also from buf) then you can also use json.
So it's quite achievable to get all the benefits you want from the gRPC ecosystem for developing your service while also allowing quick and dirty client usage (i.e. scripting with cURL).
IMO writing a schema in something like JSON for OpenAPI is way more verbose and time consuming than writing a proto file and a script or Makefile to invoke protoc. There also is the mismatch in capabilities. The JS-adjacent schema standards mostly do not directly support native types like uint64, let alone directly specify how to encode numbers like fixed64. I mean, look... the work around in JavaScript still is use "string" [2].
On the other hand, there is a standard in OpenAPI for validating values. But that doesn't make sense in the Protobuf world since usually the API is expected to be forward and backward compatible, within reason, so something like a min and max value needs to be done at runtime, and not fixed within the schema. And don't get me started on trying to define cross-field validation in a declarative schema. It is easier to just write the code. :-)
So, IMO, no, a single standard won't happen in the industry. The many dimensions like logical vs physical models, validation, or error handling that make it impossible.
[1] https://www.npmjs.com/package/@protobuf-ts/protoc [2] https://github.com/OAI/OpenAPI-Specification/issues/2617#iss...
https://protobuf.dev/programming-guides/proto3/#json
It's very handy in some situations.
Can it be done via http (ex. Postman) or it needs http2?
But compared to HTTP, if what you want is just JSON, then JSON-RPC saves you from having 5 different places to pass data, each with its own special syntax (one for method name; one for path; one for query params in the path; one for headers; one for body with the actual data).
We only not notice this because of just how many libraries are doing the hard work for us and making it seem like HTTP requests are clean and nice stuff; but no, HTTP is certainly not a nice protocol. Its main advantage is its ubiquity.
People may ask, "what about caching and stuff?". Well, with HTTP we made that work by passing a special key in the part of the request that is delimited by CRLF and where each line contains a key-value pair separated by a colon-and-space; and the clients know that a response was cached because they can read the first line of the response, which is a space-separated list of keywords that has a specific numeric literal in its second position (yes, I'm oversimplifying).
So I'm pretty sure we can do something better with fewer different syntaxes. And reverse proxies would presumably have it harder to be vulnerable to things like request smuggling[2].
[1]: https://en.wikipedia.org/wiki/JSON-RPC [2]: https://en.wikipedia.org/wiki/HTTP_request_smuggling
This is a false dichotomy, as OpenAPI and JSON Schema are complementary: as of version 3.1, OpenAPI use JSON Schema to describe request and response payload content (prior to that version it was almost compatible).
They describe different things: OpenAPI describes the request-response API model, while JSON Schema can be used to specify contents of any JSON-encoded data -- for example in streaming messages or document databases.
There is also AsyncAPI, which is designed to document and specify the event-based API, for example in a system using Apache Kafka or a similar queue to communicate.
[1] https://capnproto.org/
Proto was very convenient at Google since everything was proto, your debug tools were proto, all your data pipeline tools spoke proto, you had command line utilities for protos, etc, etc.
It seems like the rest of the world is on JSON unless they actually need performance, so if you don't get performance, should you use something like a code generator for JSON types like Conjure so that you don't actually have to deal with these binary formats everywhere?
Buf (author of the article) has made it their mission to build out that tooling and support for Protobuf. I think they're doing a good job.
Cap'n Proto's ecosystem is admittedly weak, since it as yet has no full-time team trying to build this out.
(Disclosure: I'm the author of Cap'n Proto and, long ago, Protobuf v2. I'm also a small investor in Buf.)
Jokes aside, while JSON is bulkier, binary protocols like protobuf, avro, thrift have one big penalty: they are hard to read/understand by humans without some tooling. The value of seeing the network call and being able to actually understand what's going on without having to translate the format into something human readable is hard to overstate.
The bigger payload size of JSON is not enough for me not to see the main benefit : I can check what's going on with any tool: from curl, to postman, to my browser's damn network window. With some compression, JSON is not that bad size wise either.
To read and generate protobuf, you mostly have to rely on some tooling/library provided by Google, and those, like most of their software (looking at you, Guava), is the reverse of backwards compatible, despite what they're advertising. Just search Google for "protobuf version mismatch".
Maybe I am speaking heresy here, but software by Google (like k8s, protobuf, guava, etc) for me has been notoriously backwards incompatible and overly complex, and I try to steer clear from it as much as possible. I gotta give the crown to K8s here, it specifically is an absolute clusterfuck, the pinnacle of what happens when 1000s of good ideas are combined together in a sloppy, quick execution and without regard for previous versions.
You can achieve schema with any other binary format like avro (which doesn't tie you to using google libraries, which i hate with a passion), and you can achieve a schema with xml and json, so after reading the article, there is still 0 reason for me to use protobuf, if not for performance and only within a google ecosystem.
However, I don't think structured BINARY data is encountered as often on the interwebs as JUST structured data, and simpler textual formats like JSON, XML (and even csv) shine here, despite their shortcomings in terms of message size.
As for unstructured (or loosely structured) binary data like videos or large binaries (binary blobs is what I am talkin about), protobuf is as much as a poor choice as any other structured/schema based data protocols.
Protobuf has no tremendous benefit to any of the other binary formats like avro either, so I don't see why I should be excited about it. I wish google and the google ecosystem people stop trying to make it sound like THEIR simple schema based binary protocol is a big deal and try to shove it down our throats. For simple message passing around, I intend to keep choosing compressed JSON for the foreseeable future, regardless of what narrative some fan article is peddling.