Also, cap'n proto loses a whole lot of API and semantic weirdness the author complains about: implicit auto-assignment of fields, "repeated" etc. The overall design feels much cleaner, although there are still some surprises.
I've heard good things about https://capnproto.org/ but the author of this piece may strongly disagree, since I think their objections are rather more fundamental, and also because Cap'n Proto went with banning the concept of "required" fields.
I'd also be interested in what alternatives the author might suggest.
Capnproto is built by the guy who originally built protobufs, and he cites capnproto as being better because he was able to learn from his mistakes. Makes me optimistic.
Minor clarification of the clarification: the author of Cap'n'proto was the original creator of Proto2, which was a ground-up (but binary compatible) rewrite of Proto1. The original authors of Proto1 were Jeff Dean & Sanjay Ghemawat. The current version of protobufs is Proto3, which AIUI is maintained by a team at Google (it was released after I left), and is an evolution of the Proto2 codebase with many new features.
You should definitely also look at Cap'n Proto – it is in many ways a theoretically more elegant design: far less overhead and the "time-traveling" RPC design is quite clever. Another potential important advantage, esp for crypto could be that Cap'n Proto in theory has a canonical representation, but in practice this was neither fully defined nor usable last I looked.
Cap'n Proto also suffers from a number of practical disadvantages, which can be quite severe depending on your use case. The main problem is eco system maturity and rate of adoption. So whilst protobufs have excellent cross-language support and a mature RPC system, in cap'n proto everything but C++ feels at bit second class, even other popular languages. Python and Rust, and I believe Java have workable, but not necessarily great bindings for serialization and in theory RPC, but the RPC support for other languages than C++ didn't look that great last I looked (but that was over a year ago) and even with C++ it looks much easier to get something going with gRPC (tooling, http2 based etc). For other languages I looked at (e.g. Ocaml, Lua and Haskell) the bindings looked very immature. There are a few other minor annoyance with capnproto, such as the lack of a timestamp type (you can build your own, incompatible to everyone else's, but this is still an annoying omission). The C++ library is also saddled with the weird home-grown kj library that no one other than Kenton Varda uses. However, it has to be said that the C++ API and also the Python API for Capnproto feel a lot more natural and idiomatic than the protobuf ones. If protobuf is too high overhead for your use case and you don't need the wide language and eco system support, Cap'n Proto is worth a serious look.
JSON or graphql. Personally I think the typing, API compatibility and extensive language support make protobuffers currently the winner though, despite the warts
I'm no GraphQL expert/fan but one example is requesting only the data you need -- you can cut down on what you have to send over the wire which is at least guaranteed to help some.
Also, the nested query structure can save what would have normally been multiple requests with a by-the-book RESTful HATEOAS setup.
Neither of those are binary formats though, and if I’m picking up a binary format like Protobuffers it’s probably because I want the performance benefits.
Personally, these days I’m taking the approach of just outright avoiding pretty much anything google does. Either because it solves a problem only they have, or they’ll probably just drop it out of the blue one day. Or like Tensorflow, it’s fucking impossible to get your head around because their documentation seems to be written with the goal of being maximally confusing.
I like GraphQL schema, but it would have to have a standardized serialization aspect. Moreover, it really is designed for querying and has a bunch of stuff totally irrelevant to data.
But I agree ... I almost wish they'd finish it off and go for full on serialization.
That said, it may not be that-that hard, a serialization spec can be pretty short if need be. Certainly one could 'roll one's own' for a mid-sized project if need be.
Thrift suffers pretty much all the same problems as protobufs and has many similar dysfunctional failure modes, like places that rely on serializing into Thrift structs stores in hdfs and treating that like a de facto database, with Thrift struct definitions as the schema. It is so miserable to work in code bases like that.
> Despite map fields being able to be parameterized, no user-defined types can be. This means you'll be stuck hand-rolling your own specializations of common data structures.
What a pain! Well, at least Google won't make that mistake again!
My contention with the quoted text is that you probably shouldn't be using elaborate data structures in streams/files. Using DTOs as heap/stack data has bitten me enough times that I'm fairly certain that it's an anti-pattern.
It doesn't matter if you're using a quantum binomial tree in a black hole: save it as a 'stupid' map<k,v> when it hits the network. That way everyone who interacts with your service can decide how they want to represent that structure. You can compose any in-memory data structure you can dream of, you can validate the data with more richness than "missing" or "present", and Protobuf doesn't contaminate your codebase.
Option 3 is the correct choice. Serialization libraries should be amateurish by design.
> That way everyone who interacts with your service can decide how they want to represent that structure.
Protobufs aren't a message format for publicly specced standard wire protocols. They're a serialization layer for polyglot RPC request and response messages. The whole point of them is that you're transporting the same typed data around between different languages, rather than there having to be two conversions (source language -> "standard format"; "standard format" -> dest language).
In this sense, they're a lot like, say, Erlang's External Term Format—driven entirely by the needs of an RPC-oriented distributed-computation wire protocol, doing interchange between nodes that aren't necessarily running the same version of the same software. Except, unlike ETF, Protobufs can be decoded without the backing of a garbage-collected language runtime (e.g. in C++ nodes.)
I'm not saying Protobufs are the best, but you have to understand the problem they're solving—the "rivals" to Protobufs are Cap'n Proto and Thrift, not JSON. JSON can't even be used to do a lot of the stuff Protobufs do (like talk to memory-constrained embedded kernels running on SDN infra.)
> and Protobuf doesn't contaminate your codebase
Just like in HTTP web-app codebases, a codebase properly designed to interact with an https://en.wikipedia.org/wiki/Enterprise_service_bus will tend toward Hexagonal architecture—you have your own logic, and then you have an "RPC gateway" that acts as an MVC system, exposing controllers that decode from the RPC format and forward requests into your business logic; and then build responses back into the RPC "view" format.
Once you have that, there's no point in having your application's parsed-out representation of the RPC format be different than the on-the-wire RPC format. The only thing that's touching those RPC messages is your RPC gateway's controller code anyway. So why not touch them with as little up-front design required as possible?
In the context of gRPC, it's true that Protobuf is intended to be a wire protocol, but the argument would be more convincing if the code generator toolchain Google fostered created code that integrated better with the languages that people use.
Typically, the data types and interfaces generated by these tools are so poor that you need to build another layer on top that translates between "Protobuf types" and "native types", as you describe in your comment, and shields the application from the nitty-gritty details of gRPC. So investing in gRPC means that when you've generated, say, Go files from your .proto files, you're only halfway done. Protobuf in itself introduces a kind of impedance mismatch, a kind of stupid in-bred cousin whose limited vocabulary has to be translated back and forth into proper language.
So gRPC/Protobuf solves something important at the wire level, but developers really want to productively communicate via APIs, and so what you have is just half the solution.
I wish the Protobuf/gRPC toolchain were organized in such a way that the generated code could actually be used as first-class code. Maybe similar to how parser generators like Lex/YACC or Ragel work, where you provide implementation code that is threaded through the output code.
Wow, some minor complaints on language formality issues, then discredit the thing as a whole.
But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential.
It's like complaining a car that drives fast and has great gas efficiency of cannot talk. (Well in 2018, it might be less of a stretch to demand a car that can talk)
> But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential.
Huh? It's a serialization format. This problem has been solved many times in many different ways.
The author is pointing out that Protobufs were designed poorly, and they didn't have to be.
Versioning was the biggest disappointment for me. I just want a middleware layer that can handle clients of different versions reliably. Surely everyone has the same problem.
Mostly lost me at "Make all fields in a message required. This makes messages product types."
Your constraints on protocols can change over time and context, changing something that is required to something that is optional can cause crashes in prod. Unless somehow the 'optional' design has a way to handle that? You also will have to do custom validation anyways, as your storage format will never be able to enforce all constraints you care about.
Also, it would seem like the 'debate' of optional vs required is harmful was resolved, as proto3 makes everything optional.
In context of current protobuf design required was a mistake and optional is clearly better, but author argues about grand type system that is based on different principles, including stronger validation.
Criticizing protobufs is like criticizing C++ or Java. Both have major shortcomings, but solve practical problems and de facto lingua franca with no practical solutions to replace them.
>Fields with scalar types are always present. Even if you don't set them. Did I mention that (at least in proto3) all protobuffers can be zero-initialized with absolutely no data in them?
I don't this complaint. You have to initialize data with something. proto2 had to "solve" this problem in C/C++/Go by making everything a pointer. How is dealing with null the "sane" case against zero-initialization?
Scalar primitive fields in proto1/proto2 C++ generated code are not pointers. They are just class member fields returned by accessors (foo()) and their presence is indicated by a presence vector (has_foo()). At least that's what you get if you use Google's proto compiler.
The class layout for proto3 is the same (surprise!) but the presence vector is ignored.
To be fair, in C++ you could use std::optional or something else that's type-safe.
Go has its own philosophy about zero values which is controversial, but at least with pointers you do get to express optional values. And it's not like you can't work around the ergonomic awkwardness that arises:
type Post struct {
Title *string
}
func (p *Post) GetTitle() (string, bool) {
if p.Title == nil {
return "", false
}
return *p.Title, true
}
func (p *Post) SetTitle(s string) {
p.Title = &s
}
Hardly elegant, but this is Go.
Go also run into the same issue when encoding and decoding JSON. Most languages do distinguish between empty string and a missing string, but not Go, which makes it hard to validate anything. I wrote a code generator for JSON Schema [1] recently, which applies validations while deserializing, and has to resort to a rather low-tech trick to do so:
type Post struct {
Title string `json:"title"`
}
func (v *Post) UnmarshalJSON(b []byte) error {
var raw map[string]interface{}{
if err := json.Unmarshal(b, &raw); err != nil {
return err
}
if _, ok := raw["title"]; !ok {
return errors.New("field title: must be set")
}
type plain Post
var p plain
if err := json.Unmarshal(b, &p); err != nil {
return err
}
*v = Post(p)
return nil
}
(Clearly there are slightly more refined and faster ways to do this, but not that easily without importing third-party code, which I wanted to avoid.)
C and Go are in the minority here. This doesn't come up in languages like Rust, Swift, TypeScript, Nim, Haskell, OCaml, C#, F#, or Java >= 8, all of which have some sort of optional support (algebraic data types or built-in). For example, TypeScript:
so, just for fun, here's a way to do that check without double unmarshalling and allocating maps and such for everything under your struct (also does a check for extra fields, but you could pull that out if you want):
type Post struct {
Title string `json:"title"`
}
func (v *Post) UnmarshalJSON(b []byte) error {
dec := json.NewDecoder(bytes.NewReader(b))
required := map[string]struct{}{
"title": struct{}{},
}
tok, err := dec.Token()
if err != nil {
return err
}
if d, ok := tok.(json.Delim); !ok || d != '{' {
return errors.New("Expected object")
}
for {
tok, err := dec.Token()
if err != nil {
return err
}
if d, ok := tok.(json.Delim); ok && d == '}' {
break
}
switch tok {
case "title":
delete(required, "title")
err := dec.Decode(&v.Title)
if err != nil {
return err
}
default:
(*v) = Post{}
return errors.New(fmt.Sprintf("Unexpected field %s", tok))
}
}
if len(required) > 0 {
(*v) = Post{}
return errors.New(fmt.Sprintf("Missing %v required fields", len(required)))
}
return nil
}
Thanks, that's neat! The challenge here is that I also need to validate values (e.g. support minimum/maximum), not just within structs, but also standalone values, which means an UnmarshalJSON directly on the type. I might end up doing something like your example, though. (Rejecting extra fields is on my list!)
Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration.
The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code generator, but the Go generator is just egregiously bad. There's no way to write a client or server that uses the Protobuf structs directly as first-class data types without ending up looking like a ball of spaghetti, at least not if you venture into the realm of oneofs, timestamps or the "well known types" set of types.
I think we're at a juncture where we really desperately need a better, unified way to express schemas and pass type-safe data over the network that flows through these schemas. I'm currently working on a project that involves both gRPC (which uses Protobuf), GraphQL and JSON Schema, with the backends written in Go, and the frontend in TypeScript, and the overlap between all of these is ridiculous. TypeScript is a fresh breath of air that mostly solves the brittleness of JavaScript's type system, and it's absolutely crucial to extend this type-safety all the way to the backend.
The challenge is that no language is the same, and a common schema format ends up targeting a lowest-common denominator. For example, GraphQL errs (in my opinion, wrongly) on the side of simplicity and doesn't support maps, so now you have a whole layer that needs to either custom types (JSON as an escape hatch, basically) or emulate maps using arrays of pairs, neither of which is ideal.
An approach I've used on the Go side is to build nice native Go types and converters to/from proto representations for wire marshaling. There's a minor performance hit but it's insignificant for what I'm doing. The result is quite nice, and you can largely ignore protobufs except for defining service APIs (which it's actually decent at).
Which tool are you using for TypeScript protobuf code generation? This is the next step for me, and I'd be keen to hear more of your experience there.
Yes, we use the same technique for our larger apps. We treat the Protobuf layer as a distinct layer related to the API; we have two packages, "protofy" and "unprotofy", so in the server implementation, you do something like:
Obviously, a bit less pretty in reality. But the principle is the same. Having two packages makes it easier to read — e.g. protofy.Foo() is always about taking a "native" Foo and turning into a *proto.Foo, and unprotofy.Foo() is the reverse.
For TypeScript, I'm using ts-protoc-gen [1]. The weird part, which I don't fully understand, is that the serialization code is emitted as JavaScript code. All the type definitions end up in a .d.ts file, but the client is a .js file. Which means you still get type safety and autocompletion, just like plain TS, but it's still weird to me.
It's a bit to learn but I promise you, it's worth it. The technologies are not redundant (jsonschema spec is for validation, hyperschema spec is for specifying how you interact, and LD is for semantics like language and more). If you take a few hours, read all 3 specs, you're almost guaranteed to imagine a new (albeit somewhat daunting) future, where APIs and clients can be smarter.
There's no reason I can think of that a sufficiently specified JSON schema cannot be automatically converted into a efficient binary format. I have absolutely zero reason to use Protobuf and it's whole ecosystem of tools if that's the case, I'd much rather go with tools built on a standard like JSON.
GraphQL's functionality is a subset of the above set of tools -- IIRC GraphQL can basically be boiled down to an option (or two) on a resource endpoint -- How you query on the server side does indeed change a bit to support the syntax, but this is basically the same as if you'd implemented with horizontal filtering like in POSTgrest[3], or resource embedding feature[4] (which everyone ends up doing at some point with REST-ish APIs). I honestly don't know who or what is trying so hard to drive GraphQL adoption so hard but I think it's mostly hype, and in terms of general technology, by default it forces clients to know the inner workings of the data model and that's bad for any kind of future in which you don't want to have to learn X APIs to interact with X systems.
BTW I absolutely love a good well-supported rant that agrees with all my biases (yes, Google as well as every other company has a bunch of amateurs, as well as a small amount of very skilled engineers), and all the problems noted with protobuf are real problems, and this viewpoint is sorely missed in all the talks and conferences where people are touting GRPC as the next sliced bread with Protobufs as the transport (you can change GRPC's transport layer to be JSON for example[5])
I'd love to see something with the approximate structure of JSON, but less JavaScript-bound syntax. EDN has caught my eye, but then again I've always been fonder of Lisp-y syntax as a whole.
Proper numeric types would like a word with you. (In particular, just look into how you would get infinity/NaN in there. Fun times.)
I mean, yes, you can do everything by just passing the string representation. Not exactly efficient, though. And most schema attempts in json are usually less than compelling.
You're absolutely right that it is a sticking point, but custom types can help this. Also, I don't know about you but I don't very often have Infinity or NaN as inputs that I want to see in my schemas...
Also, the JSON schema is an evolving document being developed in the open, it can be improved over time (they're already on Draft 7) -- if numeric types are lacking then suggest a way to make them better.
> And most schema attempts in json are usually less than compelling.
I'm not sure 100% what this means -- jsonschema is pretty easy to read and works pretty well for validation, people are already being very productive with it, what is not compelling?
I don't think GraphQL is over-hyped at all. Maybe it's flawed, but the design is absolutely on the right traack. GraphQL completely changes how you work with APIs in a front end.
I work on React apps, and by using GraphQL, a component's data requirements can now be entirely declarative. For example, a component can do this (simplified):
The component knows what it needs to render, so it declares that. With TypeScript, you can get type-safety all the way from the backend to the frontend, which means your IDE (e.g. VS Code) can correctly autocomplete, say, "creator." and suggest "name". It's rather magical.
I work on a product called Sanity [1], which evolves GraphQL one step further. It's a Firebase-like data store with schemas and joins, and a structured data model. Without implementing anything on the server end, you can run queries like these:
This doesn't just follow a child object (creator), it also joins with an unrelated set of objects (topPosts, which finds the top 10 most viewed posts created by the same user) as well as joining a bunch of 1:1 relations. I'm totally biased, but I think it's a game-changer when it comes to writing web apps, because you can just dump the whole query in a React component, no state management or API writing required.
Nothing about what you just posted couldn't be done with a normal RESTful endpoint with sufficient support for column-level filtering and embedded item filtering.
Your post is the perfect example of GraphQL is being over hyped. I absolutely get that there's a benefit to filtering at both these levels, and that the DSL cuts down on noise and gives you a way to "query" without thinking of web requests, but it's not a leap in thinking -- teams that have to deal with mobile environments have long been strapping on filtering options to endpoints to avoid sending unneeded bytes to mobile devices.
Yes -- GraphQL standardizes this stuff, but it does it in a way that is basically not compatible with anything else.
> The component knows what it needs to render, so it declares that.
??? The component doesn't know anything, components don't think. You're describing it as if you just gave the component a list of entities but you actually wrote a query DSL -- that's how the component "knows" -- you told it.
What GraphQL is doing for you here is:
- Enforcing consistent access patterns (which is how "posts" gets translated into the right URL)
- Ensuring "limit" is supported on the endpoint
- Ensuring horizontal filtering is supported
- Ensuring embedded entities get returned and they're filtered
This is not a paradigm shift. It's better, maybe, but that DSL will absolutely fail you at some point, when you try do a more dynamic query, and you'll have to drop back to writing code that looks a lot like life did before GraphQL.
> With TypeScript, you can get type-safety all the way from the backend to the frontend, which means your IDE (e.g. VS Code) can correctly autocomplete, say, "creator." and suggest "name". It's rather magical.
This is basically orthogonal... Write well typed javascript and your IDE is going to be able to help you out.
> Firebase-like data store with schemas and joins
You've lost me here. The excerpt you've posted looks even worse than SQL. At that point why not just send SQL directly to the backend (as long as you can get the permissions right and your DB is secure enough)?
Could one invent an ad-hoc REST API to perform joins, apply parameters and so forth? Of course. That's what people have been doing. That means every solution is ad-hoc. People get tired of this.
Have you tried any of the GraphQL client tools? You can literally point a client at an arbitrary server and not only see its schema, but also interact with the API. REST doesn't give you that, because the authors of REST neglected to actually specify anything beyond some fuzzy principles.
> ??? The component doesn't know anything, components don't think.
This is being obtuse. You know perfectly well what I meant: That the component localizes the information needed to make an informed request. This is domain knowledge encoded in the component.
This is contrary to how most developers design clients, where they might call an endpoint such as /posts?limit=10, which happens to fetch all attributes, even if the component doesn't actually need all the attributes.
> You've lost me here. The excerpt you've posted looks even worse than SQL.
Before a summary (and, honestly, tone deaf) dismissal such as this, I recommend reading up on it a bit [1], and maybe trying it out. SQL is great, but it does not handle nested documents/relationships, so you end up with a lot of messy structural mapping between flat relational data and structured data, which is why projects like Rails/ActiveRecord and Hibernate are so popular.
The point isn't that you cannot, with a sufficiently complex ORM and enough elbow grease, do that with SQL. It's that it should be unnecessary, because developers shouldn't need to write an entire data layer every time they want to bring an app up. This is part of what GraphQL brings to the table, too.
Which language are you talking about? C++ proto is built around CodedInputStream which is what it sounds like. You can use it with or without the generated message code.
I will say one thing. gzipped JSON is damn efficient over the wire. It elegantly solves the problem that field names are repeated in every object. The only problem is that adding compression and a serializationd/deserialization step cost CPU time. However if you have this problem then you're often better off by using something like cap'n proto or flatbuffers which do not have an extra serdes step. If you consider this then there is only a very very narrow band in which it makes sense to use protobuffers vs JSON (which is a defacto standard) or lesser known solutions that provide superior performance.
The problem with JSON as a wire/interop format is the lack of any sort of schema. Formats like proto are nice because if you can unmarshall the bytes successfully, you can be reasonably sure you've got a valid message and reason about its contents. A deserialized JSON object can literally be or contain anything.
Recent algorithms, like zstd or lz4 can do the same compression ratio as gzip for a fraction of the cost. Hopefully they will be replacing gzip over the coming decade.
> Option 1 is clearly the "right" solution, but its untenable with protobuffers. The language isn't powerful enough to encode types that can perform double-duty as both wire and application formats. Which means you'd need to write a completely separate datatype, evolve it synchronously with the protobuffer, and explicitly write serialization code between the two. Seeing as most people seem to use protobuffers in order to not write serialization code, this is obviously never going to happen.
This last is not the case. I've only ever seen it used when people a) already have written serialization code that works with XML/JSON, and b) don't want to write code to serialize to a binary format.
JSON also doesn't let you use objects as object keys, or have any form of polymorphism. It seems like the author wants protobufs to be an RPC system with a modern type system, when in fact it is a data interchange format.
But the author is only talking about data types. I don't see any reason it shouldn't be designed with composition in mind.
The only restriction I can think of that JSON imposes (in terms of composition) is that keys must be strings. And I don't necessarily think JSON is a great serialization format either.
This feels pretty vitriolic, I wouldn't be surprised if there is some bias here. A lot of these problems seem pretty minor and there's weird stuff like
"Unlike most companies in the tech space, paying engineers is one of Google's smallest expenses."
Google's annual revenue was $110 billion in 2017 [1]. Even if headcount doubled and salary has increased, that's $7 billion a year. That's not peanuts, but at a company level it's not a massive expense.
Anytime
Pull out the revenue gun. It sounds like the revenue is actually produced by the machines though. I wonder why there isn't another search company competing with Google...
Btw, if you use glass door to deduce Google's cost on employee, you probably have no idea how company works...
True. I'm also guessing this number changes a lot with stock grants (that number was just base pay), and that yearly number seems pretty low (NYT has the entry level engineer at 124,000
https://www.nytimes.com/2017/09/08/technology/google-salarie...) so glassdoor might not have good data.
I wouldn't be surprised if it was much more than 6%.
It's abundantly clear why this author lasted only a year at Google. Aside from using the non-idiomatic "protobuffers" ...
"nothing wants to inspect only some bits of a message and then forward it on unchanged"
In my experience it is extremely useful to partially parse a protobuf, or to not parse it at all and simply modify it by appending to it. Also useful is the ability to define a message type that is isomorphic on the wire with a different type, but is much cheaper to parse (example: a deeply nested structure where all of the fields of interest are defined at the top level).
Nobody even within Google will argue that no mistakes have been made with proto. Proto3 is generally acknowledged to have been a big mistake and it doesn't have a lot of traction within Google. Map and OneOf are both basically antagonistic to high performance application code, and people avoid those for that reason. But the author's arguments are pretty weak and his dismissal of Google itself as an existence proof is insufficiently supported.
Ah right... when you can't attack the ideas, attack the author. Sandy is a Haskell enthusiast, and it is not a surprise that his criticism of protobuf is inspired heavily by that.
The argument presented is that protobuf does not attempt to even replicate the best practices we already have in terms of data representation. It makes sense why languages like C are constrained in their data representations -- they want to be close to the machine. However, a language meant literally to specify data really ought to be able to handle the basic competencies of its field (like sum and product types and polymorphism) with ease.
There is no excuse in 2018 for a data language without polymorphism and without product and co-product types. Compatibility with legacy or substandard languages is not an excuse for poor design. Compatibility with needed legacy languages should be the thing that's tacked on, not basic functionality.
Sure that makes sense if you can write off C++ as a "legacy" language, which is fine if you just can't get your mind wrapped around the scale at which Google operates. Due to the nature of weighted averages, you can't just write off something as being a "Google-only problem". Google and its peers like Amazon and Facebook own a very large fraction of the world's computing resources.
I think the overall mistake you and the author are making is assuming it is desirable for one program examining an encoded message to respect the type system of the program that produced it. That assumption is not obviously correct. The flexibility to just treat a vector of numbers as a vector of numbers, or to skip it, or ignore it, etc are pretty important in practice. That is why protobuf is defined at the level where it is defined. It encodes numbers and strings on the wire. It happens to be a fairly good way to represent search data. It brings no CS academic wankery to the party. It's very practical.
C++ is a legacy language when it comes to data representation. Its ideas on what constitutes 'data' are from a different era. That is not a criticism of C++ or its utility. It is just a fact that data representation has changed a lot since the time of C++s development.
Products and coproduct types are not academic wankery.
The machine does not think about category theory. The machine thinks about numbers. It can add them together! The way the machine thinks about numbers has not meaningfully changed in 40+ years.
Put me firmly in the camp of "optional fields are bad." I believe all fields should be required.
I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility.
The idea of "forwards compatibility" is a pipedream, in my opinion. Just because something won't crash because a field is or isn't there doesn't mean that it will actually work. Code ages over time, and instead of looking at the spec, which will tell you which fields are required, you need to read the documentation, which may or may not tell you what is required. And especially when dealing with an older client, it simply may not work because what used to be optional-optional fields become required-optional fields.
It's the same argument as schemaless NOSQL database like Mongo vs MySQL. Having a schema is a pain in the ass, but it's better over time and ages much more gracefully than schemaless ones, because things won't just break.
If all fields are required, you cannot have middleware that processes multiple versions of the same protobuf, and every application has to be updated when a field is added, even if they do not use that field. This is one of the more important design goals underlying not only protobufs, but most of the non-language specific binary formatters.
> every application has to be updated when a field is added, even if they do not use that field
This is when, in a protocol, you reach for the hammer called "extensibility."
In this case (decoding to native structs), you'd probably have your FooMessage product-type have an "extensions" field, which is a list of zero or more FooExtensionStructs, where a FooExtensionStruct is a sum type of the known extensions to FooMessage.
Then, just make the semantics of your format such that a sum type can have a pragma in its IDL indicating that it isn't required to be decodable for the product-type containing it to be successfully decoded. I.e., the sum type becomes a (DecodeResult = Decoded(sum type) | Opaque(raw wire data)).
Opaque (unrecognized) extension data can't be manipulated by the program, but it can losslessly survive a trip back through the wire encoder. So you can choose to pass it through in your middleware, or not.
Where something like JSON or (if you must) XML is used I prefer the idea of REQUIRING the preservation of all fields in the original structure UNLESS a field or set of fields is validated and then updated in place.
This lets end users and extension authors do things that make sense, such as adding a comment tag ( JSON added key:value "__customcomment": "Any well formed string will be safe here." or XML <!-- Please don't eat me when parsing the config! --> ) to a configuration file stored in either format, and actually having it persist.
How do you handle binary rollbacks and rollouts safely with an "everything is required" approach? Do you force binaries to roll out in a strict order with appropriate soak time at each layer? How does that affect developer velocity?
or, using a language with a good type system like Haskell
data MyData = MyData Int Int deriving Generic
instance Storable MyData
alloca $ \buf -> do
let d = MyData field1 field2
poke buf d
send socket buf (sizeof d)
C doesn't make struct memory layout guarantees, endian guarantees, or size guarantees in some cases, so unless you're running the exact same binary this is a very poor serialization technique.
C does not, but few compilers implement standard C. Most C compilers do have documentation on how structs are laid out and many options to change the behavior. Most C compilers document this.
This may be easy, but it’s wrong. Endianness issues are just the start. Information leaks due to padding are a big deal. And it straight up doesn’t work if nontrivial data structures are involved.
Endianness issues are something every programmer should be aware of when sending data over the wire. I'm sorry I didn't insert the htons, htonls in my C code.
In the Haskell code, endianness is handled by your Storable implementation (you define it after all, and can customize it however you want).
I agree my example is somewhat tongue in cheek. The point though is that most languages have standard libraries for dealing with binary data that are almost universally more straightforward than proto bufs.
htons, etc are very 1980s, and I’ll make a fairly strong claim: they should never be used in new code, with a single exception. The reason is that an int with network endianness simply should not exist. In other words, when someone sends you a four byte big-endian integer, they sent four bytes, not an int. You can turn it into an int by shifting each byte by the relevant amount and oring them together. And a modern compiler will generate good code.
The sole exception is legacy APIs like inet_aton() that actually require these nonsensical conversions.
These objections are interesting. I have mixed opinions on protos, I think overall I'm mostly in favor. I'm a bit confused by this set of objections though.
While oneof fields cannot be repeated, oneof fields can be arbitrary protos, so they can contain repeated fields. In other words, you can have a (pseudo-proto)
so in practice this isn't a restriction. If anything, its a (very minor) api wart.
As for maps, I'd say don't use them. I have, I don't think they're very useful, except for prototyping things. `repeated (string key, string value)` is just as useful for prototyping things, and you should be quick to promote things to fields. Optional fields aren't costly.
I'm also in the minority who thinks that the `has_foo` methods are a smell, and everything should always be set to a default value[0]. If it really matters, one can define a maybe type via a oneof, but that should be the exception, not the norm. Most people disagree with that though, so maybe I'm crazy and you should ignore me.
[0]: But if that's not the case, everything should be optional, nothing required.
While oneof fields cannot be repeated, oneof fields can be arbitrary protos, so they can contain repeated fields. In other words, you can have a oneof, so in practice this isn't a restriction. If anything, its a(very minor) api wart.
That's right. Moreover, oneof repeated and repated oneof are two different types: in first, you have a list of X, or a list of Y, or a list of Z etc, and in the second, you have a list such that any element can be either X, Y or Z. Both can be cleanly expressed with existing API, so I see no reason to do any special treatment.
Yet protobuf is probably the most compact, efficient and performant serialization method especially when saving bandwidth is important. I experimented with protofbuf, flatbuffers and messagepack and always found protobuf messages the most compact by a noticeable margin
That's the core of the author's argument. Protobuffers optimize for something besides usability and maintainability, because Google cares more about incremental performance than developer-friendliness. Which is a fine thing to care about at Google's scale, but maybe others' calculations should be different.
I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises.
- D
P.S. I also don't believe the personal attacks to be warranted or productive.
I spent 2.5 years at Google, and most of what I did was pushing one protobuf from one place to another :) - and I loved it... Honestly though, you can complain all day, but some of the decisions made in the list you presented most likely come from experience (daily) not as a user of protobufs (which I simply was), but someone that had to support a plethora of compression formats, how protobufs gets stored in the different databases, proxying, etc. etc.
Your "oneoff" and "repeated" might've been (I dunno really), a decision based on metrics where encoding repeated oneoff would've cost more, or made problems, more testing, or who knows what.
For one, Google was, and I believe still is open to discussing all kinds of matters, and you can get on to a design doc and comment if you like, if you care, if you have something to say... You can definitely influence, and people really did talk (you can even experience this from their public design docs - like from chromium, etc.).
The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article.
Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages.
No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases.
If you are building a system where your data can be described by protobufs, it may be a good choice. If your data structures don’t mash up well and you have to manipulate them heavily, protobufs may be a bad choice.
Instead of pointing out use cases where a different serialization format may be better than protobufs and use cases where protobufs are better, and why, the author is spouting dogma about how protobufs are bad for every use case.
Be wary of working with developers who prefer to argue about why they hate certain technologies instead of providing useful data and ways to solve problems. You don’t always have to solve a problem just because you are aware of it but don’t go shouting from the rooftops that a technology sucks for every use case under the planet when that’s obviously not the case. The author’s opinion is more of: I don’t like protobufs. Not: protobufs are wrong
The author isn’t trying to say either of those things, really.
I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, they really don’t work for that use-case—and an ESB means having to try to shove a wire format into pretty much every use-case. So “when it’s bad, it’s really bad” is a bad property for a format aimed specifically at being an enterprise’s ESB-interchange-format dogma to have.
(Contrast to, say, JSON-RPC, which I’d place on the opposite end of the scale being described here. JSON-RPC is mediocre at best in terms of data fidelity, performance, etc., but it’s inoffensive—when it’s bad, it’s no worse than when it’s good. That makes it a good choice of ESB dogma... even though, in engineering terms, it sucks!)
I see protobuf as being designed to be serialized fast and with a low footprint by not too complex assembly or C or Go. All the trade-offs are correct with that mindset. Maybe the author is looking at this from an academic type-theory perspective, and that's why he can't see the "why" for each one of the trade-offs.
Hmmm, I suppose I objectively agree with some of the points the author made, but as someone who works with protocol buffers daily, those issues never actually come to be problematic in practice. In fact, I have nothing but positive things to say about protocol buffers and find them pleasant to work with. Definitely a step up from sending raw JSON down the wire.
Granted, the application I'm working on is fairly boring/vanilla so maybe I don't feel the pain points that come from going off the beaten path.
There are enough voices on the net these days that we don't have to put up with this kind of attitude anymore. This sort of stuff was "just the way things were" when we were teenagers visiting forums ran by teenagers. I don't know if the guy is right or wrong, I've just got better things to do than to sift through all the vitriol to get to whatever point he is trying to make.
I think it was Churchill said that "protobufs are the worst form of data serialization, except for all those other forms that have been tried from time to time." I really don't understand why you would ever want to serialize data without a description of what can be in the serialized data blob. And yes, I do care about the difference between a single precision float, a double precision float, or an int64. Thank you, typing system.
115 comments
[ 1.7 ms ] story [ 122 ms ] threadI'd also be interested in what alternatives the author might suggest.
Cap'n Proto also suffers from a number of practical disadvantages, which can be quite severe depending on your use case. The main problem is eco system maturity and rate of adoption. So whilst protobufs have excellent cross-language support and a mature RPC system, in cap'n proto everything but C++ feels at bit second class, even other popular languages. Python and Rust, and I believe Java have workable, but not necessarily great bindings for serialization and in theory RPC, but the RPC support for other languages than C++ didn't look that great last I looked (but that was over a year ago) and even with C++ it looks much easier to get something going with gRPC (tooling, http2 based etc). For other languages I looked at (e.g. Ocaml, Lua and Haskell) the bindings looked very immature. There are a few other minor annoyance with capnproto, such as the lack of a timestamp type (you can build your own, incompatible to everyone else's, but this is still an annoying omission). The C++ library is also saddled with the weird home-grown kj library that no one other than Kenton Varda uses. However, it has to be said that the C++ API and also the Python API for Capnproto feel a lot more natural and idiomatic than the protobuf ones. If protobuf is too high overhead for your use case and you don't need the wide language and eco system support, Cap'n Proto is worth a serious look.
Also, the nested query structure can save what would have normally been multiple requests with a by-the-book RESTful HATEOAS setup.
Personally, these days I’m taking the approach of just outright avoiding pretty much anything google does. Either because it solves a problem only they have, or they’ll probably just drop it out of the blue one day. Or like Tensorflow, it’s fucking impossible to get your head around because their documentation seems to be written with the goal of being maximally confusing.
But I agree ... I almost wish they'd finish it off and go for full on serialization.
That said, it may not be that-that hard, a serialization spec can be pretty short if need be. Certainly one could 'roll one's own' for a mid-sized project if need be.
Not sure if the best, but maybe has different tradeoffs
Transit has the following features:
- self-describing
- allows extension types on top of base types
- strong value types + dynamic serialization
- can transparently pass through data without consumer knowing about schema
- has "repeated key" optimization in the encoder to allow compression in the stream
- allows transparent binary encoding with msgpack
- is stream-based, so processing can begin immediately unlike JSON which has enclosing {}
This set of features basically allow transparent proxying to JSON as well, which is something that proto2/3 cannot do without updating the proxy.
What a pain! Well, at least Google won't make that mistake again!
It doesn't matter if you're using a quantum binomial tree in a black hole: save it as a 'stupid' map<k,v> when it hits the network. That way everyone who interacts with your service can decide how they want to represent that structure. You can compose any in-memory data structure you can dream of, you can validate the data with more richness than "missing" or "present", and Protobuf doesn't contaminate your codebase.
Option 3 is the correct choice. Serialization libraries should be amateurish by design.
Protobufs aren't a message format for publicly specced standard wire protocols. They're a serialization layer for polyglot RPC request and response messages. The whole point of them is that you're transporting the same typed data around between different languages, rather than there having to be two conversions (source language -> "standard format"; "standard format" -> dest language).
In this sense, they're a lot like, say, Erlang's External Term Format—driven entirely by the needs of an RPC-oriented distributed-computation wire protocol, doing interchange between nodes that aren't necessarily running the same version of the same software. Except, unlike ETF, Protobufs can be decoded without the backing of a garbage-collected language runtime (e.g. in C++ nodes.)
I'm not saying Protobufs are the best, but you have to understand the problem they're solving—the "rivals" to Protobufs are Cap'n Proto and Thrift, not JSON. JSON can't even be used to do a lot of the stuff Protobufs do (like talk to memory-constrained embedded kernels running on SDN infra.)
> and Protobuf doesn't contaminate your codebase
Just like in HTTP web-app codebases, a codebase properly designed to interact with an https://en.wikipedia.org/wiki/Enterprise_service_bus will tend toward Hexagonal architecture—you have your own logic, and then you have an "RPC gateway" that acts as an MVC system, exposing controllers that decode from the RPC format and forward requests into your business logic; and then build responses back into the RPC "view" format.
Once you have that, there's no point in having your application's parsed-out representation of the RPC format be different than the on-the-wire RPC format. The only thing that's touching those RPC messages is your RPC gateway's controller code anyway. So why not touch them with as little up-front design required as possible?
Typically, the data types and interfaces generated by these tools are so poor that you need to build another layer on top that translates between "Protobuf types" and "native types", as you describe in your comment, and shields the application from the nitty-gritty details of gRPC. So investing in gRPC means that when you've generated, say, Go files from your .proto files, you're only halfway done. Protobuf in itself introduces a kind of impedance mismatch, a kind of stupid in-bred cousin whose limited vocabulary has to be translated back and forth into proper language.
So gRPC/Protobuf solves something important at the wire level, but developers really want to productively communicate via APIs, and so what you have is just half the solution.
I wish the Protobuf/gRPC toolchain were organized in such a way that the generated code could actually be used as first-class code. Maybe similar to how parser generators like Lex/YACC or Ragel work, where you provide implementation code that is threaded through the output code.
But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential.
It's like complaining a car that drives fast and has great gas efficiency of cannot talk. (Well in 2018, it might be less of a stretch to demand a car that can talk)
Huh? It's a serialization format. This problem has been solved many times in many different ways.
The author is pointing out that Protobufs were designed poorly, and they didn't have to be.
Mind provide examples of such tools.
Yes, we do.
But the appropriate answer in most cases is "it's up to the business logic". Trying to solve it at the transport layer is futile.
it doesn't mean that there is a single solution that will satisfy everyone
Your constraints on protocols can change over time and context, changing something that is required to something that is optional can cause crashes in prod. Unless somehow the 'optional' design has a way to handle that? You also will have to do custom validation anyways, as your storage format will never be able to enforce all constraints you care about.
Also, it would seem like the 'debate' of optional vs required is harmful was resolved, as proto3 makes everything optional.
> For example, we can rebuild optional fields:
Criticizing protobufs is like criticizing C++ or Java. Both have major shortcomings, but solve practical problems and de facto lingua franca with no practical solutions to replace them.
I don't this complaint. You have to initialize data with something. proto2 had to "solve" this problem in C/C++/Go by making everything a pointer. How is dealing with null the "sane" case against zero-initialization?
The class layout for proto3 is the same (surprise!) but the presence vector is ignored.
Go has its own philosophy about zero values which is controversial, but at least with pointers you do get to express optional values. And it's not like you can't work around the ergonomic awkwardness that arises:
Hardly elegant, but this is Go.Go also run into the same issue when encoding and decoding JSON. Most languages do distinguish between empty string and a missing string, but not Go, which makes it hard to validate anything. I wrote a code generator for JSON Schema [1] recently, which applies validations while deserializing, and has to resort to a rather low-tech trick to do so:
(Clearly there are slightly more refined and faster ways to do this, but not that easily without importing third-party code, which I wanted to avoid.)C and Go are in the minority here. This doesn't come up in languages like Rust, Swift, TypeScript, Nim, Haskell, OCaml, C#, F#, or Java >= 8, all of which have some sort of optional support (algebraic data types or built-in). For example, TypeScript:
Or Rust: [1] https://github.com/atombender/go-jsonschemaThe utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code generator, but the Go generator is just egregiously bad. There's no way to write a client or server that uses the Protobuf structs directly as first-class data types without ending up looking like a ball of spaghetti, at least not if you venture into the realm of oneofs, timestamps or the "well known types" set of types.
I think we're at a juncture where we really desperately need a better, unified way to express schemas and pass type-safe data over the network that flows through these schemas. I'm currently working on a project that involves both gRPC (which uses Protobuf), GraphQL and JSON Schema, with the backends written in Go, and the frontend in TypeScript, and the overlap between all of these is ridiculous. TypeScript is a fresh breath of air that mostly solves the brittleness of JavaScript's type system, and it's absolutely crucial to extend this type-safety all the way to the backend.
The challenge is that no language is the same, and a common schema format ends up targeting a lowest-common denominator. For example, GraphQL errs (in my opinion, wrongly) on the side of simplicity and doesn't support maps, so now you have a whole layer that needs to either custom types (JSON as an escape hatch, basically) or emulate maps using arrays of pairs, neither of which is ideal.
Which tool are you using for TypeScript protobuf code generation? This is the next step for me, and I'd be keen to hear more of your experience there.
For TypeScript, I'm using ts-protoc-gen [1]. The weird part, which I don't fully understand, is that the serialization code is emitted as JavaScript code. All the type definitions end up in a .d.ts file, but the client is a .js file. Which means you still get type safety and autocompletion, just like plain TS, but it's still weird to me.
[1] https://github.com/improbable-eng/ts-protoc-gen
JSON + JSONSchema[0] +/- JSON Hyperschema[1] +/- JSON LD[2]
It's a bit to learn but I promise you, it's worth it. The technologies are not redundant (jsonschema spec is for validation, hyperschema spec is for specifying how you interact, and LD is for semantics like language and more). If you take a few hours, read all 3 specs, you're almost guaranteed to imagine a new (albeit somewhat daunting) future, where APIs and clients can be smarter.
There's no reason I can think of that a sufficiently specified JSON schema cannot be automatically converted into a efficient binary format. I have absolutely zero reason to use Protobuf and it's whole ecosystem of tools if that's the case, I'd much rather go with tools built on a standard like JSON.
GraphQL's functionality is a subset of the above set of tools -- IIRC GraphQL can basically be boiled down to an option (or two) on a resource endpoint -- How you query on the server side does indeed change a bit to support the syntax, but this is basically the same as if you'd implemented with horizontal filtering like in POSTgrest[3], or resource embedding feature[4] (which everyone ends up doing at some point with REST-ish APIs). I honestly don't know who or what is trying so hard to drive GraphQL adoption so hard but I think it's mostly hype, and in terms of general technology, by default it forces clients to know the inner workings of the data model and that's bad for any kind of future in which you don't want to have to learn X APIs to interact with X systems.
BTW I absolutely love a good well-supported rant that agrees with all my biases (yes, Google as well as every other company has a bunch of amateurs, as well as a small amount of very skilled engineers), and all the problems noted with protobuf are real problems, and this viewpoint is sorely missed in all the talks and conferences where people are touting GRPC as the next sliced bread with Protobufs as the transport (you can change GRPC's transport layer to be JSON for example[5])
[0]: https://json-schema.org/
[1]: https://datatracker.ietf.org/doc/draft-handrews-json-schema-... (also on json-schema.org)
[2]: https://json-ld.org/
[3]: https://postgrest.org/en/v5.1/api.html#horizontal-filtering-...
[4]: https://postgrest.org/en/v5.1/api.html#resource-embedding
[5]: https://grpc.io/blog/grpc-with-json
I mean, yes, you can do everything by just passing the string representation. Not exactly efficient, though. And most schema attempts in json are usually less than compelling.
Also, the JSON schema is an evolving document being developed in the open, it can be improved over time (they're already on Draft 7) -- if numeric types are lacking then suggest a way to make them better.
> And most schema attempts in json are usually less than compelling.
I'm not sure 100% what this means -- jsonschema is pretty easy to read and works pretty well for validation, people are already being very productive with it, what is not compelling?
I work on React apps, and by using GraphQL, a component's data requirements can now be entirely declarative. For example, a component can do this (simplified):
The component knows what it needs to render, so it declares that. With TypeScript, you can get type-safety all the way from the backend to the frontend, which means your IDE (e.g. VS Code) can correctly autocomplete, say, "creator." and suggest "name". It's rather magical.I work on a product called Sanity [1], which evolves GraphQL one step further. It's a Firebase-like data store with schemas and joins, and a structured data model. Without implementing anything on the server end, you can run queries like these:
This doesn't just follow a child object (creator), it also joins with an unrelated set of objects (topPosts, which finds the top 10 most viewed posts created by the same user) as well as joining a bunch of 1:1 relations. I'm totally biased, but I think it's a game-changer when it comes to writing web apps, because you can just dump the whole query in a React component, no state management or API writing required.[1] https://www.sanity.io/
Your post is the perfect example of GraphQL is being over hyped. I absolutely get that there's a benefit to filtering at both these levels, and that the DSL cuts down on noise and gives you a way to "query" without thinking of web requests, but it's not a leap in thinking -- teams that have to deal with mobile environments have long been strapping on filtering options to endpoints to avoid sending unneeded bytes to mobile devices.
Yes -- GraphQL standardizes this stuff, but it does it in a way that is basically not compatible with anything else.
> The component knows what it needs to render, so it declares that.
??? The component doesn't know anything, components don't think. You're describing it as if you just gave the component a list of entities but you actually wrote a query DSL -- that's how the component "knows" -- you told it.
What GraphQL is doing for you here is:
- Enforcing consistent access patterns (which is how "posts" gets translated into the right URL)
- Ensuring "limit" is supported on the endpoint
- Ensuring horizontal filtering is supported
- Ensuring embedded entities get returned and they're filtered
This is not a paradigm shift. It's better, maybe, but that DSL will absolutely fail you at some point, when you try do a more dynamic query, and you'll have to drop back to writing code that looks a lot like life did before GraphQL.
> With TypeScript, you can get type-safety all the way from the backend to the frontend, which means your IDE (e.g. VS Code) can correctly autocomplete, say, "creator." and suggest "name". It's rather magical.
This is basically orthogonal... Write well typed javascript and your IDE is going to be able to help you out.
> Firebase-like data store with schemas and joins
You've lost me here. The excerpt you've posted looks even worse than SQL. At that point why not just send SQL directly to the backend (as long as you can get the permissions right and your DB is secure enough)?
Have you tried any of the GraphQL client tools? You can literally point a client at an arbitrary server and not only see its schema, but also interact with the API. REST doesn't give you that, because the authors of REST neglected to actually specify anything beyond some fuzzy principles.
> ??? The component doesn't know anything, components don't think.
This is being obtuse. You know perfectly well what I meant: That the component localizes the information needed to make an informed request. This is domain knowledge encoded in the component.
This is contrary to how most developers design clients, where they might call an endpoint such as /posts?limit=10, which happens to fetch all attributes, even if the component doesn't actually need all the attributes.
> You've lost me here. The excerpt you've posted looks even worse than SQL.
Before a summary (and, honestly, tone deaf) dismissal such as this, I recommend reading up on it a bit [1], and maybe trying it out. SQL is great, but it does not handle nested documents/relationships, so you end up with a lot of messy structural mapping between flat relational data and structured data, which is why projects like Rails/ActiveRecord and Hibernate are so popular.
For example, a query such as this:
will return something like: The point isn't that you cannot, with a sufficiently complex ORM and enough elbow grease, do that with SQL. It's that it should be unnecessary, because developers shouldn't need to write an entire data layer every time they want to bring an app up. This is part of what GraphQL brings to the table, too.[1] https://www.sanity.io/docs/data-store/how-queries-work
https://developers.google.com/protocol-buffers/docs/referenc...
This last is not the case. I've only ever seen it used when people a) already have written serialization code that works with XML/JSON, and b) don't want to write code to serialize to a binary format.
JSON also doesn't let you use objects as object keys, or have any form of polymorphism. It seems like the author wants protobufs to be an RPC system with a modern type system, when in fact it is a data interchange format.
The only restriction I can think of that JSON imposes (in terms of composition) is that keys must be strings. And I don't necessarily think JSON is a great serialization format either.
"Unlike most companies in the tech space, paying engineers is one of Google's smallest expenses."
According to here https://www.quora.com/How-many-software-engineers-does-Googl... there are ~28,000 engineers in 2014, which paying at 120,00 a year (glassdoor) would be... $3.3 billion a year? And this was 2014, and doesn't include stock compensation
Don't get me wrong - there are definitely problems with protobufs. But this post is citing a lot of stuff to back itself up that doesn't add up.
Also, required vs optional - definitely only optional, there should be nothing required. So I flat out disagree with that one.
[1]: https://www.androidauthority.com/alphabet-q4-2017-earnings-8...
Btw, if you use glass door to deduce Google's cost on employee, you probably have no idea how company works...
I wouldn't be surprised if it was much more than 6%.
"nothing wants to inspect only some bits of a message and then forward it on unchanged"
In my experience it is extremely useful to partially parse a protobuf, or to not parse it at all and simply modify it by appending to it. Also useful is the ability to define a message type that is isomorphic on the wire with a different type, but is much cheaper to parse (example: a deeply nested structure where all of the fields of interest are defined at the top level).
Nobody even within Google will argue that no mistakes have been made with proto. Proto3 is generally acknowledged to have been a big mistake and it doesn't have a lot of traction within Google. Map and OneOf are both basically antagonistic to high performance application code, and people avoid those for that reason. But the author's arguments are pretty weak and his dismissal of Google itself as an existence proof is insufficiently supported.
The argument presented is that protobuf does not attempt to even replicate the best practices we already have in terms of data representation. It makes sense why languages like C are constrained in their data representations -- they want to be close to the machine. However, a language meant literally to specify data really ought to be able to handle the basic competencies of its field (like sum and product types and polymorphism) with ease.
There is no excuse in 2018 for a data language without polymorphism and without product and co-product types. Compatibility with legacy or substandard languages is not an excuse for poor design. Compatibility with needed legacy languages should be the thing that's tacked on, not basic functionality.
I think the overall mistake you and the author are making is assuming it is desirable for one program examining an encoded message to respect the type system of the program that produced it. That assumption is not obviously correct. The flexibility to just treat a vector of numbers as a vector of numbers, or to skip it, or ignore it, etc are pretty important in practice. That is why protobuf is defined at the level where it is defined. It encodes numbers and strings on the wire. It happens to be a fairly good way to represent search data. It brings no CS academic wankery to the party. It's very practical.
Products and coproduct types are not academic wankery.
I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility.
The idea of "forwards compatibility" is a pipedream, in my opinion. Just because something won't crash because a field is or isn't there doesn't mean that it will actually work. Code ages over time, and instead of looking at the spec, which will tell you which fields are required, you need to read the documentation, which may or may not tell you what is required. And especially when dealing with an older client, it simply may not work because what used to be optional-optional fields become required-optional fields.
It's the same argument as schemaless NOSQL database like Mongo vs MySQL. Having a schema is a pain in the ass, but it's better over time and ages much more gracefully than schemaless ones, because things won't just break.
This is when, in a protocol, you reach for the hammer called "extensibility."
In this case (decoding to native structs), you'd probably have your FooMessage product-type have an "extensions" field, which is a list of zero or more FooExtensionStructs, where a FooExtensionStruct is a sum type of the known extensions to FooMessage.
Then, just make the semantics of your format such that a sum type can have a pragma in its IDL indicating that it isn't required to be decodable for the product-type containing it to be successfully decoded. I.e., the sum type becomes a (DecodeResult = Decoded(sum type) | Opaque(raw wire data)).
Opaque (unrecognized) extension data can't be manipulated by the program, but it can losslessly survive a trip back through the wire encoder. So you can choose to pass it through in your middleware, or not.
Boom: you've reinvented the "chunks" concept of the https://en.wikipedia.org/wiki/Interchange_File_Format, as seen in PNG and ELF.
This lets end users and extension authors do things that make sense, such as adding a comment tag ( JSON added key:value "__customcomment": "Any well formed string will be safe here." or XML <!-- Please don't eat me when parsing the config! --> ) to a configuration file stored in either format, and actually having it persist.
In the Haskell code, endianness is handled by your Storable implementation (you define it after all, and can customize it however you want).
I agree my example is somewhat tongue in cheek. The point though is that most languages have standard libraries for dealing with binary data that are almost universally more straightforward than proto bufs.
The sole exception is legacy APIs like inet_aton() that actually require these nonsensical conversions.
While oneof fields cannot be repeated, oneof fields can be arbitrary protos, so they can contain repeated fields. In other words, you can have a (pseudo-proto)
so in practice this isn't a restriction. If anything, its a (very minor) api wart.As for maps, I'd say don't use them. I have, I don't think they're very useful, except for prototyping things. `repeated (string key, string value)` is just as useful for prototyping things, and you should be quick to promote things to fields. Optional fields aren't costly.
I'm also in the minority who thinks that the `has_foo` methods are a smell, and everything should always be set to a default value[0]. If it really matters, one can define a maybe type via a oneof, but that should be the exception, not the norm. Most people disagree with that though, so maybe I'm crazy and you should ignore me.
[0]: But if that's not the case, everything should be optional, nothing required.
That's right. Moreover, oneof repeated and repated oneof are two different types: in first, you have a list of X, or a list of Y, or a list of Z etc, and in the second, you have a list such that any element can be either X, Y or Z. Both can be cleanly expressed with existing API, so I see no reason to do any special treatment.
So you have
Or what I did in my previous comment.I'm an actual author of Protocol Buffers :)
I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises.
- D
P.S. I also don't believe the personal attacks to be warranted or productive.
Your "oneoff" and "repeated" might've been (I dunno really), a decision based on metrics where encoding repeated oneoff would've cost more, or made problems, more testing, or who knows what.
For one, Google was, and I believe still is open to discussing all kinds of matters, and you can get on to a design doc and comment if you like, if you care, if you have something to say... You can definitely influence, and people really did talk (you can even experience this from their public design docs - like from chromium, etc.).
And I love protobufs, so I guess I'm biased...
The tone of this article is a little less than objective which I feel discredits it a bit. But hey it made it to the front page of hacker news so...
Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages.
No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases.
If you are building a system where your data can be described by protobufs, it may be a good choice. If your data structures don’t mash up well and you have to manipulate them heavily, protobufs may be a bad choice.
Instead of pointing out use cases where a different serialization format may be better than protobufs and use cases where protobufs are better, and why, the author is spouting dogma about how protobufs are bad for every use case.
Be wary of working with developers who prefer to argue about why they hate certain technologies instead of providing useful data and ways to solve problems. You don’t always have to solve a problem just because you are aware of it but don’t go shouting from the rooftops that a technology sucks for every use case under the planet when that’s obviously not the case. The author’s opinion is more of: I don’t like protobufs. Not: protobufs are wrong
I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, they really don’t work for that use-case—and an ESB means having to try to shove a wire format into pretty much every use-case. So “when it’s bad, it’s really bad” is a bad property for a format aimed specifically at being an enterprise’s ESB-interchange-format dogma to have.
(Contrast to, say, JSON-RPC, which I’d place on the opposite end of the scale being described here. JSON-RPC is mediocre at best in terms of data fidelity, performance, etc., but it’s inoffensive—when it’s bad, it’s no worse than when it’s good. That makes it a good choice of ESB dogma... even though, in engineering terms, it sucks!)
Granted, the application I'm working on is fairly boring/vanilla so maybe I don't feel the pain points that come from going off the beaten path.
This article was clearly written by amateurs.