string and bytes are the same on the wire but generate different application code. string is defined to be utf-8, so if you are planning to put arbitrary bytes in there, you can't use that type. in some languages string is checked for utf8 conformance, which is expensive. In some languages allocating strings is more expensive than allocating bytes.
I recently had this debate in a professional context. My correspondent argued that string was good for filenames, because it's a human-readable string. I pointed out all the different ways that a filename (on a variety of platforms!) can be constructed such that it will not conform to utf8. In a new system I would want to see an iron-clad reason for string in this case, because it's not obviously correct and the efficiency story isn't on the side of string, either.
Luckily you can change from string to bytes at any time, since the wire format is the same, so this isn't the kind of mistake that gets cast in stone.
> never use type “string” for keys and values in protocol buffers. Always use “bytes”.
What are "keys and values" with regard to protobuf? Surely you don't mean maps as bytes can't be "keys" anyways. And why shouldn't you use string as a field type?
Internally, google had best-practices (but could be outdated), that each RPC service function should take unique input message, and unique output message, and never reuse one, even if empty, or having same structure, at the cost of more writing.
Later this allows one to change things independently.
I dunno. I'm simply recalling what I remember as best-practices by memory, could've been wrong about some of them, but for my own protobufs that I use I keep the same practice.
But mainly for that reason, I would like to keep as much as backward compatibility, and if I have need to add a field, I can do it without changing some other function. (If need to, I'll add it in both places). Hard to tell really, for some things it might not make sense at all.
No, you're right. EmptyMessage exists but you shouldn't make it an input or output to an RPC if that RPC might ever in the future accept or return arguments. Since people are super bad about predicting the future, just call it WhateverRequest and WhateverReply and if there are no arguments today, don't put any fields in those types.
I would also not use Any as a request or reply type, but Any, in general, is fairly common in practical use.
https://google.aip.dev/ gives lots of advice for structuring a gRPC API. What you say is mostly in line with that, with some exceptions. For example https://google.aip.dev/135 suggests that a delete operation on a resource should normally return google.protobuf.Empty. Of course, this is implicitly a sum type; you either get an empty response or an error message. If you needed to return anything else, you're not just implementing a delete. Two exceptions they call out are soft-deletes (not just a delete...) returning the resource itself, and a delete that's a long-running operation.
I'm more interested in DB's that store protocol buffers, and can expose their innards (like spanner, bigtable, etc.). What's good to use/try on the outside?
Protobuf has well-defined JSON semantics, so leveraging the JSON encoding seems like the practical way there and many databases can query/index/manipulate the contents (and if space is a concern, often JSONB and similar types are quite compact).
I'm not sure what the internal representation is. Spanner is aware of the proto type through some mixture of DDL and build rules. Updates to proto types get pushed when updated schemas are pushed. You can use deeply-nested fields in various query clauses, create indexes, etc. But I believe updates must replace the whole object. Much like JSONB columns in other RDBMS.
"... The effort to standardize our own implementations both covers gaps in the SQL standard where details are unclear or left to the implementation and covers our own Google specific extensions. Before defining our own SQL dialect, we did consider other options. Using the front end of an open-source SQL processing system, such as Postgres, would have accomplished several of the same
goals including filling the gaps left by the SQL standard and ensuring uniform syntax/semantics across our systems. We ruled out this approach because internal systems at Google make heavy use of Protocol Buffers in their applications, and we wanted Protocol Buffer message and enum types to be first class types in our SQL language"
It allows you to execute code on another machine. Your code makes a function call that returns a status / error, except the function runs on another node. It's another RPC framework and there are more caveats like what guarantees a framework makes. Did the function execute exactly once, at most once or at least once.
I have been poking around building an eerily similar example[1] service. I am trying to come up with a good pattern, using go-micro, where a team can quickly and easily leverage the protobuf IDL for gRPC/REST interfaces across large teams. It sounds like it should be easy, right?
The tooling around pulling in protobuf at compile time is a huge pain in the ass. I've been spending an hour or so every night over past few weeks searching issues to figure out why things aren't working. This is a great reference point for me, thanks!
29 comments
[ 1.9 ms ] story [ 99.1 ms ] threadMy google-fu only turns up references to invalid utf-8 data, but that didn’t sound like what you were referring to.
Only use string when you really mean utf8.
Luckily you can change from string to bytes at any time, since the wire format is the same, so this isn't the kind of mistake that gets cast in stone.
What are "keys and values" with regard to protobuf? Surely you don't mean maps as bytes can't be "keys" anyways. And why shouldn't you use string as a field type?
Later this allows one to change things independently.
I would also not use Any as a request or reply type, but Any, in general, is fairly common in practical use.
Neither Spanner nor Bigtable has this feature. If you store a protobuf in a Bigtable column, that's your own business. Bigtable itself won't notice.
This is from memory. I'm not at Google anymore.
also some bare mention here - https://static.googleusercontent.com/media/research.google.c...
"... The effort to standardize our own implementations both covers gaps in the SQL standard where details are unclear or left to the implementation and covers our own Google specific extensions. Before defining our own SQL dialect, we did consider other options. Using the front end of an open-source SQL processing system, such as Postgres, would have accomplished several of the same goals including filling the gaps left by the SQL standard and ensuring uniform syntax/semantics across our systems. We ruled out this approach because internal systems at Google make heavy use of Protocol Buffers in their applications, and we wanted Protocol Buffer message and enum types to be first class types in our SQL language"
The tooling around pulling in protobuf at compile time is a huge pain in the ass. I've been spending an hour or so every night over past few weeks searching issues to figure out why things aren't working. This is a great reference point for me, thanks!
[1]: https://github.com/johnbellone/cache-service