(author here, the blog is under an alias) those are serialization formats, they don't allow for defining the physical representation of the data. for instance, you can't write a protobuf schema that de-serializes a .gif header.
What exactly do you mean by "defining the physical representation of the data"? Like, "here is a type called uint32, it is a 32 bit little endian unsigned number. Here is struct Foo, it has two uint32 fields". Like, that level of granularity?
Basically Katai struct but can also be used to serialize, not just deserialize?
(author here, the blog is under an alias) oh, interesting! i hadn't seen this one yet. i think this one would fall under the category "transpile-first languages" (like wuffs or spicy); i.e. it is useful for portable parsers but faces obstacles for generating ancillary tooling.
In my view, gRPC is the closest "binary" analog to OpenAPI, which is a transport-layer concern first and foremost with discoverability features.
If your problem space is "a way to assign meaning to arbitrary bitfields", I don't feel that OpenAPI is a very good comparison. Its purview is "representing web APIs in terms of structured JSON/XML payloads". That is, OpenAPI also deals in logical and not physical representation. (unless it specifies UTF-8 or something - it might, I don't know!)
you know, i think you're right. on reflection, a comparison to json schema[1] (rather than openapi) would be more appropriate, as mentioned be defanor in another comment.
Automatic generation of wireshark dissectors would be amazing.
I wrote a few in Lua, but it still felt second class citizen to built-in C dissectors, loading them was annoying and the UI to use them was too.
A good dissector will annotate fields with semantic meaning and group and nest them, giving wireshark the precise bit offset+length where a value came from.
Those are some different requirements than other parsers and reuse is therefore a bit difficult.
I was checking out Kaitai the other day (generate binary parsers from declarative structures) and noticed its tutorials section has great content on reverse-engineering binary formats with it: https://doc.kaitai.io. Might be interesting to anyone interested in OP's question.
(author here, the blog is under an alias) ASN.1 is in a similar category to protobuf: it doesn't allow definition of the physical representation of the data. (it does support multiple serialization profiles, but ultimately ASN.1 data can only be serialized in a few different ways.) you can't write an ASN.1 file to parse arbitrary binary formats found in the wild.
I think OpenAPI is a rather arbitrary and approximate description of models built upon HTTP with JSON, but that's constrained enough, and the problems solved are often similar enough, for such a solution to be useful. Actually maybe JSON schema would be more useful to consider here, since that is about formats, rather than interfaces or protocols.
With binary formats, on the other hand, there's neither data model nor usage constraints like that, and a generic tool to handle all the things would be just the Turing machine -- and then we have a bunch of languages for that. As for kaitai struct and similar attempts to handle arbitrary binary formats, I hear of those from time to time, and in some cases they seem to be useful, but they too should be either Turing-complete (and then it probably makes more sense to use general-purpose languages) or would fail to be useful occasionally.
This seems obviously to be the right answer to me.
Think about handling a ZIP file: you can't start at the beginning of the file and work forward from there. No; you have to start at/near the end of file and scan for an end-of-central-directory record based on a signature (it's not at a fixed offset due to a variable-length comment field). That then enables you to find the actual offset for the central directory itself; which in turns allow you to find the files themselves.
(author here) aye, this is a crucial point. any description language for binary formats quickly approaches turing complete, and i think this is what leads to so many of the "transpile-first parser languages". i'm impressed how well kaitai has managed to wrangle that tendency towards turing-completeness; they do have an embedded expression language for attribute binding, and they do have 'repeat' and 'switch' constructs, but the language is astoundingly declarative.
i think something very similar to kaitai could work if the tools to work with the files programmatically were ergonomic and available in many different languages. e.g., if the kaitai compiler were a native compiled library with lots of packaged FFI wrappers for many languages.
Well, nothing stops Turing-complete languages from being declarative: lambda calculus is both, for instance, as well as some of the languages based on it (in particular parser combinators in Haskell and similar languages are neat and flexible).
And there's indeed a practice of embedding some languages into others, though I think it ends up quite awkward to use, and in this case potential limitations (actually I don't know if kaitai is Turing-complete) may be undesirable. I imagine in many cases it may be simpler to ship, say, a parser/decoder (and/or printer/encoder) library providing a C API that is easy to call via FFI, rather than a description in a less common language, to be used with a particular library. But then again, probably that'd have its uses too.
I don't know OpenAPI and only did a quick read. Does it support any formats other than JSON and XML? If not, then it hardly seems like an appropriate analog to what you're asking for.
Can it support s-expr based formats? Plain text keyword:value pairs? Any arbitrary input? Because that's what you're asking for for "binary formats". Which is a wide-open field. To have a universal system for that you will need something closer to the parser description formats that you have already rejected. Just like you'd have to do to properly support any arbitrary plain text format instead of just JSON and XML.
(author here) i did not know about the reflection protocol. maybe this could be a model for how kaitai struct could avoid the "centralized bottleneck" they have now in terms of ecosystem growth.
Tangential to the rest of the conversation, but I often use/write binary templates [0] for 010 Editor [1]. The GUI-driven approach to navigating structures (such as PE headers) can be very useful.
I wish there was a similar tool that allowed me to simply highlight and bookmark blocks of binary data when I'm dealing with an unknown file format, such as a firmware dump for something exotic and/or proprietary.
> both of these tools are only interested in allowing the author to model the logical representation of the data, not the physical. they are useful if you have data and want to automatically convert that into a file format or protocol, but not useful if you want to describe an existing file format or protocol.
OpenAPI isn't necessarily great at this either. If your JSON API wasn't designed with OpenAPI in mind (or in a way that's similar to typical OpenAPI designs), it might not be possible to write an OpenAPI spec that describes the format well. But "type: object" is valid, so it's always possible to write a "spec".
I think the question ("why is there no schema language for binary formats?") is valid, but that OpenAPI isn't as encompassing as it's laid out to be.
Yes and that's a fundamentally different problem; I think this is because there is always some kind of state in a protocol. SOAP has this problem, and also YANG[1] and tools that try to replace SNMP. You can never abstract away the state that the consumer and producer of the protocol has. There are lots of tools to disect, debug, resend and parse network protocols like IP and TCP but in those protocols we know there is a state built in that can not be broken, so we do not build OpenApi for them.
OpenApi specifications that need three or four requests to reason with are a lot harder to work with than ones that are only one request, and these are different beast they become protocols. In my personal experience the binary specification often are protocols with a lot more implicit state in them, e.g. binary diff of in memory struct etc. You quickly lose the usefullness of Swagger the more state you have.
That said there are lots of good example of binary protocols that try to be stateless.
Conclussion: This blog post need to be scoped a bit better, what kind of binary communications do you want an OpenApi for and how is that different from what exists today?
As soon as you have to encode the state of the application in the specification of the protocol (like TCP above) things get really complicated.
OP might be conflating OpenAPI [1] which specifies HTTP endpoints of APIs with JSON Schema [2] which specifies data schemas of JSON documents. OpenAPI specs often use the application/json media type with schemas specified using JSON Schema, but they can also use other media types including binary formats. What OP is really asking is: Why isn’t there {JSON Schema,XML Schema,.proto files} for arbitrary binary formats?
Kaitai [3] mentioned by OP and sibling comments looks interesting, don’t think I’ve seen something like it before.
Can it parse a binary file/stream containing heterogeneous binary structures? Documentation seems not that clear regarding this.
Also the structure definition seems to needs compiling, what if there is change of structure at runtime? Can it handle it?
Is it possible to for an application to have editable structures and KaiTai to parse them without needing to recompile the whole application (I'm speaking here in a java context).
I did research into session types to try and solve this exact problem. However the theory still new and implementations are lacking. Again session types encode the logic, you still need some sort of structure/content language to describe the messages. Session types should be able to encode complex protocols and combined with kaitai, maybe something can emerge.
The author sort of answers the title question without appearing to realize it.
The answer is: "The things that exist are good enough for what people are usually trying to do"
It's unclear what the author's real use case is.
The reason proto/etc suffice for most people is that they do not care about the physical format, and control it.
As a result, there is no need to define it explicitly or separately, and mostly people don't care to.
The folks that need to control the physical format are often those reading other people's formats. In a lot of cases, those producing the format don't care if you can read it or not :). They are often part of some ecosystem (say, photoshop, or whatever).
As a result, they wouldn't use an "openapi for binary formats" even if it existed.
For lots of other kinds of standardized formats to read (video, image, etc), these days the processing is harder than the parsing. Not that it's worth nothing, mind you, but that i doubt you would find a lot of people care enough to think it's worth it.
You also start to get into fairly different kinds of formats quickly (IE network protocol formats vs file formats), and those differences often lend themselves to different abstractions (IE you often want to be able to stream decode network formats, but may care less about stream decoding file formats, etc).
The market that remains to be served well here seems pretty small.
Certainly there are folks who want to control the physical format for some reason (whether they need to or not), etc.
But again, that feels like a small fraction of cases.
As a result, nobody felt enough of a need for OpenAPI for binary formats to push very hard on it.
(and attempts at doing so in the past have failed for lack of value)
Can KaiTai parse a binary file/stream containing heterogeneous binary structures? Documentation seems not that clear regarding this.
Also the structure defenition seems to needs compiling, what if there is change of structure at runtime? Can it handle it?
Is it possible to for an application to have editable structures and KaiTai to parse them without needing to recompile the whole application (I'm speaking here in a java context).
54 comments
[ 0.65 ms ] story [ 109 ms ] threadBasically Katai struct but can also be used to serialize, not just deserialize?
If your problem space is "a way to assign meaning to arbitrary bitfields", I don't feel that OpenAPI is a very good comparison. Its purview is "representing web APIs in terms of structured JSON/XML payloads". That is, OpenAPI also deals in logical and not physical representation. (unless it specifies UTF-8 or something - it might, I don't know!)
1: https://json-schema.org/
Well, it and every other binary RPC technology...
I wrote a few in Lua, but it still felt second class citizen to built-in C dissectors, loading them was annoying and the UI to use them was too.
A good dissector will annotate fields with semantic meaning and group and nest them, giving wireshark the precise bit offset+length where a value came from.
Those are some different requirements than other parsers and reuse is therefore a bit difficult.
Happened to just see this article when reading Kaitai docs: https://www.incibe-cert.es/en/blog/understanding-industrial-...
I was checking out Kaitai the other day (generate binary parsers from declarative structures) and noticed its tutorials section has great content on reverse-engineering binary formats with it: https://doc.kaitai.io. Might be interesting to anyone interested in OP's question.
I like the first blog link that reverses the binary format of one of those interactive anime novels from an .exe: https://hackernoon.com/reverse-engineering-visual-novels-101...
Really cool walkthrough of how to do such a thing regardless of whether you use Kaitai to do it.
Welcome to hell, abandon all hope, etc.
[0] https://en.wikipedia.org/wiki/Encoding_Control_Notation
With binary formats, on the other hand, there's neither data model nor usage constraints like that, and a generic tool to handle all the things would be just the Turing machine -- and then we have a bunch of languages for that. As for kaitai struct and similar attempts to handle arbitrary binary formats, I hear of those from time to time, and in some cases they seem to be useful, but they too should be either Turing-complete (and then it probably makes more sense to use general-purpose languages) or would fail to be useful occasionally.
Think about handling a ZIP file: you can't start at the beginning of the file and work forward from there. No; you have to start at/near the end of file and scan for an end-of-central-directory record based on a signature (it's not at a fixed offset due to a variable-length comment field). That then enables you to find the actual offset for the central directory itself; which in turns allow you to find the files themselves.
i think something very similar to kaitai could work if the tools to work with the files programmatically were ergonomic and available in many different languages. e.g., if the kaitai compiler were a native compiled library with lots of packaged FFI wrappers for many languages.
And there's indeed a practice of embedding some languages into others, though I think it ends up quite awkward to use, and in this case potential limitations (actually I don't know if kaitai is Turing-complete) may be undesirable. I imagine in many cases it may be simpler to ship, say, a parser/decoder (and/or printer/encoder) library providing a C API that is easy to call via FFI, rather than a description in a less common language, to be used with a particular library. But then again, probably that'd have its uses too.
Can it support s-expr based formats? Plain text keyword:value pairs? Any arbitrary input? Because that's what you're asking for for "binary formats". Which is a wide-open field. To have a universal system for that you will need something closer to the parser description formats that you have already rejected. Just like you'd have to do to properly support any arbitrary plain text format instead of just JSON and XML.
[1] https://github.com/grpc/grpc/blob/master/doc/server-reflecti...
[2] https://github.com/grpc-swagger/grpc-swagger
https://github.com/fullstorydev/grpcui
I wish there was a similar tool that allowed me to simply highlight and bookmark blocks of binary data when I'm dealing with an unknown file format, such as a firmware dump for something exotic and/or proprietary.
[0] https://www.sweetscape.com/010editor/templates.html [1] https://www.sweetscape.com/010editor/
OpenAPI isn't necessarily great at this either. If your JSON API wasn't designed with OpenAPI in mind (or in a way that's similar to typical OpenAPI designs), it might not be possible to write an OpenAPI spec that describes the format well. But "type: object" is valid, so it's always possible to write a "spec".
I think the question ("why is there no schema language for binary formats?") is valid, but that OpenAPI isn't as encompassing as it's laid out to be.
OpenApi specifications that need three or four requests to reason with are a lot harder to work with than ones that are only one request, and these are different beast they become protocols. In my personal experience the binary specification often are protocols with a lot more implicit state in them, e.g. binary diff of in memory struct etc. You quickly lose the usefullness of Swagger the more state you have.
That said there are lots of good example of binary protocols that try to be stateless.
Conclussion: This blog post need to be scoped a bit better, what kind of binary communications do you want an OpenApi for and how is that different from what exists today?
As soon as you have to encode the state of the application in the specification of the protocol (like TCP above) things get really complicated.
[1] https://en.wikipedia.org/wiki/YANG
Kaitai [3] mentioned by OP and sibling comments looks interesting, don’t think I’ve seen something like it before.
[1]: https://swagger.io/specification/
[2]: https://json-schema.org/
[3]: https://kaitai.io/
katai allows you to declaratively define a binary format bit for bit - they even have an IDE
Is it possible to for an application to have editable structures and KaiTai to parse them without needing to recompile the whole application (I'm speaking here in a java context).
The answer is: "The things that exist are good enough for what people are usually trying to do"
It's unclear what the author's real use case is.
The reason proto/etc suffice for most people is that they do not care about the physical format, and control it. As a result, there is no need to define it explicitly or separately, and mostly people don't care to.
The folks that need to control the physical format are often those reading other people's formats. In a lot of cases, those producing the format don't care if you can read it or not :). They are often part of some ecosystem (say, photoshop, or whatever). As a result, they wouldn't use an "openapi for binary formats" even if it existed.
For lots of other kinds of standardized formats to read (video, image, etc), these days the processing is harder than the parsing. Not that it's worth nothing, mind you, but that i doubt you would find a lot of people care enough to think it's worth it.
You also start to get into fairly different kinds of formats quickly (IE network protocol formats vs file formats), and those differences often lend themselves to different abstractions (IE you often want to be able to stream decode network formats, but may care less about stream decoding file formats, etc).
The market that remains to be served well here seems pretty small. Certainly there are folks who want to control the physical format for some reason (whether they need to or not), etc.
But again, that feels like a small fraction of cases.
As a result, nobody felt enough of a need for OpenAPI for binary formats to push very hard on it. (and attempts at doing so in the past have failed for lack of value)
SWIG is built on top of C .h files, which are the _real_ Swagger for binary formats.
Is it possible to for an application to have editable structures and KaiTai to parse them without needing to recompile the whole application (I'm speaking here in a java context).