In short: I wanted to talk a bit about ASN.1, a bit about D, and a bit about the compiler itself, but couldn't think of any real cohesive format.
So I threw a bunch of semi-related ramblings together and I'm daring to call it a blog post.
Sorry in advance since I will admit it's not the greatest quality, but it's really not easy to talk about so much with such brevity (especially since I've already forgot a ton of stuff I wanted to talk about more deeply :( )
I salute your for deep dive into this. History would have it that ASN.1 was already there as both an IDL and serialization format when HTTPS certs were defined. If it were today, would it be the same or would we end up with protobuf or thrift or similar?
I worked on a Swift ASN.1 compiler [1] a while back (not swift-asn1, mine used Codable). I saved myself some time by using the Heimdal JSON compiler, which can transform ASN.1 into a much more parseable JSON AST.
(i worked with asn1c (not sure which fork) and had to hack in custom allocator and 64bit support. i shiver every time something needs attention in there)
I really love D, it's one of my favorite languages. I've started implementing a vim-like text editor in it from scratch (using only Raylib as a dependency) and was surprised how far I was able to get and how good my test coverage was for it. My personal favorite features of D:
* unit tests anywhere, so I usually write my methods/functions with unit tests following them immediately
* blocks like version(unittest) {} makes it easy to exclude/include things that should only be compiled for testing
* enums, unions, asserts, contract programming are all great
I would say I didn't have to learn D much. Whatever I wanted to do with it, I would find in its docs or asked ChatGPT and there would always be a very nice way to do things.
The only goal of such ridiculous standards is to act as a form of vendor lock-in for vendors implementing those standards; the vendors get to say to governments that it is a standard and the sellers of the standards also get some money.
Any system designed picking such standards is basically betraying their client.
I think, if you want to annoy these people maximally, you should write an annotated version of the standard in a mathematical formal language.
I read the table constraints, which try to do something simple, but it's written in the most convoluted way possible.
I think I considered ASN.1 for a system once, but rejected it because of more modern technically superior system.
If the parser for something like ASN.1 doesn't fit in 80 lines of Haskell, perhaps you just shouldn't use it.
I don't know who these assholes are that say "Sure, let's make things slow and buggy, since we all hail Satan after all".
For those of you who missed this, there was a very interesting thing that happened in the growth of the internet.
At the time people were evolving the protocols through the IETF. So all the things that you rely on now - for the most part - just came into being. One day there was email. There was ftp. There was TCP. There were the Van Jacobson TCP mods.
At this time corporate types paid no attention to the internet. Academic types and the IETF were from what I saw the main developers.
Then one day the corporate world realized they might make money. But the development process of the protocols was incomprehensible (and incompatible) with the corporate culture. TCP was clearly a mess, all these protocols like DNS were a mess. From the corporate perspective.
Whether ASN.1 was a product of that war or just a product of the corporate mentality, it serves as a powerful instance of the what the corporate world looks like vs the academic world looks like. You can find the wreckage from the war littered around. If you see and X.something protocol it could well be one of the relics. There were a few X.things that were adopted and useful, but were others that would haunt your dreams.
Although this is ancient history, and pretty much now told from the corporate perspective, it suggests to us that the corporate process for thinking is not as effective as the alternative - the IETF and Academic.
One is a sort of recipe culture. You write a recipe, everyone follows it and you are happy.
The other is a sort of functional culture. If you can make bread and eat it you are happy. When the bread doesn't taste good you fix it.
Given the kind of bread that is commonly available in the US now, we can draw some conclusions about recipe thinking, recipe culture, corporate culture etc. One could even extend this paradigm of thinking to new things like AI. Or not.
Normally, you could say when implementing some standard that you get 80% of the functionality with 20% of the planned time. But with ASN.1 the remaining 20% could take the rest of your life.
Ack. I wrote an ASN.1 compiler in Java in the 90s. Mostly just to make sure I understood how it and BER/DER were used in X.509. I think the BER interpretation bits are still being used somewhere
I'm sorry you had to waste a year of your life.
There are few things I dislike more in the computing world than ASN.1/BER. It seems to encourage over-specification and is suboptimal for loosely coupled systems.
Looking at the comments, it seem that many people hate ASN.1 and many people like ASN.1 (I am the latter). (I consider BER to be messy and overly complicated, but DER is much better.) (I wrote a implementation of DER (decoding and encoding), although not the schema format (which I have never needed).)
I have also had to work with this in many contexts... Deeply embedded systems with no parsers available and where no "proper" ones would fit. So i have hand written but basic parsing and generation a few times.
Oh and there is also non compliant implementations. E.g. some passports (yes the passports with chip use tons of ASN.1) even have incorrect including of big integers (supposed to be the minimum two complement, as I recall some passports used a fixed non-complement format yanked into the 0x02 INTEGER type... Some libraries have special non-compliant parsing modes to deal with it).
Every time I have ever had the displeasure of looking at an X.whatever spec, I always end up coming away with the same conclusion.
Somehow, despite these specifications being 90% metadata by weight, they seem to consistently forget the part of the spec that lets you actually know what something is. and that part is just left up to context.
I could well be missing something, but a majority of the time it feels to me like they set out to make a database schema, and accidentally wrote the sqlite file format spec instead.
Like thanks, its nice that I can parse this into a data structure :). It would be nicer, however if doing so gave me any idea of what I can do with the data I've parsed.
Though to be fair I feel the same way about the entire concept of XML schemas. The fact that you theoretically can validate an xml document against a schema is honestly completely useless. If I am parsing XML, its because my code already knows what information it needs from the XML document, and really it should also know where that information is. I don't need a separate schema definition to tell me that! its already expressed!! In the part where I am parsing out the data I need!!!
The quality does definitely seem to vary depending on the spec.
As someone who's not very well-learned I found x.690 to be my personal gold standard for how a specification should be written - it left me with very few questions, and after a small period of getting used to its general use of language, didn't struggle too much with it.
x.680 (the 2003 revision more so than the 2021 one) on the other hand has definitely given me a few struggles with general comprehension, and leaving me some similar feelings. I believe that to be an issue of how they've structured the spec (and how they've spread out some important info across multiple specs).
According to ASN.1 Wikipedia entry, most of the tools supporting ASN.1 do the following:
1) parse the ASN.1 files,
2) generates the equivalent declaration in a programming language (like C or C++),
3) generates the encoding and decoding functions based on the previous declarations
All of these of exercise are apparently part of data engineering process or lifecycle [1].
Back in early 21st century Python is just another interpreted general purpose programming language alternative, not for web (PHP), not for command tool (TCL), not for system (C/C++), not for data wrangling (Perl), not for numerical (Matlab/Fortran), not for statistics (R).
D will probably follow similar trajectory of Python, but it really needs a special kind of killer application that will bring it to the fore.
I'm envisioning that real-time data streaming, processing and engineering can be D killer utility and defining moment that D is for data.
Very neat article. I too have spent countless hours (but not as many) hacking on an ASN.1 compiler, adding a subset of X.681, X.682, X.683 functionality to make it possible to -in a single codec invocation!- a whole certificate, with all its details like extensions and OtherName SANs and what not decoded recursively. So it's very nice to see a post about similar work!
ASN.1 really is quite awesome. It gets a lot of hate, but it's not deserved. ASN.1 is not just a syntax or set of encoding rules -- it's a type system, and a very powerful one at that.
There's a Turkish saying, "a human will [use] this, a human!", to signify that the thing is so abnormal/out-of-proportion that it doesn't seem to be made for people. The verb changes based on the context. If you had made too much food for example, the verb would be "eat". I think it's a great motto for design.
Remember the Game of Thrones quote, "the man who passes the sentence should swing the sword"? I think it should also be applied to specs. Anyone who comes up with a spec must be the first responsible party to develop a parser for it. The spec doesn't get ratified unless it comes with working parser code with unit tests.
That kind of requirement might actually improve specs.
Wow, I needed to parse just one small ASN.1 with one value (one signature), but I didn't know ASN.1 can have a specification (to generate parser from it). So I ended up untangling it myself, just for that specific 256 bits.
Still I think it's better to have overapecified format for security stuff, json and xml are just too vague and parsers are unpredictable.
I'm fascinated by ASN.1, I don't know why it appeals to me so much but I find working with it oddly fun. I've always wanted to find the time to write an ASN.1 compiler for Rust, because for some reason all of the Rust implementations I've seen end up just either being derive macros on Rust structs (so going the other direction), or even just providing a bunch of ASN.1 types and functions and expecting you to manually chain them together using nom.
As someone who extended the ASN.1 parser in the Netscape code base to handle PKCS#12 and at one point knew more about RSA standards and varios ASN.1 definitions than anyone should, I respect the blog owners diligence and masochism.
It would not be the project I would pick out to learn D.
The schema format is complicated, and the data format also is, but most of it is ignorable complexity, often making it simpler than it is.
I do not use the ASN.1 schema notation, since I have been able to use ASN.1 without it. (I wrote my own implementation of the data format (DER) in C.)
> On the one hand the sheer amount of possible encodings is daunting, but on the other hand it shows a certain flexbility that ASN.1 provides - you could even invent your own domain-specific encoding if needed.
It is true. My opinion is, of the standard encodings, DER is the only one that is any good (and is the one that I usually use (although most of my uses are not having to do with cryptography); X.509 (which is a common use of ASN.1) also uses DER), but I did make up my own encodings as well: DSER (between BER and DER), SDSER (like DSER but the length is encoded differently so that streaming is possible, in a (in my opinion) better way than CER does), and TER (a text format which is intended to be converted to DER).
> ANY DEFINED BY
I still effectively use it (as well as ANY), since I think it is very useful. (As I mentioned, I did not actually use the ASN.1 schema format, but I sometimes use what effectively works like ANY and ANY DEFINED BY.)
(I also sometimes use a nonstandard feature that I made up, which is called OBJECT IDENTIFIER RELATIVE TO, which is like OBJECT IDENTIFIER but it can be encoded more efficiently (using the RELATIVE OID type) if the value has the prefix that the schema expects it to have.)
31 comments
[ 2.3 ms ] story [ 50.2 ms ] threadSo I threw a bunch of semi-related ramblings together and I'm daring to call it a blog post.
Sorry in advance since I will admit it's not the greatest quality, but it's really not easy to talk about so much with such brevity (especially since I've already forgot a ton of stuff I wanted to talk about more deeply :( )
[1] https://github.com/PADL/ASN1Codable
[2] https://github.com/heimdal/heimdal/tree/master/lib/asn1
(i worked with asn1c (not sure which fork) and had to hack in custom allocator and 64bit support. i shiver every time something needs attention in there)
* unit tests anywhere, so I usually write my methods/functions with unit tests following them immediately
* blocks like version(unittest) {} makes it easy to exclude/include things that should only be compiled for testing
* enums, unions, asserts, contract programming are all great
I would say I didn't have to learn D much. Whatever I wanted to do with it, I would find in its docs or asked ChatGPT and there would always be a very nice way to do things.
Any system designed picking such standards is basically betraying their client.
I think, if you want to annoy these people maximally, you should write an annotated version of the standard in a mathematical formal language.
I read the table constraints, which try to do something simple, but it's written in the most convoluted way possible.
I think I considered ASN.1 for a system once, but rejected it because of more modern technically superior system.
If the parser for something like ASN.1 doesn't fit in 80 lines of Haskell, perhaps you just shouldn't use it.
I don't know who these assholes are that say "Sure, let's make things slow and buggy, since we all hail Satan after all".
For those of you who missed this, there was a very interesting thing that happened in the growth of the internet.
At the time people were evolving the protocols through the IETF. So all the things that you rely on now - for the most part - just came into being. One day there was email. There was ftp. There was TCP. There were the Van Jacobson TCP mods.
At this time corporate types paid no attention to the internet. Academic types and the IETF were from what I saw the main developers.
Then one day the corporate world realized they might make money. But the development process of the protocols was incomprehensible (and incompatible) with the corporate culture. TCP was clearly a mess, all these protocols like DNS were a mess. From the corporate perspective.
So began the protocol wars https://en.wikipedia.org/wiki/Protocol_Wars.
Whether ASN.1 was a product of that war or just a product of the corporate mentality, it serves as a powerful instance of the what the corporate world looks like vs the academic world looks like. You can find the wreckage from the war littered around. If you see and X.something protocol it could well be one of the relics. There were a few X.things that were adopted and useful, but were others that would haunt your dreams.
Although this is ancient history, and pretty much now told from the corporate perspective, it suggests to us that the corporate process for thinking is not as effective as the alternative - the IETF and Academic.
One is a sort of recipe culture. You write a recipe, everyone follows it and you are happy. The other is a sort of functional culture. If you can make bread and eat it you are happy. When the bread doesn't taste good you fix it.
Given the kind of bread that is commonly available in the US now, we can draw some conclusions about recipe thinking, recipe culture, corporate culture etc. One could even extend this paradigm of thinking to new things like AI. Or not.
I'm sorry you had to waste a year of your life.
There are few things I dislike more in the computing world than ASN.1/BER. It seems to encourage over-specification and is suboptimal for loosely coupled systems.
But it looks like you had a decent time...
I think if I were more seasoned with ASN.1 itself, as well as with compiler development it'd have been a smoother experience.
Though I definitely still could've chosen more higher value things to do throughout the last year :D
Oh and there is also non compliant implementations. E.g. some passports (yes the passports with chip use tons of ASN.1) even have incorrect including of big integers (supposed to be the minimum two complement, as I recall some passports used a fixed non-complement format yanked into the 0x02 INTEGER type... Some libraries have special non-compliant parsing modes to deal with it).
Somehow, despite these specifications being 90% metadata by weight, they seem to consistently forget the part of the spec that lets you actually know what something is. and that part is just left up to context.
I could well be missing something, but a majority of the time it feels to me like they set out to make a database schema, and accidentally wrote the sqlite file format spec instead.
Like thanks, its nice that I can parse this into a data structure :). It would be nicer, however if doing so gave me any idea of what I can do with the data I've parsed.
Though to be fair I feel the same way about the entire concept of XML schemas. The fact that you theoretically can validate an xml document against a schema is honestly completely useless. If I am parsing XML, its because my code already knows what information it needs from the XML document, and really it should also know where that information is. I don't need a separate schema definition to tell me that! its already expressed!! In the part where I am parsing out the data I need!!!
As someone who's not very well-learned I found x.690 to be my personal gold standard for how a specification should be written - it left me with very few questions, and after a small period of getting used to its general use of language, didn't struggle too much with it.
x.680 (the 2003 revision more so than the 2021 one) on the other hand has definitely given me a few struggles with general comprehension, and leaving me some similar feelings. I believe that to be an issue of how they've structured the spec (and how they've spread out some important info across multiple specs).
1) parse the ASN.1 files, 2) generates the equivalent declaration in a programming language (like C or C++), 3) generates the encoding and decoding functions based on the previous declarations
All of these of exercise are apparently part of data engineering process or lifecycle [1].
Back in early 21st century Python is just another interpreted general purpose programming language alternative, not for web (PHP), not for command tool (TCL), not for system (C/C++), not for data wrangling (Perl), not for numerical (Matlab/Fortran), not for statistics (R).
D will probably follow similar trajectory of Python, but it really needs a special kind of killer application that will bring it to the fore.
I'm envisioning that real-time data streaming, processing and engineering can be D killer utility and defining moment that D is for data.
[1] Fundamentals of Data Engineering:
https://www.oreilly.com/library/view/fundamentals-of-data/97...
ASN.1 really is quite awesome. It gets a lot of hate, but it's not deserved. ASN.1 is not just a syntax or set of encoding rules -- it's a type system, and a very powerful one at that.
Remember the Game of Thrones quote, "the man who passes the sentence should swing the sword"? I think it should also be applied to specs. Anyone who comes up with a spec must be the first responsible party to develop a parser for it. The spec doesn't get ratified unless it comes with working parser code with unit tests.
That kind of requirement might actually improve specs.
Still I think it's better to have overapecified format for security stuff, json and xml are just too vague and parsers are unpredictable.
It would not be the project I would pick out to learn D.
Sounds like you have some fun potential war stories in that case.
huh.
Must be on vacation today.
The schema format is complicated, and the data format also is, but most of it is ignorable complexity, often making it simpler than it is.
I do not use the ASN.1 schema notation, since I have been able to use ASN.1 without it. (I wrote my own implementation of the data format (DER) in C.)
> On the one hand the sheer amount of possible encodings is daunting, but on the other hand it shows a certain flexbility that ASN.1 provides - you could even invent your own domain-specific encoding if needed.
It is true. My opinion is, of the standard encodings, DER is the only one that is any good (and is the one that I usually use (although most of my uses are not having to do with cryptography); X.509 (which is a common use of ASN.1) also uses DER), but I did make up my own encodings as well: DSER (between BER and DER), SDSER (like DSER but the length is encoded differently so that streaming is possible, in a (in my opinion) better way than CER does), and TER (a text format which is intended to be converted to DER).
> ANY DEFINED BY
I still effectively use it (as well as ANY), since I think it is very useful. (As I mentioned, I did not actually use the ASN.1 schema format, but I sometimes use what effectively works like ANY and ANY DEFINED BY.)
(I also sometimes use a nonstandard feature that I made up, which is called OBJECT IDENTIFIER RELATIVE TO, which is like OBJECT IDENTIFIER but it can be encoded more efficiently (using the RELATIVE OID type) if the value has the prefix that the schema expects it to have.)