> Finally, taking an OpenSSL public API and attempting to trace the implementation to see how it is implemented has become an exercise in self-flagellation. Being able to read the source to understand how something works is important both as part of self-improvement in software engineering, but also because as sophisticated consumers there are inevitably things about how an implementation works that aren’t documented, and reading the source gives you ground truth. The number of indirect calls, optional paths, #ifdef, and other obstacles to comprehension is astounding. We cannot overstate the extent to which just reading the OpenSSL source code has become miserable — in a way that both wasn’t true previously, and isn’t true in LibreSSL, BoringSSL, or AWS-LC.
OpenSSL code was not pleasant or easy to read even in v1 though and figuring out what calls into where under which circumstances when e.g. many optimized implementations exist (or will exist, once the many huge perl scripts have generated them) was always a headache with only the code itself. I haven't done this since 3.0 but if it regressed so hard on this as well then it has to be really quite bad.
I published a book about SSL a while back - my original plan for the book was going to be to work through the OpenSSL source code and relate each piece back to the relevant specifications, step by step. I found that the OpenSSL source code was so complex that I would have spent a lot more time discussing the intricacies of the C code itself than the cryptographic algorithms they were implementing - so much so that it made more sense to just write my own SSL implementation and walk through that.
In fairness to OpenSSL, though, I can see how and why it got so complex: they're trying to be all things to all people in a backwards compatible way. At the time, OpenSSL still had support for SSLv2, an albatross that LibreSSL doesn't have around its neck.
> Later, moving public key parsing to our own Rust code made end-to-end X.509 path validation 60% faster — just improving key loading led to a 60% end-to-end improvement, that’s how extreme the overhead of key parsing in OpenSSL was.
> The fact that we are able to achieve better performance doing our own parsing makes clear that doing better is practical. And indeed, our performance is not a result of clever SIMD micro-optimizations, it’s the result of doing simple things that work: we avoid copies, allocations, hash tables, indirect calls, and locks — none of which should be required for parsing basic DER structures.
I was involved in the design/implementation of the X.509 path validation library that PyCA cryptography now uses, and it was nuts to see how much performance was left on the ground by OpenSSL. We went into the design prioritizing ergonomics and safety, and left with a path validation implementation that's both faster and more conformant[1] than what PyCA would have gotten had it bound to OpenSSL's APIs instead.
I'm glad that they're considering getting rid of OpenSSL as a hard dependency. I've built parts of pyca/cryptography with OpenSSL replaced or stripped out for better debugging. OpenSSL's errors just suck tremendously. It shouldn't be tremendously difficult for them to do it for the entire package.
Though I'd also love to see parts of pyca/cryptography being usable outside of the context of Python, like the X.509 path validation mentioned in other comments here.
By the way, pyca/cryptography is a really excellent cryptography library, and I have confidence that they're making the right decisions here. The python-level APIs are well thought-out and well documented. I've made a few minor contributions myself and it was a pleasant experience.
> I set out to remove deprecated calls to SHA256_xxx to replace them with the EVP_Digestxxx equivalent in my code. However it seems the EVP code is slow. So I did a quick test (test case B vs C below), and it is indeed about 5x slower.
Once upon a time, OpenSSL was the place to go for crypto
primitives that were hardware specific and well optimized, and you would pay the price of using a nasty API. Now it’s an even nastier API and it’s not even fast anymore?
SHA256 is almost the prototype of a pure function. There should not be concepts like “EVP”. Output sizes should be static. Failure should be entirely impossible unless I opt in to using an async interface for an async accelerator. The only complexity should be the hidden part that selects the best synchronous implementation.
It is honestly surprising that OpenSSL has been the standard for so long given how difficult it is to work with. I think moving the backend to Rust is probably the right move for long term stability.
Since that Haproxy has effectively abandoned OpenSSL in favor or AWS-LC. Packages Re still built with both, but AWS-LC is clearly the path forward for them.
I wrote about OpenSSL's performance regressions in the December issue of Feisty Duck's cryptography newsletter [1]. In addition to Alex's and Paul's talk on Python cryptography, at the recent OpenSSL conference there have been several other talks worth watching:
Note: be careful with these recorded talks as they have a piercing violin sound at the beginning that's much louder than the rest. I've had to resort to muting the first couple of seconds of every talk.
The go standard library has an implementation of ed25519 although I did not find ed448 it also has some NIST curves. There are a few libraries that do ed448 like one from cloudflare.
> OpenSSL allowed replacing any algorithm at any point in program execution
Just this is completely nuts. What in the world is the usecase for this? Fanatics of hotpatching systems for zero-downtime-ever? No wonder the performance is crippled by locks left and right, it's pure recipe for disaster.
I’ve done this kind of thing before: I wrote a 3D game engine that could switch between DirectX and OpenGL at runtime and also switch GPUs dynamically.
At some point the architecture becomes a toy to play with, like a train set. Trying to push the abstractions as far as they can possibly go within the bounds of the parent language is a fun challenge to tackle.
Of course, this is pointless and even counterproductive. I only did this decades ago when I was a junior developer. With more experience I now deeply appreciate simple, straightforward, to-the-point code.
Honestly, I rarely ever use openssl these days unless I must. Now, I go for Botan, or cryptography, or monocypher, any number of cryptographic library alternatives that are designed well and are really fast.
18 comments
[ 0.18 ms ] story [ 40.8 ms ] threadOpenSSL code was not pleasant or easy to read even in v1 though and figuring out what calls into where under which circumstances when e.g. many optimized implementations exist (or will exist, once the many huge perl scripts have generated them) was always a headache with only the code itself. I haven't done this since 3.0 but if it regressed so hard on this as well then it has to be really quite bad.
I published a book about SSL a while back - my original plan for the book was going to be to work through the OpenSSL source code and relate each piece back to the relevant specifications, step by step. I found that the OpenSSL source code was so complex that I would have spent a lot more time discussing the intricacies of the C code itself than the cryptographic algorithms they were implementing - so much so that it made more sense to just write my own SSL implementation and walk through that.
In fairness to OpenSSL, though, I can see how and why it got so complex: they're trying to be all things to all people in a backwards compatible way. At the time, OpenSSL still had support for SSLv2, an albatross that LibreSSL doesn't have around its neck.
> Later, moving public key parsing to our own Rust code made end-to-end X.509 path validation 60% faster — just improving key loading led to a 60% end-to-end improvement, that’s how extreme the overhead of key parsing in OpenSSL was.
> The fact that we are able to achieve better performance doing our own parsing makes clear that doing better is practical. And indeed, our performance is not a result of clever SIMD micro-optimizations, it’s the result of doing simple things that work: we avoid copies, allocations, hash tables, indirect calls, and locks — none of which should be required for parsing basic DER structures.
I was involved in the design/implementation of the X.509 path validation library that PyCA cryptography now uses, and it was nuts to see how much performance was left on the ground by OpenSSL. We went into the design prioritizing ergonomics and safety, and left with a path validation implementation that's both faster and more conformant[1] than what PyCA would have gotten had it bound to OpenSSL's APIs instead.
[1]: https://x509-limbo.com
Though I'd also love to see parts of pyca/cryptography being usable outside of the context of Python, like the X.509 path validation mentioned in other comments here.
And my personal "new OpenSSL APIs suck" anecdote: https://github.com/openssl/openssl/issues/19612 (not my gh issue but I ran into the exact same thing myself)
> I set out to remove deprecated calls to SHA256_xxx to replace them with the EVP_Digestxxx equivalent in my code. However it seems the EVP code is slow. So I did a quick test (test case B vs C below), and it is indeed about 5x slower.
Once upon a time, OpenSSL was the place to go for crypto primitives that were hardware specific and well optimized, and you would pay the price of using a nasty API. Now it’s an even nastier API and it’s not even fast anymore?
SHA256 is almost the prototype of a pure function. There should not be concepts like “EVP”. Output sizes should be static. Failure should be entirely impossible unless I opt in to using an async interface for an async accelerator. The only complexity should be the hidden part that selects the best synchronous implementation.
Since that Haproxy has effectively abandoned OpenSSL in favor or AWS-LC. Packages Re still built with both, but AWS-LC is clearly the path forward for them.
https://mta.openssl.org/pipermail/openssl-users/2020-July/01...
- William Bellingrath, from Juniper Networks, benchmarked versions from 1.1.1 to 3.4.x https://www.youtube.com/watch?v=b01y5FDx-ao
- Tomáš Mráz wrote about how to get better performance, which, in turn, explains why it's bad by default: https://www.youtube.com/watch?v=Cv-43gJJFIs
- Martin Schmatz from IBM presented about their _very detailed_ study of post-quantum cipher suite performance https://www.youtube.com/watch?v=69gUVhOEaVM
Note: be careful with these recorded talks as they have a piercing violin sound at the beginning that's much louder than the rest. I've had to resort to muting the first couple of seconds of every talk.
[1] https://www.feistyduck.com/newsletter/issue_132_openssl_perf...
These are requirements for my current work, and OpenSSL 3+ was the only crypto library that delivered.
Just this is completely nuts. What in the world is the usecase for this? Fanatics of hotpatching systems for zero-downtime-ever? No wonder the performance is crippled by locks left and right, it's pure recipe for disaster.
At some point the architecture becomes a toy to play with, like a train set. Trying to push the abstractions as far as they can possibly go within the bounds of the parent language is a fun challenge to tackle.
Of course, this is pointless and even counterproductive. I only did this decades ago when I was a junior developer. With more experience I now deeply appreciate simple, straightforward, to-the-point code.
I tell juniors: “Just do the thing.”
WolfSSL... five
We are still suffering from that mistake, and LibreSSL is well-maintained and easier to migrate to than it ever was.
What the hell are we waiting for?
Is nobody at Debian, Fedora or Ubuntu able to step forward and set the direction?