I've posted this once before but it failed to get traction. I'm reposting this with the permission of HN admins.
In a nutshell: This open-source project provides a framework for fuzzing cryptographic libraries. It can find both memory bugs and implementation errors. So far it has found about 35 bugs of varying severity in the stable releases of major cryptographic libraries. The fuzzers run continually on Google's OSS-Fuzz platform which helps detect bugs before they end up in stable releases.
Thanks for posting this. It looks like some really great tooling. I tried fuzzing the Argon2 library a while ago and very quickly hit some of the issues you appear to have addressed.
I use the high level concept pretty regularly in my day-to-day as a security consultant specializing in cryptography, and this project is a fantastic way to democratize the use of differential fuzzing. The only negative thought I have about this is that I didn't think of it first!
In the Bitcoin project we found and reported several bugs in OpenSSL based on similar harsnesses that compared our code with similar functions in OpenSSL (where they exist).
It's a pretty powerful technique at least in problem domains where fully specified behavior is a realistic expectation.
Mutation testing-- where you break the code and confirm that your tests fail-- is also pretty powerful in those same domains.
(in particular: mutate the code, if the tests pass, test random cases differential with the original, if any discrepancies are found then you just found a missing test case.)
dumb question: but why is it so hard to create very-secure crypto libraries? i recall NASA's code used during the Apollo missions had like an ungodly low error rate, so surely it's feasible in practice? admittedly, comparing anything to NASA standards is a very high bar to set, but considering the stakes involved with cryptography, you'd imagine a similar scrutiny?
I would add that in the beginning, a number of crypto libraries (I'm looking at you in particular, openSSL) made a design decision to support a huge variety of algorithms, options, modes of operation etc. This makes it very difficult to test the full functional surface of the software and means you just have to write a very large amount of code.
Some of the modern libraries take a different approach where they deliberately limit the functionality to a few select things that can be implemented easily (thinking nacl, libsodium, boringSSL etc here) and which guide the user towards sensible defaults that work rather than having huge numbers of levers to tweak, only some of which arrive at a secure and supported solution.
The government has very deep pockets. And working for the biggest project of the century probably attracted decent talent, whereas there's no glory in writing supporting libs - you either did it right and nobody notices, or you messed up and get blame.
12 comments
[ 2.2 ms ] story [ 38.6 ms ] threadIn a nutshell: This open-source project provides a framework for fuzzing cryptographic libraries. It can find both memory bugs and implementation errors. So far it has found about 35 bugs of varying severity in the stable releases of major cryptographic libraries. The fuzzers run continually on Google's OSS-Fuzz platform which helps detect bugs before they end up in stable releases.
The article describes the technical details of the software. Here you can find a summary of the bugs it has found: https://github.com/guidovranken/cryptofuzz#hall-of-fame
1) Compare two programs
2) Generate two inputs for one program which should produce the same output
3) generate two inputs for one program where there is a simple relationship between the outputs.
In all cases, it has been a great way of testing, and has shuck out some incredibly subtle bugs I don't think we would have found any other way.
It's a pretty powerful technique at least in problem domains where fully specified behavior is a realistic expectation.
Mutation testing-- where you break the code and confirm that your tests fail-- is also pretty powerful in those same domains.
Crypto libs (or any libs) typically can be used in a vast number of ways with completely untrustworthy input.
Some of the modern libraries take a different approach where they deliberately limit the functionality to a few select things that can be implemented easily (thinking nacl, libsodium, boringSSL etc here) and which guide the user towards sensible defaults that work rather than having huge numbers of levers to tweak, only some of which arrive at a secure and supported solution.