OSS-Fuzz is an amazing program. The awards are really generous (in my experience), but more importantly, continuous fuzzing enabled me to make major changes to large codebases written in C without having to worry about introducing new security issues. If you maintain an open-source project, especially one written in an unsafe language, you should definitely check it out.
I’d go further and say that writing most software without fuzz testing is insane. Fuzz testing is one of those things they should teach in school. They’re a super useful technique - up there with TDD and it’s a tragedy they aren’t more wildly used.
Fuzzers are so good because they find so many bugs relative to programmer effort (lines of code). They’re some of the most efficient testing you can do. If I had to choose between a full test suite and a fuzzer, I’d choose the fuzzer.
I use fuzzers whenever I have a self contained “machine” in my code which should have well defined behaviour. For example, a b-tree. I write little custom fuzzers each time. The fuzzing code randomly mutates the data structure and keeps a list of the expected btree content. Then periodically I verify that the list and the btree agree on what should be contained inside the list. In the project I’m working on at the moment, I have about 6 different fuzzers sprinkled throughout my testing code. (Btree fuzzer, rope fuzzer, file serialisation fuzzer, a few crdt fuzzers, and so on).
Writing fuzzers is quite devastating for the ego. Usually the first time I point a fuzzer at my code, even when my code has a lot of tests, the fuzzer throws an assertion failure instantly. “Iteration 2 … the state doesn’t match what was expected”.
Getting a fuzzer running all night without finding any bugs is a balm for the soul.
13 comments
[ 3.5 ms ] story [ 11.6 ms ] threadhttps://www.code-intelligence.com/
[1] https://sos.dev/
https://bughunters.google.com/about/rules/5604090422493184/g...
https://clickhouse.com/blog/fuzzing-click-house
And I can confirm - writing software in C++ without fuzzing is insane.
We are finding and fixing memory-safety bugs in every third-party library that we use.
For example, there are bugs in RocksDB: https://github.com/facebook/rocksdb/issues?q=is%3Aissue+clic...
Here are some bugs in JeMalloc: https://github.com/jemalloc/jemalloc/issues/2222
Strangely, these bugs were found by the CI of ClickHouse, and not by any of the hundreds of other products using these libraries.
> Strangely, these bugs were found by the CI of ClickHouse, and not by any of the hundreds of other products using these libraries.
A lot of software projects that have "OSS Fuzz integration" have very naive or superficial fuzz harnesses that couldn't ever get thorough coverage.
If the code is never executed by the fuzzer, bugs won't be found by fuzzing.
In your case, fuzzing your application code, you likely end up executing code in those dependencies that their own fuzzers miss.
Fuzzers are so good because they find so many bugs relative to programmer effort (lines of code). They’re some of the most efficient testing you can do. If I had to choose between a full test suite and a fuzzer, I’d choose the fuzzer.
I use fuzzers whenever I have a self contained “machine” in my code which should have well defined behaviour. For example, a b-tree. I write little custom fuzzers each time. The fuzzing code randomly mutates the data structure and keeps a list of the expected btree content. Then periodically I verify that the list and the btree agree on what should be contained inside the list. In the project I’m working on at the moment, I have about 6 different fuzzers sprinkled throughout my testing code. (Btree fuzzer, rope fuzzer, file serialisation fuzzer, a few crdt fuzzers, and so on).
Writing fuzzers is quite devastating for the ego. Usually the first time I point a fuzzer at my code, even when my code has a lot of tests, the fuzzer throws an assertion failure instantly. “Iteration 2 … the state doesn’t match what was expected”.
Getting a fuzzer running all night without finding any bugs is a balm for the soul.
The code looks like this, if anyone is curious. Here’s a fuzzer for a rope (fancy string) implementation: https://github.com/josephg/jumprope-rs/blob/master/tests/tes...