I can't wait for the time when if your project reads files, fuzzing it is a well-known, recommended practice. (unless you're writing fully managed code)
There were some fuzzing projects before afl, but even with afl's recent popularity we're still in the situation where you just have to grab a random application, point afl at it and get some basic crashes in a few minutes. With clang-analyzer, coverity, afl, and many other projects available for free, there's no reason this should be possible.
Then again, I'm still waiting for the time when people don't code with sql injection issues...
Even if you're writing fully managed code you should still fuzz. It can help find logic errors in the code which can cause erroneous program states or crashes.
The fact that the code is managed helps prevent against some nasty exploits, but it doesn't necessarily mean the code always does what the programmer intended.
Agreed! I like to think of the fuzzers as merely an automated tool to drive test cases, maximizing on coverage.
So when asked, "Why fuzz test?" I can answer, "remember when you wrote fifteen test cases and wondered to yourself whether you needed a sixteenth? But then you decided that it would take more work than it's worth to do that. -- Well, now we can tell the computer to just keep trying stuff until it finds unique cases."
There is also generative testing, which can be a good middle-ground between hand-crafted tests and fuzzing. You can think of it as a fuzzer with more guidance regarding input generation, and can uncover edge-cases that a fuzzer would take forever to discover. I tend to think of those 3 approaches as complementary.
Sure, I mean fuzzing should be critical for applications which can sigsegv, but it doesn't mean others shouldn't also give it a go. It just requires some integration support (you need to let the fuzzer know that an exception got all the way to the top for example), so it's up to the project if they want to go that way.
There were and are a lot of fuzzing tools before AFL, it's been the normal go-to approach for around 10 years for finding security bugs.
Because most software projects are so lazy about doing fuzzing themselves, making and promoting easier to use tools is a good way to nudge people into picking the low hanging fruit (bugs?).
Fuzzing works for managed code, but since you are looking for logic bugs instead of memory corruption crashes, more effort is needed to verify the behaviour of your app in the face of fuzzed data. Things like TOCTOU bugs, data sanitation bugs like path traversal or SQL injection vulns, etc. Of course this finds non-crashy security bugs in C code too, like in the Heartbleed case...
One of the things that made me a little twitchy is that FFmpeg replaced the very thoroughly fuzzed official Opus decoder with their own implementation that wasn't.
It's actually good to have multiple independent implementations of a decoder to find inconsistencies and bugs. In fact Mozilla sponsored the work, and Google also did indirectly through GSoC.
OT: I can't describe how much I loathe "for fun and for profit" in titles - especially since the majority aren't linked to anything describing a monetary or resource gain. If you want to say something is useful, find a different way to say it!
I wish those titles had a weight to drag them off the front page much quicker.
Maybe you know this, maybe you do not but these are all a nod to http://insecure.org/stf/smashstack.html which itself did not directly involve profit. It's a security thing.
As a Java programmer I would love a version of afl. The JVM certainly exposes enough information for other profilers/tracers and things dynamically adding code (spring/hibernate) that it seems like it should be possible.
There's a tool that does similar to AFL called EvoSuite (http://www.evosuite.org). It uses a genetic algorithm like AFL and generates JUnit suites for the target classes. It doesn't look for crashes like AFL - it's closer to MS's IntelliTest tool but uses metaheuristics instead of SMT to break conditions.
Can someone explain why you would want to fuzz ffmpeg?
What's wrong with ffmpeg crashing when you feed it invalid input? What's the alternative to it not crashing? Should it continue transcoding or should it exit quietly?
Fuzzing is useful for finding bugs that may or may not be serious security issues. If a program crashes when you feed it invalid input, it could potentially crash in a way that allows code execution.
After finding a crash, a researcher will generally explore this to see if it allows for arbitrary code execution. If it does, it is possible for someone to create a weaponized exploit.
For a utility as widely-used as ffmpeg, any bugs pose a threat to many systems. Fuzzing it makes the Linux ecosystem safer for all of us (hopefully).
The alternative to crashing is exiting cleanly with an error. A crash can often be transformed into remote code execution with some work. That would mean that your computer could be taken over if you transcode the wrong video using ffmpeg. Which is pretty wrong.
There is nothing wrong with crashing if you can do it in a "safe" manner. If an invalid input allows an attacker to take over your computer, then that is a problem :). Especially for something as wide-spread as ffmpeg.
Video can often come from a less than perfectly reliable source (eg. satellite downlink), so video transcoders should be able to handle errors as gracefully as possible.
No mention of security impact of these bugs in the tracker. Anyone know if FFmpeg has some vulnerability process, or do they just commit fixes and let other people worry about exploitability, security hotfixes and CVEs, like Linux?
I didn't see any note about what version these bugs were found in, or will be fixed in. For reference, at the time of this writing, my GNU/Linux box has ffmpeg 2.8.1.
This mentions the fixes they found (NULL pointer dereferences, Invalid pointer arithmetic leading to SIGSEGV due to unmapped memory access, Out-of-bounds reads and writes to stack, heap and static-based arrays, Invalid free() calls, Double free() calls over the same pointer, Division errors, Assertion failures, Use of uninitialized memory.)
Some of those things could have been caught by static code analysis or stricter coding standards. Or, if it had been a C++ project you could replace some of those problems (NULL pointer dereferences) with using references in the first place instead of passing pointers around. That might be a bit of an oversimplification as it is a very complex project but for me it was a reminder to change my coding style in C++ instead of sticking with the C-style way of doing it.
C++ would make it a horrible project to work on. No doubt you would advocate for it to use the slow features of c++. Templates or something, right? Then you would probably say that it should use boost too because that seems to be the standard library of c++.
Wrong. I WOULD NOT advocate the use of templates just because I use C++. But I WOULD advocate using safe language features and compile-time type checking. Throwing pointers around is bad practice, hence C++'s encouragement to use references, move semantics and emphasis on use of containers, clear tidy-up practices, obvious demarkation of what owns an object, RAII etc. This moves the burden onto the compiler, not the developer or team of developers to understand where data is owned, when it will be cleaned up etc. etc.
Further, templates do not make C++ slow; it is simply compile-time polymorphism.
Does runtime polymorphism make software slow? Does casting to or from a void* in C suddenly make it slow? The entire emphasis in C++ for templates is that you pass the error checking to the compiler, not at runtime to guess that this incoming void* is the type that you want. reinterpret_cast is only used where absolutely necessary!! It is far safer to NOT cast. Avoid casting. Templates (and the STL) help out in this regard.
I would use the STL for projects, and encourage every else in C++ land to do the same. Containers make sense, as do the algorithms associated with them (and they save you reinventing the wheel).
But I would NOT advocate use of Boost just because it is popular or written in C++. That would be a stupid reason to encourage its use; I personally do not use Boost. Boost != STL. You appear to be confusing Boost with the STL ??
No doubt your unbridled hatred of C++ and libraries written in C++ (eg. Boost) has stopped you reading reference works on the STL. You can find a comprehensive reference here: http://en.cppreference.com/w/
There's no Boost on there, btw. That's at boost.org.
Personally, I was only making the comment regarding a reminder of my own coding style in light of the problems they found in their C codebase. I do not write C, but I can see that the pitfalls they fell into could easily be carried out by me in C++ if I stuck to unsafe coding styles.
Furthermore, many of their problems could be solved in C++ land very easily: NULL pointer dereferences (don't use pointers, use references - it'll never be NULL), Invalid pointer arithmetic leading to SIGSEGV due to unmapped memory access (use STL containers), Out-of-bounds reads and writes to stack, heap and static-based arrays (use STL containers, you may still get out-of-bounds problems but at least an exception is thrown), Invalid free() calls, Double free() calls over the same pointer (use move semantics, never see a new or delete ever again), Division errors (use a static_assert to check that a value is in range, if possible. Or use an assert in a defensive coding style), Assertion failures (use static_assert to catch at compile time if possible), Use of uninitialized memory (RAII - initialise before use).
Your (or anyone's) invalid use of C++ does not make the language invalid, in the same way that stabbing someone with a knife does not suddenly mean all knives must be banned.
Doesn't STL containers make your program slower though (e.g. game devs generally manage memory manually, for better performance)? For something like FFmpeg, that's a valid concern.
I thought typically for efficient pipelines on x86 hardware, or to avoid cache hits where the data being shoved in does not fit the hardware perfectly.
For FFmpeg that is a concern, but I believe the replier to my original comment was disparaging C++ for no good reason, other than it wasn't C...
39 comments
[ 2.9 ms ] story [ 79.8 ms ] threadThere were some fuzzing projects before afl, but even with afl's recent popularity we're still in the situation where you just have to grab a random application, point afl at it and get some basic crashes in a few minutes. With clang-analyzer, coverity, afl, and many other projects available for free, there's no reason this should be possible.
Then again, I'm still waiting for the time when people don't code with sql injection issues...
The fact that the code is managed helps prevent against some nasty exploits, but it doesn't necessarily mean the code always does what the programmer intended.
So when asked, "Why fuzz test?" I can answer, "remember when you wrote fifteen test cases and wondered to yourself whether you needed a sixteenth? But then you decided that it would take more work than it's worth to do that. -- Well, now we can tell the computer to just keep trying stuff until it finds unique cases."
Example in Clojure: https://github.com/clojure/test.check
Because most software projects are so lazy about doing fuzzing themselves, making and promoting easier to use tools is a good way to nudge people into picking the low hanging fruit (bugs?).
Fuzzing works for managed code, but since you are looking for logic bugs instead of memory corruption crashes, more effort is needed to verify the behaviour of your app in the face of fuzzed data. Things like TOCTOU bugs, data sanitation bugs like path traversal or SQL injection vulns, etc. Of course this finds non-crashy security bugs in C code too, like in the Heartbleed case...
I wish those titles had a weight to drag them off the front page much quicker.
I wish someone would do a JVM version (combined with the lines of QuickCheck and typed generators).
[0] http://lcamtuf.coredump.cx/afl/
[1] https://github.com/cseed/arachne-pnr
[1] https://googleonlinesecurity.blogspot.com/2014/01/ffmpeg-and...
What's wrong with ffmpeg crashing when you feed it invalid input? What's the alternative to it not crashing? Should it continue transcoding or should it exit quietly?
What's the problem here trying to be solved?
After finding a crash, a researcher will generally explore this to see if it allows for arbitrary code execution. If it does, it is possible for someone to create a weaponized exploit.
For a utility as widely-used as ffmpeg, any bugs pose a threat to many systems. Fuzzing it makes the Linux ecosystem safer for all of us (hopefully).
This mentions the fixes they found (NULL pointer dereferences, Invalid pointer arithmetic leading to SIGSEGV due to unmapped memory access, Out-of-bounds reads and writes to stack, heap and static-based arrays, Invalid free() calls, Double free() calls over the same pointer, Division errors, Assertion failures, Use of uninitialized memory.)
Some of those things could have been caught by static code analysis or stricter coding standards. Or, if it had been a C++ project you could replace some of those problems (NULL pointer dereferences) with using references in the first place instead of passing pointers around. That might be a bit of an oversimplification as it is a very complex project but for me it was a reminder to change my coding style in C++ instead of sticking with the C-style way of doing it.
Further, templates do not make C++ slow; it is simply compile-time polymorphism.
Does runtime polymorphism make software slow? Does casting to or from a void* in C suddenly make it slow? The entire emphasis in C++ for templates is that you pass the error checking to the compiler, not at runtime to guess that this incoming void* is the type that you want. reinterpret_cast is only used where absolutely necessary!! It is far safer to NOT cast. Avoid casting. Templates (and the STL) help out in this regard.
I would use the STL for projects, and encourage every else in C++ land to do the same. Containers make sense, as do the algorithms associated with them (and they save you reinventing the wheel).
But I would NOT advocate use of Boost just because it is popular or written in C++. That would be a stupid reason to encourage its use; I personally do not use Boost. Boost != STL. You appear to be confusing Boost with the STL ??
No doubt your unbridled hatred of C++ and libraries written in C++ (eg. Boost) has stopped you reading reference works on the STL. You can find a comprehensive reference here: http://en.cppreference.com/w/
There's no Boost on there, btw. That's at boost.org.
Personally, I was only making the comment regarding a reminder of my own coding style in light of the problems they found in their C codebase. I do not write C, but I can see that the pitfalls they fell into could easily be carried out by me in C++ if I stuck to unsafe coding styles.
Furthermore, many of their problems could be solved in C++ land very easily: NULL pointer dereferences (don't use pointers, use references - it'll never be NULL), Invalid pointer arithmetic leading to SIGSEGV due to unmapped memory access (use STL containers), Out-of-bounds reads and writes to stack, heap and static-based arrays (use STL containers, you may still get out-of-bounds problems but at least an exception is thrown), Invalid free() calls, Double free() calls over the same pointer (use move semantics, never see a new or delete ever again), Division errors (use a static_assert to check that a value is in range, if possible. Or use an assert in a defensive coding style), Assertion failures (use static_assert to catch at compile time if possible), Use of uninitialized memory (RAII - initialise before use).
Your (or anyone's) invalid use of C++ does not make the language invalid, in the same way that stabbing someone with a knife does not suddenly mean all knives must be banned.
For FFmpeg that is a concern, but I believe the replier to my original comment was disparaging C++ for no good reason, other than it wasn't C...