I immediately thought of that Bones episode where a computer is set on fire because a bone put into a scanner had some kind of malware embedded in it. The future is now.
The way you get bitten is that there's an opportunity to trade off security versus functionality.
Imagine you've got the alleged bomber's computer, it has hundreds of thousands of images with long hexadecimal names like 384d38b0c32aaf03.jpg and you're under pressure to figure out where the missing fifth bomb is before everybody finds out the hard way.
Now, on one hand you've got a simple image viewer built with WUFFS: https://github.com/google/wuffs which you are completely sure is safe for viewing the images, you could perhaps glance at a hundred images per hour, although if there's anything subtle you'd miss it.
On the other hand, you've got a fancy AI-assisted tool that processes over 1000 images per minute and cross-matches them with over a hundred law enforcement databases and other sources to understand what it sees. It might be able to pick out anomalous images in time to make a difference. Still, it was written in C++ and the last time you used it the program crashed three times with mysterious errors.
Which do you choose? If the bomb blows up and kills sixty elementary school kids while you're glancing at image #406 having not noticed that school is in the background of images #284, #301, #305, #313, and #326 and never guessing that the unknown interiors in images #192-198 and #350-400 are from that same school - you're going to feel like an idiot for choosing the slow path.
But if the bomb blows up when your AI assisted image processor is exploited by a C++ JPEG library zero day inside image #4861 and that sends the "explode now" code to a custom web site the bomber set up for exactly this eventuality, you're going to feel like an idiot for not using WUFFS.
This sort of trade-off is how Vinge's "A Fire Upon the Deep" begins. Humans (roughly) are poking around in an apparently forgotten archive in deep space. They could be slow and careful, but it's too tempting to choose quicker methods, smarter methods, letting the archive direct how to improve their AI to better understand it. By the time the humans realise their AI is doing what the archive wanted it to do, not what they wanted it to do, it's far too late for them.
Not if you're actually parsing, with an explicit parse step and a well-defined language. Exploits tend to happen when parsing is done by the so-called "shotgun parser" - aka. various checks and conditionals randomly scattered throughout your code, that implicitly define an input language that's different from what you think it is.
"the researchers admit, they also had to take some serious shortcuts in their proof-of-concept that verge on cheating. Rather than exploit an existing vulnerability in the fqzcomp program, as real-world hackers do, they modified the program's open-source code to insert their own flaw allowing the buffer overflow."
I think they should leave out the verge. But it is an interesting thing to think about, that you can encode DNA with the specific intent to cause a buffer overflow in a vulnerable program. But not this one.
That seems uncharitable; it was not that trivial of a cheat. They inserted a realistic type of vulnerability that was not specific to their test.
> we first copied fqzcomp from https://sourceforge.net/projects/fqzcomp/and inserted a vulnerability into version 4.6 of its source code; a function that processes and compresses DNA reads individually, using a fixed-size buffer to store the compressed data. This modification lets us perform a buffer overflow with a longer than expected DNA read in order to hijack control flow. While the use of such a fixed-size buffer is an obvious vulnerability, we note that fqzcomp already contains over two dozen static buffers.
> The modified fqzcomp program contained a buffer too small for the 177 base pair read length, so it would overflow after processing the first read.
If the attack is to e.g. get the software to report a different gene sequence to the real one, this isn't going to help you. For example you might encode such an attack to mask the gene sequence of a viral weapon.
I guess the point is to prove this novel attack vector. And they found interesting things that don't always show up in a buffer attack: there are certain restrictions on patterns that can be viably encoded physically, parallel sequencing can chunk up the input in unpredictable ways, the input might be read backwards by the sequencer.
I used to work in the office next to James'. Don't know how he feels about it, but this seemed pretty unfair to be honest. It's an experimental codebase released by an academic institute. I think he originally wrote it for one of the DNA data compression competitions?
If this was in something one of Illumina's tools, on a platform that is used for diagnostics (some of their platforms have FDA approval). Then I think this would be fairer... but it's not, and they probably didn't do that because it would be too much work.
As it stands, they might as well have written their own buggy tool and exploited it to be honest.
If the gene sequencer is exploitable by some DNA sequence, maybe is possible for a physical virus to evolve in a way to trigger the bug so his survival probability increases by confusing the scientists looking at them :) Maybe the future will be full of unexpected interplay's between software and the biological world.
At first I thought that was silly, the feedback loop from the variation in the virus to this attack having any effect on it's propagation is too slow. But then I realised in the future we might have virus fighting nanomachines with built-in gene sequencers, and suddenly this becomes much more credible.
Sure you could encode the computer virus in the gene sequence as a payload, and use this attack to compromised the sequencer and then load the code into the computer system.
If you go one level meta, it's not hard to imagine a Ken Thompson "Reflections on Trusting Trust" situation where that DNA codes for some brain malware that will inject more malware whenever its host is at work writing software, such as for a DNA sequencer.
I helped build a cloud product that parsed BAM and VCF files (Google Cloud Genomics). When we first started I looked around for various open source BAM parsers so we didn't have to write our own. I showed the parsers to my engineer, who came back a day later with 10 files that caused the parsers to crash (common problem: if you put a negative length in a field that's supposed to be positive, it will reverse index into a string).
We contacted all the BAM parser developers (academic) and they all said they had no interest in identifying and fixing bugs in their parsers. We wrote an industrial strength BAM parser in Go.
I think a year or two later one of the major sequencing centers got hozed by a malicious BAM file and that was when all the open source BAM developers got security religion.
I don't think this situation is possible because from the point of view of sequencers, all DNA is equally valid input. Because systems are usually tested in happy flows, vulnerabilities can only happen in cases the input is in some way different from the happy flow, triggering an untested behavior.
Because all DNA is equally valid to these tools, it's impossible to have a vulnerability that won't get triggered and patched by the normal testing and developing routine.
37 comments
[ 3.4 ms ] story [ 87.4 ms ] threadsanitize your inputs ---> no exceptions!
Imagine you've got the alleged bomber's computer, it has hundreds of thousands of images with long hexadecimal names like 384d38b0c32aaf03.jpg and you're under pressure to figure out where the missing fifth bomb is before everybody finds out the hard way.
Now, on one hand you've got a simple image viewer built with WUFFS: https://github.com/google/wuffs which you are completely sure is safe for viewing the images, you could perhaps glance at a hundred images per hour, although if there's anything subtle you'd miss it.
On the other hand, you've got a fancy AI-assisted tool that processes over 1000 images per minute and cross-matches them with over a hundred law enforcement databases and other sources to understand what it sees. It might be able to pick out anomalous images in time to make a difference. Still, it was written in C++ and the last time you used it the program crashed three times with mysterious errors.
Which do you choose? If the bomb blows up and kills sixty elementary school kids while you're glancing at image #406 having not noticed that school is in the background of images #284, #301, #305, #313, and #326 and never guessing that the unknown interiors in images #192-198 and #350-400 are from that same school - you're going to feel like an idiot for choosing the slow path.
But if the bomb blows up when your AI assisted image processor is exploited by a C++ JPEG library zero day inside image #4861 and that sends the "explode now" code to a custom web site the bomber set up for exactly this eventuality, you're going to feel like an idiot for not using WUFFS.
This sort of trade-off is how Vinge's "A Fire Upon the Deep" begins. Humans (roughly) are poking around in an apparently forgotten archive in deep space. They could be slow and careful, but it's too tempting to choose quicker methods, smarter methods, letting the archive direct how to improve their AI to better understand it. By the time the humans realise their AI is doing what the archive wanted it to do, not what they wanted it to do, it's far too late for them.
"the researchers admit, they also had to take some serious shortcuts in their proof-of-concept that verge on cheating. Rather than exploit an existing vulnerability in the fqzcomp program, as real-world hackers do, they modified the program's open-source code to insert their own flaw allowing the buffer overflow."
I think they should leave out the verge. But it is an interesting thing to think about, that you can encode DNA with the specific intent to cause a buffer overflow in a vulnerable program. But not this one.
> we first copied fqzcomp from https://sourceforge.net/projects/fqzcomp/and inserted a vulnerability into version 4.6 of its source code; a function that processes and compresses DNA reads individually, using a fixed-size buffer to store the compressed data. This modification lets us perform a buffer overflow with a longer than expected DNA read in order to hijack control flow. While the use of such a fixed-size buffer is an obvious vulnerability, we note that fqzcomp already contains over two dozen static buffers.
> The modified fqzcomp program contained a buffer too small for the 177 base pair read length, so it would overflow after processing the first read.
https://www.usenix.org/system/files/conference/usenixsecurit...
https://sourceforge.net/projects/fqzcomp/
needs a space before "and".
Also, one solution seems simple: run the software inside a virtual machine.
It should be noted that "static buffers" are not per se a vulnerability. This sentence reads like they equate the two.
If this was in something one of Illumina's tools, on a platform that is used for diagnostics (some of their platforms have FDA approval). Then I think this would be fairer... but it's not, and they probably didn't do that because it would be too much work.
As it stands, they might as well have written their own buggy tool and exploited it to be honest.
It's amazing, but it also shows how poor the defense against buffer overflows is in many programs.
https://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-tho...
We contacted all the BAM parser developers (academic) and they all said they had no interest in identifying and fixing bugs in their parsers. We wrote an industrial strength BAM parser in Go.
I think a year or two later one of the major sequencing centers got hozed by a malicious BAM file and that was when all the open source BAM developers got security religion.
Because all DNA is equally valid to these tools, it's impossible to have a vulnerability that won't get triggered and patched by the normal testing and developing routine.