"Linux kernel patches regularly get added in a way that bypasses public review and discussion, a practice that opens at least a theoretical risk of backdoored code"
I wonder about how theoretical this risk of backdoored code is and what kind of backdoors the authors envision.
Security patches are discussed in private by some, if not most, of the same technically capable people who would otherwise discuss the patches on public mailing lists (like LKML).
After the code is committed, it is available for everyone to see and discuss. All new code, including "backdoored code" would also get visibility and attention.
Even if anyone can see the discussion on LKML it does not mean all those people actively participate in the discussion, or even review the code.
There's really no reason to believe in the provenance of any of the code in the Linux kernel. People mail around patchsets, claim they are "signed off by" someone, and they get patched into various trees and merged here and there, and eventually wind up in Linus's tree. It has all of the safety of email, which is to say none at all. Compared to code review practices at software companies, where diffs are reviewed and approved by identities authenticated and authorized by secure computer systems, the linux kernel system is a complete joke.
Oh, come on, the code review process at most companies is fairly easy to bypass. Even at the company I'm sure you're going to bring up I would seriously doubt it wouldn't be possible to slip something into a long patch, even though I'm sure there are some sort of guidelines on who and how many people should be reviewing things.
Ive said this before on here, but I'm 95% sure I could find two or three on my team and say "hey I have a simple fix for X, needs to go in quick, can I tag you in it?", and they would rubber stamp it knowing it comes from me.
Patches are reviewed just not in the way you are accustomed to, but it doesn't make it any less secure. The kernel is continually audited and tested, and is the most widely used kernel in security devices.
What you are asking for is that developers should have to prove their real-life identity and authenticate to a public SCM system and that somehow makes it more secure, when hidden exploits could as easily be sent as a PR on GitHub and merged by someone who didn't know better or who can just play dumb.
I don't buy "continually audited and tested". That is a variant of the "many eyes make all bugs shallow" theory or "Linus's Law", but that hasn't proven to be the case. The Linux kernel is full of bugs and many of them hang around for years. Also the kernel testing story is a total joke. Perhaps you meant that people use the kernel and therefore it's "tested"?
Calling Linux testing a "total joke" is a total joke in itself -- of course Linux is not perfect (nothing is).
As for testing, there is kernelci.org, LTP, Syzbot, LKFT and so on. It's not just a bunch people in basements booting up the latest Linux and calling it testing.
Isn't testing supposed to find bugs? And aren't fuzzers identifying issues that /might/ exist in theory but dont manifest in practice?
From your assessment I'm surprised my Linux boxen don't crash every day and really surprised I had one stay online for five years - but, many things things about Linux surprise me.
Testing is supposed to prevent bugs from being committed. Finding the bugs in released software after the fact is the worst possible process.
The fact is that people discover and fix bugs, and then years later someone else commits the same mistake again, and nothing prevents this because the kernel has literally zero pre-commit unit tests. Here's an example (thread by the author of syzkaller). https://twitter.com/dvyukov/status/1169544165023240194
When commits move from one tree to another (which is the only time when full code review isn't done, because the maintainer is generally trusted), it is done with a GPG-signed tag. Email's lack of signing therefore is not an issue.
The author managed to get it in and it lay undiscovered for several years.
So even though this bug had the visibility and attention it wasn’t discovered in the field until several years later.
It may be that the bug was intentionally added or it may have been a simple coding mistake, but either way it’s an example of something innocuous looking being added to a project which could have been used covertly for much longer than it was publicly known about.
I agree, I actually know one of the guys who discovered the bug and had a talk with him the same week the CVE was released. He seemed very sure that it was simply a mistake. While fuzzing has been a thing since the 50s, it wasn't as common to fuzz stuff back when with larger input sets just due to practical issues. You'd be lucky to own a quad core CPU and nobody spent their free time throwing cycles at random Linux package source codes for fun.
Stuff like this goes uncovered all the time. The Ring 0 RISC exploit from 2018 comes to mind at first. It wasn't a whole two years after Shellshock and heartbleed that Google releases their Fuzzing service.
> “The existence of such channels, shows that trusted individuals can easily infiltrate the project, and secretly introduce malicious artifacts”
That is like saying that trusted code can easily do malicious things. Of course trusted people can do bad things, that’s why you have to trust them. Even so, source control history means that a trusted person should be found out if they introduce malicious artifacts.
> while maintainers are aware of that they sometimes intentionally bypass the process, they were surprised of the magnitude of unreviewed patches
Would love to see an analysis of these changes - are they just simple merge style fixes or rearrangements, or more significant?
And then there is the hard to define distinction between a security bug and a normal bug, which is then mixed into the the incredible productivity and pace of kernel development:
> Koah-Hartman argues that only a small fraction of Linux kernel security fixes are assigned to CVE entries. From 2006-2018, 1005 CVEs were assigned to the kernel. He argues that, on average, bugs with CVE entries are 100 days fixed in mainline before they get a CVE assigned.
Seems there is long lag between the bug being introduced and the exploit discovered, so there must be many potential security exploits that are never discovered before they are fixed - and so are not practically exploitable as they never get into downstream distribution kernels.
The word “boffin” is a colloquial British term for a smart person (conjuring up an image of someone in a white lab coat and glasses) and is used in an affectionate way. The British version of TheReg (.co.uk) often refers to
boffins in this way. It’s probable that this piece was written by the British team and then reposted to the US site.
23 comments
[ 2.8 ms ] story [ 57.1 ms ] threadI wonder about how theoretical this risk of backdoored code is and what kind of backdoors the authors envision.
Security patches are discussed in private by some, if not most, of the same technically capable people who would otherwise discuss the patches on public mailing lists (like LKML).
After the code is committed, it is available for everyone to see and discuss. All new code, including "backdoored code" would also get visibility and attention.
Even if anyone can see the discussion on LKML it does not mean all those people actively participate in the discussion, or even review the code.
What you are asking for is that developers should have to prove their real-life identity and authenticate to a public SCM system and that somehow makes it more secure, when hidden exploits could as easily be sent as a PR on GitHub and merged by someone who didn't know better or who can just play dumb.
As for testing, there is kernelci.org, LTP, Syzbot, LKFT and so on. It's not just a bunch people in basements booting up the latest Linux and calling it testing.
https://groups.google.com/forum/m/#!forum/syzkaller-bugs
Isn't testing supposed to find bugs? And aren't fuzzers identifying issues that /might/ exist in theory but dont manifest in practice?
From your assessment I'm surprised my Linux boxen don't crash every day and really surprised I had one stay online for five years - but, many things things about Linux surprise me.
The fact is that people discover and fix bugs, and then years later someone else commits the same mistake again, and nothing prevents this because the kernel has literally zero pre-commit unit tests. Here's an example (thread by the author of syzkaller). https://twitter.com/dvyukov/status/1169544165023240194
https://github.com/openssl/openssl/commit/bd6941cfaa31ee8a3f...
The author managed to get it in and it lay undiscovered for several years.
So even though this bug had the visibility and attention it wasn’t discovered in the field until several years later.
It may be that the bug was intentionally added or it may have been a simple coding mistake, but either way it’s an example of something innocuous looking being added to a project which could have been used covertly for much longer than it was publicly known about.
I don't think anyone says that open source code makes bugs impossible, but it definitely makes it harder to insert intentional backdoors.
Stuff like this goes uncovered all the time. The Ring 0 RISC exploit from 2018 comes to mind at first. It wasn't a whole two years after Shellshock and heartbleed that Google releases their Fuzzing service.
That is like saying that trusted code can easily do malicious things. Of course trusted people can do bad things, that’s why you have to trust them. Even so, source control history means that a trusted person should be found out if they introduce malicious artifacts.
> while maintainers are aware of that they sometimes intentionally bypass the process, they were surprised of the magnitude of unreviewed patches
Would love to see an analysis of these changes - are they just simple merge style fixes or rearrangements, or more significant?
And then there is the hard to define distinction between a security bug and a normal bug, which is then mixed into the the incredible productivity and pace of kernel development:
> Koah-Hartman argues that only a small fraction of Linux kernel security fixes are assigned to CVE entries. From 2006-2018, 1005 CVEs were assigned to the kernel. He argues that, on average, bugs with CVE entries are 100 days fixed in mainline before they get a CVE assigned.
Seems there is long lag between the bug being introduced and the exploit discovered, so there must be many potential security exploits that are never discovered before they are fixed - and so are not practically exploitable as they never get into downstream distribution kernels.