> This Saturday, the UMN research team apologized to the Linux community via an open letter posted to the Linux Kernel Mailing List. The nearly 800-word open letter comes across as more "wait, you don't understand" than apology:
Wow, it really is a non-apology. They spend far more time defending themselves and don’t really seem remorseful at all. At no point did they say they seem to have punished the people involved or changed the process that allowed this research to be conducted in the first place.
What I don't understand is what the theoretical endgame would have been here?
Putting the circumstance aside that their submissions are effectively indistinguishable from malicious content (and should be treated as such anyway), if you stay in their line of "it's just research", then getting banned is just another possible outcome that should have been expected as a possibility from the beginning, no?
What they are doing now is basically saying "Yeah we know we did this to see whether actors like us get blocked, but you see, it's US, we should clearly get different treatment".
The ban happened now because GKH got fed up with a different researcher from the same group who was submitting (qualitatively) bad patches and said that it looked like a continuation of the "Hypocrite Commits" work, which the kernel team strongly objects to.
Thus it's a relevant clarification that only small set of patches in 2020 belonged to that project, they did not continue it, and the recent patches are not related to it and are merely bad, not malicious. (Of course the student should have done a better job with that, but it's a categorical difference between him also trying to trick people vs merely messing up and getting in extra trouble because his advisor pissed off the kernel team earlier)
> As many observers have pointed out to us, we made a mistake by not finding a way to consult with the community and obtain permission before running this study; we did that because we knew we could not ask the maintainers of Linux for permission, or they would be on the lookout for the hypocrite patches.
In some ways this reminds me of a phishing exercise at work that was done under the guise of security awareness training. Is it just me or is there a pattern of behavioral issues among security people and a lack of empathy.
In my experience, it's not a lack of empathy, but trying to prove to management that they shouldn't get rid of the security team. At most companies I've done security testing at, telling management that you didn't find any vulnerabilities immediately puts you and your team on the chopping block because they think you're a waste of money if you don't find anything. I've noticed it's mostly new people who aren't confident they can easily find a job are the ones most often going out of their way to report security issues that aren't actually a big deal.
At least those phishing exercises are done with the informed consent of the organization.
If they go too far and impact day-to-day work, or if people complain, the executives understand what is going on and who they can talk to about improving the process.
And if it feels lacking in empathy, consider that the company's management needs to sign off on any (legal) phishing tests that their employees are subjected to.
Please note that the senior management is always in the loop for the phishing tests along with some of the security team. It's just the users who are not told. You can't just send out a phishing email without telling anyone about it, you are liable to get fired for it. Remember the first rule of cybersecurity testing is to get written permission and then establish rules of engagement.
All these researchers had to do was reach out to Greg, loop him in the study to get his go ahead for it.
Now that several of these 'patches' are now in stable and in fact did introduce issues is a big problem and a major waste of time for the maintainers.
What is relevant in this specific Ars story for us outsiders to discuss? Is this current state a surprise to anyone? What makes this worth our time to read, aside from being ”omg, look at the day-old drama”?
An apology should actually contain admission of wrongdoing and an attempt at making amends, as well as an honest undertaking to not repeat that wrongdoing.
UoM got caught doing wrong. Their "why" doesn't help because their actions are rightly seen by their targets as malicious. This is like breaking into a store, trying to steal something, getting caught and them saying "well done! We were just testing!" Yet this was without any agreement that such testing was prior sanctioned: their actions must be seen as malicious. They set out to deliberately do something where they could be caught. They did get caught. They didn't have permission to test.
They need to actually apologize and make amends. Otherwise they might get the idea that trying again is acceptable. The next time they do this things will be even worse.
So what was the damage or harm? Loss of trust is one big ticket item. They wasted other's time and effort. Some important feature or functionality was delayed and is likely still being delayed. Or documentation not written. Even something as simple as "enthusiasm to volunteer" might be damaged. These aren't trivial aspects to be ignored.
They still seem to believe that what they did, increases the security of Linux. And comments still suggest the fix to make Linux less "open" to contributions.
It does not seem to me that they have thought much about what they are saying. For example, in a project like Linux, there is new code, but they majority of contributions, especially small fixes (as compared to new drivers and rewrites of subsystems), are very likely bug fixes. And many bugs can intruduce security issues, too, even if that might not be obvious. Therefore, it is not clear at all that making Linux more "closed" (which seems to be the solution which the UMN people and their friends suggest) would make it any more secure. More scrutiny by top maintainers means more time is needed; needing more time means automatically less fixes can be applied, since the maintainer's time is the bottleneck; applying less fixes means more open bugs, and having more open bugs can at least easily mean to have more security issues.
There are plenty of closed source projects with a security which is clearly worse than Linux. And not only that, but core Internet security infrastructure like DNS or OpenSSL is open source. The issue is rather that companies cannot be arsed to support that infrastructure they depend on. It would be interesting if the UMN people could come up with research or a hack which motivates them to change. (OTOH, one mayor windows worm per decade such as locky, wannacry, petya, tiny banker, cryptolocker, flame, and so on may be sufficient for the latter in the long run.)
I have looked at several reverts and their review comments. Obviously none of those were hypocrite. The biggest problem is that even experienced kernel developers far too often cannot clearly decide whether the code is better with or without the patch.
There are several reasons:
* C is not an easy language that makes it hard to write unsafe code
* hardware can fail in any unspecified way
* the kernel community prefers kludges over simpler, visibly more robust code if the latter has performance impact (The biggest example of this is of course choosing a monolithic kernel instead of a microkernel.)
My performance and scalability requirements are much less than what the Linux kernel can do. So if there were a serious competitor based on code that humans can understand, my choice would be clear. In lack of such option I continue to use Linux and hope that nothing too bad happens...
> The biggest problem is that even experienced kernel developers far too often cannot clearly decide whether the code is better with or without the patch.
It was a bulk revert; not "we can't tell if these are bad" but "we're reverting all of these until we can re-review them". Or, if you're claiming they shouldn't have been merged in the first place if it wasn't possible to be 100% confident in them... that's not how humans work, nor programming. Even if you specify your code in coq or something, you still have to verify that the model is correct and there's still room for human error.
> the kernel community prefers kludges over simpler, visibly more robust code if the latter has performance impact (The biggest example of this is of course choosing a monolithic kernel instead of a microkernel.)
Yes, they should have followed in the footsteps of all those other successful feature-complete microkernels like, um.... maybe QNX? Maybe? To be fair, I don't know that there's a reason why microkernels seem stuck in the "theoretically great but nobody's made one that works in the real world", but here we are.
So in broad strokes: You seem to be very confident that if kernel devs would just Do It Right then they could eliminate all bugs and malicious code, but this seems unlikely.
> You seem to be very confident that if kernel devs would just Do It Right then they could eliminate all bugs and malicious code
You wrote that, I didn't. The Linux kernel has chosen its route, I am not going tell them that they should do different.
All I say is should be possible to write a kernel that is reviewable by more people. It will be less scalable and less performant. But that's a price I might be ready to pay both for myself and for my work. My machines are hardly ever limited by kernel performance. Others might certainly have use cases where that is not true.
> * the kernel community prefers kludges over simpler, visibly more robust code if the latter has performance impact (The biggest example of this is of course choosing a monolithic kernel instead of a microkernel.)
I don't think Minix is actively maintained. Intel secretly chose to run Minix for their ME and the results have been desastrous.
I run Linux because it's the best option available today. That does not mean it's the best option conceivable if you value simplicity and visible robustness over performance and scalability.
It's good to see Greg KH and the Linux foundation taking a firm stance wrt this issue. The motivation put forward by the group from UMN is unacceptable.
It's pure sabotage and deep disrespect for the hours of toil professionals and volunteers worldwide put into the Linux project.
27 comments
[ 2.7 ms ] story [ 79.9 ms ] threadWow, it really is a non-apology. They spend far more time defending themselves and don’t really seem remorseful at all. At no point did they say they seem to have punished the people involved or changed the process that allowed this research to be conducted in the first place.
Putting the circumstance aside that their submissions are effectively indistinguishable from malicious content (and should be treated as such anyway), if you stay in their line of "it's just research", then getting banned is just another possible outcome that should have been expected as a possibility from the beginning, no?
What they are doing now is basically saying "Yeah we know we did this to see whether actors like us get blocked, but you see, it's US, we should clearly get different treatment".
Like that matters.
Like they got away with it and the statute of limitations should be up and they shouldn’t be punished at this point.
Thus it's a relevant clarification that only small set of patches in 2020 belonged to that project, they did not continue it, and the recent patches are not related to it and are merely bad, not malicious. (Of course the student should have done a better job with that, but it's a categorical difference between him also trying to trick people vs merely messing up and getting in extra trouble because his advisor pissed off the kernel team earlier)
In some ways this reminds me of a phishing exercise at work that was done under the guise of security awareness training. Is it just me or is there a pattern of behavioral issues among security people and a lack of empathy.
If they go too far and impact day-to-day work, or if people complain, the executives understand what is going on and who they can talk to about improving the process.
And if it feels lacking in empathy, consider that the company's management needs to sign off on any (legal) phishing tests that their employees are subjected to.
All these researchers had to do was reach out to Greg, loop him in the study to get his go ahead for it.
Now that several of these 'patches' are now in stable and in fact did introduce issues is a big problem and a major waste of time for the maintainers.
The open letter: https://lore.kernel.org/lkml/CAK8KejpUVLxmqp026JY7x5GzHU2YJL...
The response: https://lore.kernel.org/lkml/YIV+pLR0nt94q0xQ@kroah.com/
This kind of probing is more akin to an "operation" funded by government rather than research funded by government.
https://www.zdnet.com/article/the-linux-foundations-demands-...
UoM got caught doing wrong. Their "why" doesn't help because their actions are rightly seen by their targets as malicious. This is like breaking into a store, trying to steal something, getting caught and them saying "well done! We were just testing!" Yet this was without any agreement that such testing was prior sanctioned: their actions must be seen as malicious. They set out to deliberately do something where they could be caught. They did get caught. They didn't have permission to test.
They need to actually apologize and make amends. Otherwise they might get the idea that trying again is acceptable. The next time they do this things will be even worse.
So what was the damage or harm? Loss of trust is one big ticket item. They wasted other's time and effort. Some important feature or functionality was delayed and is likely still being delayed. Or documentation not written. Even something as simple as "enthusiasm to volunteer" might be damaged. These aren't trivial aspects to be ignored.
It does not seem to me that they have thought much about what they are saying. For example, in a project like Linux, there is new code, but they majority of contributions, especially small fixes (as compared to new drivers and rewrites of subsystems), are very likely bug fixes. And many bugs can intruduce security issues, too, even if that might not be obvious. Therefore, it is not clear at all that making Linux more "closed" (which seems to be the solution which the UMN people and their friends suggest) would make it any more secure. More scrutiny by top maintainers means more time is needed; needing more time means automatically less fixes can be applied, since the maintainer's time is the bottleneck; applying less fixes means more open bugs, and having more open bugs can at least easily mean to have more security issues.
There are plenty of closed source projects with a security which is clearly worse than Linux. And not only that, but core Internet security infrastructure like DNS or OpenSSL is open source. The issue is rather that companies cannot be arsed to support that infrastructure they depend on. It would be interesting if the UMN people could come up with research or a hack which motivates them to change. (OTOH, one mayor windows worm per decade such as locky, wannacry, petya, tiny banker, cryptolocker, flame, and so on may be sufficient for the latter in the long run.)
There are several reasons:
* C is not an easy language that makes it hard to write unsafe code
* hardware can fail in any unspecified way
* the kernel community prefers kludges over simpler, visibly more robust code if the latter has performance impact (The biggest example of this is of course choosing a monolithic kernel instead of a microkernel.)
My performance and scalability requirements are much less than what the Linux kernel can do. So if there were a serious competitor based on code that humans can understand, my choice would be clear. In lack of such option I continue to use Linux and hope that nothing too bad happens...
It was a bulk revert; not "we can't tell if these are bad" but "we're reverting all of these until we can re-review them". Or, if you're claiming they shouldn't have been merged in the first place if it wasn't possible to be 100% confident in them... that's not how humans work, nor programming. Even if you specify your code in coq or something, you still have to verify that the model is correct and there's still room for human error.
> the kernel community prefers kludges over simpler, visibly more robust code if the latter has performance impact (The biggest example of this is of course choosing a monolithic kernel instead of a microkernel.)
Yes, they should have followed in the footsteps of all those other successful feature-complete microkernels like, um.... maybe QNX? Maybe? To be fair, I don't know that there's a reason why microkernels seem stuck in the "theoretically great but nobody's made one that works in the real world", but here we are.
So in broad strokes: You seem to be very confident that if kernel devs would just Do It Right then they could eliminate all bugs and malicious code, but this seems unlikely.
You wrote that, I didn't. The Linux kernel has chosen its route, I am not going tell them that they should do different.
All I say is should be possible to write a kernel that is reviewable by more people. It will be less scalable and less performant. But that's a price I might be ready to pay both for myself and for my work. My machines are hardly ever limited by kernel performance. Others might certainly have use cases where that is not true.
You should just run Minix.
I run Linux because it's the best option available today. That does not mean it's the best option conceivable if you value simplicity and visible robustness over performance and scalability.