I don't think there have been any recent comments from anyone at U.Mn. So, back when the original research (happened last year) the following clarification was offered by Qiushi Wu and Kangjie Lu which atleast paints their research in somewhat better light: https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
That said the current incident seems to have gone beyond the limits of that one and is a new incident. I just thought it would be fair to include their "side"
(3). We send the incorrect minor patches to the Linux community through email to seek their feedback.
(4). Once any maintainer of the community responds to the email, indicating “looks good”, we
immediately point out the introduced bug and request them to not go ahead to apply the patch. At the
same time, we point out the correct fixing of the bug and provide our proper patch. In all the three cases,
maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. This
way, we ensure that the incorrect patches will not be adopted or committed into the Git tree of Linux.
------------------------
But this shows a distinct lack of understanding of the problem:
> This is not ok, it is wasting our time, and we will have to report this,
> AGAIN, to your university...
------------------------
You do not experiment on people without their consent. This is in fact the very FIRST point of the Nuremberg code:
1. The voluntary consent of the human subject is absolutely essential.
Holy cow!! I'm a researcher and don't understand how they thought it would be okay to not do an IRB, and how an IRB would not catch this. The linked PDF by the parent post is quite illustrative. The first few paras seem to be downplaying the severity of what they did (did not introduce actual bugs into the kernel) but that is not the bloody problem. They experimented on people (maintainers) without consent and wasted their time (maybe other effects too .. e.g. making them vary of future commits from universities)! I'm appalled.
Even if those they did get approved were actual security holes (not benign decoys), all that it validates is no human is infallible. Well CONGRATULATIONS.
Right. And you would need a larger sample size to determine what % of the time that occurs, on average. But even then, is that useful and valid information? And is it actionable? (And if so, what is the cost of the action, and the opportunity cost of lost fixes in other areas?)
Open Source is not water proof if known committer, from well known faculty (in this case University of Minnesota) decides to send buggy patches. However, this was catched relatively quickly, but the behavior even after being caught is reprehensible:
> You, and your group, have publicly admitted to sending known-buggy patches to see how the kernel community would react to them, and published a paper based on that work.
>
> Now you submit a new series of obviously-incorrect patches again, so what am I supposed to think of such a thing?
If they kept doing it even after being caught, is beyond understandable.
Maybe they should conduct a meta-experiment where they submit unethical experiments for IRB review. Immediately when the IRB approves the proposal, they withdraw, pointing out the ways in which it would be unethical.
Meta-meta-experiment: submit the proposal above for IRB review and see what happens.
Do IRBs typically have a process by which you can file a complaint from outside the university? Maybe they never thought they would need to even check up on computer science faculty...
If you actually read the PDF linked in this thread:
* Is this human research? This is not considered human research. This project studies some issues with the patching process instead of individual behaviors, and we did not collect any personal information. We send the emails to the Linux community and seek community feedback. The study does not blame any maintainers but reveals issues in the process. The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained).
The irony is that the IRB process failed in the same way that the commit review process did. We're just missing the part where the researchers tell the IRB board they were wrong immediately after submitting their proposal for review.
In any university I've ever been to, this would be a gross violation of ethics with very unpleasant consequences. Informed consent is crucial when conducting experiments.
If this behaviour is tolerated by the University of Minnesota (and it appears to be so) then I suppose that's another institution on my list of unreliable research.
I do wonder what the legal consequences are. Would knowingly and willfully introducing bad code constitute a form of vandalism?
>>>On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits
Qiushi Wu, and Kangjie Lu.
To appear in Proceedings of the 42nd IEEE Symposium on Security and Privacy (Oakland'21). Virtual conference, May 2021.
IEEE is a garbage organization. Or atleast their India chapter is. 3 out of 5 professors in our university would recommend to avoid any paper published by Indians from IEEE. Here in India, publishing trash papers with the help of one's 'influence' is a common occurrence
IMNAL. In addition to possibly cause the research paper retracted due to the ethical violation, I think there are potentially civil or even criminal liability here. The US law on hacking is known to be quite vague (see Aaron Swartz’s case for example)
In this post they say the patches come from a static analyser and they accuse the other person of slander for their criticisms
> I respectfully ask you to cease and desist from making wild accusations that are bordering on slander.
> These patches were sent as part of a new static analyzer that I wrote and it's sensitivity is obviously not great. I sent patches on the hopes to get feedback. We are not experts in the linux kernel and repeatedly making these statements is disgusting to hear.
They did not say that they were hoping for feedback on their tool when they submitted the patch, they lied about their code doing something it does not.
>How does that fit in with your explanation?
It fits in the narrative of doing hypocritical changes to the project.
But lashing out when confronted after the fact? (I can't figure out how to browse to the messages that contain said purported 'slander' - maybe it is indeed terrible slander). Normally after the show is over one stops with the performance...
edit: oh, ok I guess that post with the accusations was mid-performance? Not inconsistent, so, maybe (I'm still not clear what the timeline is).
They obviously were _NOT_ created by a static analysis tool that is of
any intelligence, as they all are the result of totally different
patterns, and all of which are obviously not even fixing anything at
all. So what am I supposed to think here, other than that you and your
group are continuing to experiment on the kernel community developers by
sending such nonsense patches?
When submitting patches created by a tool, everyone who does so submits
them with wording like "found by tool XXX, we are not sure if this is
correct or not, please advise." which is NOT what you did here at all.
You were not asking for help, you were claiming that these were
legitimate fixes, which you KNEW to be incorrect.
They apparently didn't consider this "human research"
As I understand it, any "experiment" involving other people that weren't explicitly informed of the experiment before hand needs to be a lot more carefully considered than what they did here.
> You do not experiment on people without their consent.
Exactly this. Research involving human participants is supposed to have been approved by the University's Institutional Review Board; the kernel developers can complain to it: https://research.umn.edu/units/irb/about-us/contact-us
It would be interesting to see what these researches told the IRB they were doing (if they bothered).
Edited to add:
From the link in GP: "The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained)"
Okay so this IRB needs to be educated about this. Probably someone in the kernel team should draft an open letter to them and get everyone to sign it (rather than everyone spamming the IRB contact form)
These two sentences seem contradictory from the author's response is contradictory: " The IRB of UMN reviewed the study and determined that this is not human research (a
formal IRB exempt letter was obtained). Throughout the study, we honestly did not think this is human research, so we did not apply for an IRB approval in the beginning."
I would guess their IRB had a quick sanity check process to ensure there was no human subject research in the experiment. This is actually a good thing if scientists use their ethics and apply good judgement. Now, whoever makes that determination does so based on initial documentation supplied by the researchers. If so, the researchers should show what they submitted to get the exemption.
Again, the implication is their University will likely make it harder to get exemptions after this fiasco. This mistake hurts everyone (be it indirectly). Although, and this is being quite facetious and macabre, the researchers have inadvertently exposed a bug in their own institutions IRB process!
Combined with their lack of awareness of a possible breach of ethics in their response to Greg, I find it hard to believe they did not mislead the UMN IRB.
I hope they release what they submitted to the IRB to receive that exemption and there are some form of consequences if the mistake is on their part.
1. You have to submit for review any work involving human subjects before you start interacting with them. The authors clearly state that they sought retroactive approval after being questioned about their work. That would be a big red flag for my IRB and they wouldn't approve work retroactively.
2. There are multiple levels of IRB approval. The lowest is non regulated, which means that the research falls outside of human subject research. Individual researchers can self-certify work as non regulated or get a non-regulated letter from their IRB.
From there, it goes from exempt to various degrees of regulated. Exempt research means that it is research involving human subjects that is exempt from continued IRB review past the initial approval. That means that IRB has found that their research involves human subjects but falls within one (or more) of the exceptions for continued review.
In order to be exempt, a research project must meet one of the exemptions categories (see here https://hrpp.msu.edu/help/required/exempt-categories.html for a list). The requirements changed in 2018, so what they had to show depends on when they first received their exempt status.
The bottom line is that the research needs to (a) have less than minimal risks for participants and (b) needs to be benign in nature. In my opinion, this research doesn't meet these requirements as there are significant risks to participants to both their professional reputation and future employability for having publicly merged a malicious patch. They also pushed intentionally malicious patches, so I am not sure if the research is benign to begin with.
3. Even if a research project is found exempt from IRB review, participants still need to consent to participate in it and need to be informed of the risks and benefits of the research project. It seems that they didn't consent their participants before their participation in the research project. Consent letters usually use a common template that clearly states the goals for the research project, lists the possible risks and benefits of participating in it, states the name and contact information of the PI, and data retention policies. IRB could approve projects without proactive participant consent but those are automatically "bumped up" to full IRB approval and approvals are given only in very specific circumstances. Plus, once a participant removes their consent to participate in a research project, the research team needs to stop all interactions with them and destroy all data collected from them. It seems that the kernel maintainers did not receive the informed consent materials before starting their involvement with the research project and have expressed their desire not to participate in the research after finding out they were participating in it, so the interaction with them should stop and any data collected from them should be destroyed.
4. My impression is that they got IRB approval on a technicality. That is, their research is on the open source community and its processes rather than the individual people that participate in them. My impression of their paper is that they are very careful in addressing the "Linux community" and they really never talk about their interaction with people in the paper (e.g., there is no data collection section or a description of their interactions on the mailing list). Instead, it's my impression that they present the patches that they submitted as happening "naturally" in the community and that they are describing publicly available interactions. That seems to be a little misleading of what actually happened and their role in producing and submitting the patches.
I’m interested in MSU’s list of exempt categories. Most of them are predicated on the individual subjects not being identifiable. Since this research is being done on a public mailing list that is archived and available for all to read, it is trivial to go through the archive and find the patches they quote in their paper to find out who reviewed them, and their exact responses. Would that disqualify the research from being exempt, even if the researchers themselves do not record that data or present it in their paper?
What if they did a survey of passers–by on a public street, that might be in view of CCTV operated by someone else?
The federal government has updated the rules for exemption in 2018. The MSU link is more of a summary than the actual rules.
The fact that a mailing list is publicly available is what made me worry about the applicability of any sort of exemption. In order for human subject research to be exempt from IRB review, the research needs to be deemed less than minimal risk to participants.
The fact that their experiment happens in public and that anyone can find their patches and individual maintainers' responses (and approval) of them makes me wonder if the participants are at risk of losing professional reputation (in that they approved a patch that was clearly harmful) or even employment (in that their employer might find out about their participation in this study and move them to less senior positions as they clearly cannot properly vet a patch). This might be extreme, but it is still a likely outcome given the overall sentiment of the paper.
All research that poses any harm to participants has to be IRB approved and the researchers have to show that the benefits to participants (and the community at large) surpass the individual risks. I am still not sure what benefits this work has to the OSS community and I am very surprised that this work did not require IRB supervision at all.
As far as work on a public street is concerned, IRB doesn't regulate common activities that happen in public and for which people do not have a reasonable expectation of privacy. But, as soon as you start interacting with them (e.g., intervene in their environment), IRB review is required.
You can read and analyze a publicly available mailing list (and this would even qualify as non human subject research if the data is properly anonymized) without IRB review or at most a deliberation of exempt status but you cannot email the mailing list yourself as a researcher as the act of emailing is an intervention that changes other people's environment, therefore qualifying as human subject research.
It does seem rather unethical, but I must admit that I find the topic very interesting. They should definitely have asked for consent before starting with the "attack", but if they did manage to land security vulnerabilities despite the review process it's a very worrying result. And as far as I understand they did manage to do just that?
I think it shows that this type of study might well be needed, it just needs to be done better and with the consent of the maintainers.
“Hey, we are going to submit some patches that contain vulnerabilities. All right?”
If they do so, the maintainers become more vigilant and the experiment fails. But, the key to the experiment is that maintainers are not vigilant as they should be. It’s not an attack to the maintainers though, but to the process.
In penetration testing you are doing the same thing, but you get the go-ahead for someone responsible for the project or organization since they are interested in the results as well.
A red team without approval is just a group of criminals. They must have been able to find active projects with a centralized leadership they could ask for permission.
I don’t know much about penetration testing so excuse me for the dumb question: are you required to disclose the exact methods that you’re going to use?
It depends on the organization. Most that I've worked with have said everything is fine except for social engineering, but some want to know every tool you'll be running, and every type of vulnerability you'll try to exploit.
Yes. You have agreements about what is fair game and what is off limits. It can be that nothing can be physically altered, what times of day or office locations are OK, if it should only be a test against web services or anything in between.
Do you? You have agreement with part of the company and work it out with them, but does this routinely include the people who would be actively looking for your intrusion and trying to catch it? Often that is handled by automated systems which are not updated to have any special knowledge about the up coming penetration test and most of those supporting the application aren't made aware of the details either. The organization is aware, but not all of the people who may be impacted.
What you do during pentesting is against the law, if you do not discuss this with your client. You're trying to gain access to a computer system that you should have no access to. The only reason this is OK, is that you have prior permission from the client to try these methods. Thus, it is important to discuss the methods used when you are executing a pentest.
With every pentesting engagement I've had, there always were rules of engagement, and what kind of things you are and are not allowed to do. They even depend on what kind of test you are doing. (for example: if you're testing bank software, it matters a lot if you test against their production environment or their testing environment)
"We're going to, as part of a study, submit various patches to the kernel and observe the mailing list and the behavior of people in response to these patches, in case a patch is to be reverted as part of the study, we immediately inform the maintainer."
But it wouldn't let maintainers know what is happening, it only informs them that someone will be submitting some patches, some of which might not be merged. It doesn't push people into vigilance onto a specific detail of the patch and doesn't alert them that there is something specific. If you account for that in your experiment priors, that is entirely fine.
>Your message would push maintainers to put even more focus on the patches, thus invalidating the experiment.
The Tuskegee Study wouldn't have happened if its participants were voluntarily, and it's effects still haunt the scientific community today. The attitude of "science by any means, including by harming other people" is reprehensible and has lasting consequences for the entire scientific community.
However, unlike the Tuskegee Study, it's totally possible to have done this ethically by contacting the leadership of the Linux project and having them announce to maintainers that anonymous researchers may experiment with the contribution process, and allowing them to opt out if they do not consent, and to ensure that harmful commits never reach stable from these researchers.
The researchers chose to instead lie to the Linux project and introduce vulnerabilities to stable trees, and this is why their research is particularly deplorable - their ethical transgressions and possibly lies made to their IRB were not done out of any necessity for empirical integrity, but rather seemingly out of convenience or recklessness.
And now the next group of researchers will have a harder time as they may be banned and every maintainer now more closely monitors academics investigating open source security :)
I don't want to defend what these researchers did, but to equate infecting people with syphilis to wasting a bit of someones time is disingenuous. Informed consent is important, but only if the magnitude of the intervention is big enough to warrant reasonable concerns.
>to wasting a bit of someones time is disingenuous
This introduced security vulnerabilities to stable branches of the project, the impact of which could have severely affected Linux, its contributors, and its users (such as those who trust their PII data to be managed by Linux servers).
The potential blast radius for their behavior being poorly tracked and not reverted is millions if not billions of devices and people. What if a researcher didn't revert one of these commits before it reached a stable branch and then a release was built? Linux users were lucky enough that Greg was able to revert the changes AFTER they reached stable trees.
There was a clear need of informed consent of *at least* leadership of the project, and to say otherwise is very much in defense of or downplaying the recklessness of their behavior.
I acknowledged that lives are not at play, but that doesn't mean that the only consequence or concern here was wasting the maintainers time, especially when they sought an IRB exemption for "non-human research" when most scientists would consider this very human research.
If the attack surface is large enough and the duration of the experiment long enough it'll return to baseline soon enough I think. It's a reasonable enough compromise. After all if the maintainers are not already considering that they might be under attack I'd argue that something is wrong with the system, a zero-day in the kernel would be invaluable indeed.
And well, if the maintainers become more vigilant in the long run it's a win/win in my book.
Yes, and people generally don't seem upset by viral Youtube social experiments. The Nuremberg code may be the status quo and nothing more. No one here is trying to justify the code on its merits, just blindly quoting it as an authority.
Here's another idea: If it's ethical to do it in a non-experimental context, it's also ethical to do it in an experimental context. So if it's OK to walk up to a stranger and ask them a weird question, it's also OK to do it in the context of a Youtube social experiment. Anything other than this is blatantly anti-scientific IMO.
In the last year when it came to experimental Covid-19 projections, modeling and population-wide recommendations from major academic centers, the IRB's were silent and academics did essentially whatever they wanted, regardless of "consent" from the populations that were the subjects of their speculative hypotheses.
You could argue that they are doing the maintainers a favor. Bad actors could exploit this, and the researchers are showing that maintainers are not paying enough attention.
If I were at the receiving end, I’d think checking a patch multiple times before accepting it.
I'm sure that they thought this. But this is a bit like doing unsolicited pentests or breaking the locks on somebody's home at night without their permission. If people didn't ask for it and consent, it is unethical.
And further, pretty much everybody knows that malicious actors - if they tried hard enough - would be able to sneak through hard to find vulns.
> You do not experiment on people without their consent.
Applied strictly, wouldn’t every single A/B test done by a product team be considered unethical?
From a common sense standpoint, it seems to me this is more about medical experiments. Yesterday I put some of my kids toys away without telling them to see if they’d notice and still play with them. I don’t think I need IRB approval.
"I was testing how the bank processes having a ton of cash taken out by someone without an account, I wasn't testing the staff or police response, geez!"
> wouldn’t every single A/B test done by a product team be considered unethical?
Potentially yes, actually.
I still think it should be possible to run some A/B tests, but a lot depends on the underlying motivation. The distance between such tests and malicious psychological manipulation can be very, very small.
> Applied strictly, wouldn’t every single A/B test done by a product team be considered unethical?
Assuming this isn't being asked as a rhetorical question, I think that's exactly what turned the now infamous Facebook A/B test into a perceived unethical mass manipulation of human emotions. A lot of folks are now justifiably upset and skeptical of Facebook (and big tech) as a result.
So to answer your question: yes, if that test moves into territory that would feel like manipulation once the subject is aware of it. Maybe especially so because users are conceivably making a /choice/ to use said product and may switch to an alternative (or simply divest) if trust is lost.
IRB (as in Institutional Review Board) is a local (as in each research institution has one) regulatory board that ensures that any research conducted by people employed by the institution follows the federal government's common rule for human subject research. Most institutions receiving federal funding for research activities have to show that the funded work follows common rule guidelines for interaction with human subjects.
It is unlikely that a business conducting A/B testing or a parent interacting with their children are receiving federal funds to support it. Therefore, their work is not subject to IRB review.
Instead, if you are a researcher who is funded by federal funds (even if you are doing work on your own children), you have to receive IRB approval for any work involving human interaction before you start conducting it.
It should be for all science done for the sake of science, not just medical work. When I did experiments that just involved people playing an existing video game I still had to get approval from IRB and warn people of all the risks that playing a game is associated with (like RSI, despite the gameplay lasting < 15 minutes).
Researchers at a company could arguably be deemed as engaging in unethical research and barred from contributing to the scientific community due to unethical behavior. Even doing experiments on your kids may be deemed crossing the line.
The question I have is when does it apply. If you research on your own kids but never publish, is it okay? Does the act of attempting to publish results retroactively make an experiment unethical? I'm not certain these things have been worked out because of how rare people try to publish anything that wasn't part of an official experiment.
> Applied strictly, wouldn’t every single A/B test done by a product team be considered unethical?
I would argue that ordinary A/B tests, by their very nature, are not "experiments" in the sense that restriction is intended for, so there is no reason for them to be considered unethical.
The difference between an A/B test and an actual experiment that should require the subjects' consent is that either of the test conditions, A or B, could have been implemented ordinarily as part of business as usual. In other words, neither A nor B by themselves would need a prior justification as to why they were deployed, and if the reasoning behind either of them was to be disclosed to the subjects, they would find them indistinguishable from any other business decision.
Of course, this argument would not apply if the A/B test involved any sort of artificial inconvenience (e.g. mock errors or delays) applied to either of the test conditions. I only mean A/B tests designed to compare features or behaviours which could both legitimately be considered beneficial, but the business is ignorant of which.
But this is all a lie. If you read the linked thread you till see that they refused to admit to their experiment and even sent a new, differently broken patch.
There is sometimes an exception for things like interviews when n is only a couple of people. This was clearly unethical and it’s certain that at least some of those involved knew that. It’s common knowledge universities.
> You do not experiment on people without their consent. This is in fact the very FIRST point of the Nuremberg code:
> 1. The voluntary consent of the human subject is absolutely essential.
Which is rather useless, as for many experiments to work, participants have to either be lied to, or kept in the dark as to the nature of the experiment, so whatever “consent” they give is not informed consent. They simply consent to “participate in an experiment” without being informed as to the qualities thereof so that they truly know what they are signing up for.
Of course, it's quite common in the U.S.A. to perform practice medical checkups on patients who are going under narcosis for an unrelated operations, and they never consented to that, but the hospitals and physicians that partake in that are not sanctioned as it's “tradition”.
Know well that so-called “human rights” have always been, and shall always be, a show of air that lack substance.
Most one can find of it also only deals with “intimate parts”; I am quite sceptical that this is the only thing that medical students require practice on and I think it more likely that the media only cares in this case and that in fact it is routine with many more body parts.
> You do not experiment on people without their consent. This is in fact the very FIRST point of the Nuremberg code:
> 1. The voluntary consent of the human subject is absolutely essential.
The Nuremberg code is explicitly about medical research, so it doesn't apply here.
More generally, I think that the magnitude of the intervention is also relevant, and that an absolutist demand for informed consent in all - including the most trivial - cases is quite silly.
Now, in this specific case I would agree that wasting people's time is an intervention that's big enough to warrant some scrutiny, but the black-and-white way of some people to phrase this really irks me.
PS: I think people in these kinds of debate tend to talk past one another, so let me try to illustrate where I'm coming from with an experiment I came across recently:
To study how the amount of tips waiters get changes in various circumstances, some psychologists conducted an experiment where the waiter would randomly either give the guests some chocolate with the bill, or not (control condition)[0]
This is, of course, perfectly innocuous, but an absolutist claim about research ethics ("You do not experiment on people without their consent.") would make research like this impossible without any benefit.
I'm confused - how is this an experiment on humans? Which humans? As far as I can tell, this has nothing to do with humans, and everything to do with the open-source review process - and if one thinks that it counts as a human experiment because humans are involved, wouldn't that logic apply equally to pentesting?
For that matter, what's the difference between this and pentesting?
Penetration testing is only ethical when you are hired by the organization you are testing.
Also, IRB review is only for research funded by the federal government. If you’re testing your kid’s math abilities, you’re doing an experiment on humans, and you’re entirely responsible for determining whether this is ethical or not, and without the aid of an IRB as a second opinion.
Even then, successfully getting through the IRB process doesn’t guarantee that your study is ethical, only that it isn’t egregiously unethical. I suspect that if this researcher got IRB approval, then the IRB didn’t realize that these patches could end up in a released kernel. This would adversely affect the users of billions of Linux machines world–wide. Wasting half an hour of a reviewer’s time is not a concern by comparison.
>>>On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits
Qiushi Wu, and Kangjie Lu.
To appear in Proceedings of the 42nd IEEE Symposium on Security and Privacy (Oakland'21). Virtual conference, May 2021.
If the original research results in a paper and IEEE conference presentation, why not? There's no professional consequences for this conduct, apparently.
Given that this conference hasn't happened yet, there should still be time for the affected people to report the inappropriate conduct to the organizers and possibly get the paper pulled.
FYI .. many ACM conferences are now asking explicitly if an IRB was required, and if so, was it received. This does not prevent researchers from saying IRB doesn't apply, but perhaps it can be caught during peer review.
Btw .. I posted a few times on the thread, and want to acknowledge that researchers are humans, and humans do make mistakes. Thankfully in this case, the direct consequence was time wasted, and this is a teaching moment for all involved. In my humble opinion, the researchers should acknowledge in stronger terms they screwed up, do a post-mortem on how this happened, and everyone (including the researchers) should move on with their lives.
Given current academia which puts a significant negative on discussing why research failed, I doubt your idea of post-mortems, public or private, will gain any traction.
The same group did the same thing last year (that's what the paper is about - may 2021 paper obviously got written/submitted last year), when the preprint got published they got criticized publicly. And now they are doing it again, so its not just a matter of "acknowledge they screwed up".
I just wanted to highlight that S&P/Oakland is one of the top 3 or 4 security conferences in the security community in academia. This is a prestigious venue lending its credibility to this paper.
I would go even further and say that Oakland is the most prestigious security conference. That this kind of work was accepted is fairly baffling to me, since I'd expect both ethical concerns and also concerns about the "duh" factor.
I'm a little salty because I personally had two papers rejected by Oakland on the primary concern that their conclusions were too obvious already. I'd expect everybody to already believe that it wouldn't be too hard to sneak vulns into OSS patches.
If this is actually presented, someone present should also make the following clear: "As a result of the methods used by the presenters, the entire University of Minnesota system has been banned from the kernel development process and the kernel developers have had to waste time going back and re-evaluating all past submissions from the university system. The kernel team would also like to advise other open-source projects to carefully review all UMN submissions in case these professors have simply moved on to other projects."
Their first suggestion to the process is pure gold:"OSS projects would be suggested to update the code of conduct, something like “By submitting
the patch, I agree to not intend to introduce bugs”"
Like somebody picking your locks, and suggesting, 'to stop this one approach would be to post a sign "do not pick"'
This does paint there side better, but it also makes me wonder if they're being wrongly accused of this current round of patches? That clarification says that they only submitted 3 patches, and that they used a random email address when doing so (so presumably no @umn.edu).
These ~200 patches from UMN being reverted might have nothing to do with these researchers at all.
Hopefully someone from the university clarifies what's happening soon before the angry mob tries to eat the wrong people.
But that question is as deep and important to answer as yours :D What can anyone hope to accomplish by doing fake research ? Progress, wealth, peer approval, mating, pleasure ?
So answering that they hope to get more material for papers, which is the only goal of researchers (and their main KPI), is quite deeper an answer than the question required.
My guess is: a journal that does not focus on studies of human behavior and whose editors are a) not aware of the ethical problems or b) happy to ignore ethics concerns if the publication is prone to receive much attention (which it is).
>>>On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits
Qiushi Wu, and Kangjie Lu.
To appear in Proceedings of the 42nd IEEE Symposium on Security and Privacy (Oakland'21). Virtual conference, May 2021.
They apparently made a tool to find vulnerabilities that could later lead to bugs is a different patch was introduced.
And for some insane reason, they decided to test if these kinds of bugs would be caught by inventing some and just submitting the patches, without informing anyone beforehand.
The problem here is really that they’re wasting time of the maintainers without their approval. Any ethics board would require prior consent to this. It wouldn’t even be hard to do.
1) They identified vulnerabilities with a process
2) They contributed the correct code after showing the maintainer the security vulnerability they missed.
3) Getting the consent of the people behind the process would invalidate the results.
Go hack a random organization without a vulnerability disclosure program in place and see how much goodwill you have. There is a very established best practice in how to do responsible disclosure and this is far from it.
1) Contact a single maintainer and explore feasibility of the study
2) Create a group of maintainers who know the experiment is going to happen, but leave a certain portion of the org out of it
3) Orchestrate it so that someone outside of the knowledge group approves one or more of these patches
4) Interfere before any further damage is done
Besides, are you arguing that ends justify the means if the intent behind the research is valid?
> 1) Contact a single maintainer and explore feasibility of the study
That has the risk that the contacted maintainer is later accused of collaborating with saboteurs or that they consult others. Either very awful or possibly invalidates results.
> 2) Create a group of maintainers who know the experiment is going to happen, but leave a certain portion of the org out of it
Assuming the leadership agrees and won't break confidentiality, which they might if the results could make them look bad. Results would be untrustworthy or potentially increase complacency.
> 4) Interfere before any further damage is done
That was done, was it not?
> Besides, are you arguing that ends justify the means if the intent behind the research is valid?
The allegation being made on the mailing list is that some incorrect patches of theirs made it into git and even the stable trees. As there is not presently an enumeration of them, or which ones are alleged to be incorrect, I cannot state whether this is true.
But that's the claim.
edit: And looking at [1], they have a bunch of relatively tiny patches to a lot of subsystems, so depending on how narrowly gregkh means "rip it all out", this may be a big diff.
edit 2: On rereading [2], I may have been incorrectly conflating the assertion about "patches containing deliberate bugs" with "patches that have been committed". Though if they're ripping everything out anyway, it appears they aren't drawing a distinction either...
Perhaps I'm missing something obvious, but what's the point of all this subterfuge in the first place? Couldn't they just look at the history of security vulnerabilities in the kernel, and analyze how long it took for them to be detected? What does it matter whether the contributor knew ahead of time that they were submitting insecure code?
It's seems equivalent to vandalising Wikipedia to see how long it takes for someone to repair the damage you caused. There's no point doing this, you can just search Wikipedia's edits for corrections, and start your analysis from there.
> What does it matter whether the contributor knew ahead of time that they were submitting insecure code?
It's a specific threat model they were exploring: a malicious actor introducing vulnerability on purpose.
> Couldn't they just look at the history of security vulnerabilities in the kernel, and analyze how long it took for them to be detected?
Perhaps they could. I guess it'd involve much more work, and could've yielded zero results - after all, I don't think there are any documented examples when a vulnerability was proven to have been introduced on purpose.
> what's the point of all this subterfuge in the first place?
Control over the experimental setup, which is important for validity of research. Notice how most research involves gathering up fresh subjects and controls - scientists don't chase around the world looking for people or objects that, by chance, already did the things they're testing for. They want fresh subjects to better account for possible confounders, and hopefully make the experiment reproducible.
(Similarly, when chasing software bugs, you could analyze old crash dumps all day to try and identify a bug - and you may start with that - but you always want to eventually reproduce the bug yourself. Ultimately, "I can and did that" is always better than "looking at past data, I guess it could happen".)
> It's seems equivalent to vandalising Wikipedia to see how long it takes for someone to repair the damage you caused.
Honestly, I wouldn't object to that experiment either. It wouldn't do much harm (little additional vandalism doesn't matter on the margin, the base rate is already absurd), and could yield some social good. Part of the reason to have public research institutions is to allow researchers to do things that would be considered bad if done by random individual.
Also note that both Wikipedia and Linux kernel are essentially infrastructure now. Running research like this against them makes sense, where running the same research against a random small site / OSS project wouldn't.
> It's a specific threat model they were exploring: a malicious actor introducing vulnerability on purpose.
But does that matter? We can imagine that the error-prone developer who submitted the buggy patch just had a different mindset. Nothing about the patch changes. In fact, a malicious actor is explicitly trying to act like an error-prone developer and would (if skilled) be indistinguishable from one. So we'd expect the maintainer response to be the same.
> I guess it'd involve much more work, and could've yielded zero results - after all, I don't think there are any documented examples when a vulnerability was proven to have been introduced on purpose.
In line with UncleMeat's comment, I'm not convinced it's of any consequence that the security flaw was introduced deliberately, rather than by accident.
> scientists don't chase around the world looking for people or objects that, by chance, already did the things they're testing for
That doesn't sound like a fair description of what's happening here.
There are two things at play. Firstly, an analysis of the survival function [0] associated with security vulnerabilities in the kernel. Secondly, the ability of malicious developers to deliberately introduce new vulnerabilities. (The technical specifics detailed in the paper are not relevant to our discussion.)
I'm not convinced that this unethical study demonstrates anything of interest on either point. We already know that security vulnerabilities make their way into the kernel. We already know that malicious actors can write code with intentional vulnerabilities, and that it's possible to conceal these vulnerabilities quite effectively.
> Honestly, I wouldn't object to that experiment either. It wouldn't do much harm (little additional vandalism doesn't matter on the margin, the base rate is already absurd), and could yield some social good.
That's like saying It's ok to deface library books, provided it's a large library, and provided other people are also defacing them.
Also, it would not yield a social good. As I already said, it's possible to study Wikipedia's ability to repair vandalism, without committing vandalism. This isn't hypothetical, it's something various researchers have done. [0][1]
> Part of the reason to have public research institutions is to allow researchers to do things that would be considered bad if done by random individual.
It isn't. Universities have ethics boards. They are held to a higher ethical standard, not a lower one.
> Running research like this against them makes sense
No one is contesting that Wikipedia is worthy of study.
It potentially has long term negative impact on the experimental subjects involved and has no research benefit. The researchers should be removed from university and the university itself should be sued and lose enough money that they act more responsible in the future. It’s a very slippery slope to from casual irb wavers to Tuskegee experiments.
> 3) Orchestrate it so that someone outside of the knowledge group approves one or more of these patches
Isn't this part still experimenting on people without their consent? Why does one group of maintainers get to decide that you can experiment on another group?
It is, but that is how security testing goes about in general (in the commercial world.) Of its application to research and ethics, I’m not much of an authority.
In general you try to obtain consent from their boss, so that if the people you pentested on complain you can point to their boss and say "Hey they agreed to it" and that will be the end of the story. In this case it's not clear who the "boss" is but something like the Linux Foundation would be a good start.
If you can’t make an experiment without violating ethical standards, you simply don’t do it, you can’t use this as an excuse to violate ethical standards.
Kernel maintainers are volunteering their time and effort to make Linux better, not to be entertaining test subjects for the researchers.
Even if there is no ethical violation, they are justified to be annoyed at having their time wasted, and taking measures to discourage and prevent such malicious behaviour in the future.
Given the importance of the Linux kernel, there has to be a way to make contributions safer. Some people even compare it to the "water supply" and others bring in "national security".
> they are justified to be annoyed at having their time wasted, and taking measures to discourage and prevent such malicious behaviour in the future.
"Oh no, think of the effort we have to spend at defending a critical piece of software!"
In every commercial pentest I have been in, you have 1-2 usually senior employees on the blue team in the know. They have the job to stop employees from going to far on defense, as well as stop the pentesters from going too far. The rest of the team stays in the dark to test their response and observation.
In this case, in my opinion, a small set of maintainers and linus as "management" would have to be in the know to e.g. stop a merge of such a patch once it was accepted by someone in the dark.
> 3) Getting the consent of the people behind the process would invalidate the results.
This has not been a valid excuse since the 1950s. Scientists are not allowed to ignore basic ethics because they want to discover something. Deliberately introducing bugs into any open source project is plainly unethical; doing so in the Linux kernel is borderline malicious.
We should ban A/B testing then. Google didn’t tell me they were using me to understand which link color is more profitable for them.
There are experiments and experiments. Apart from the fact that they provided the fix right away, they didn’t do anyone harm.
And, by the way, it’s their job. Maintainers must approve patches after they ensured that the patch is fine. It’s okay to do mistakes, but don’t tell me “you’re wasting my time” after I showed you that maybe there’s something wrong with the process. If anything, you should thank me and review the process.
If your excuse is “you knew the patch was vulnerable”, then how are you going to defend the project from bad actors?
Several of the patches are claimed to have landed in stable. Also, distributions and others (like the grsecurity people) pick up lkml patches that are not included in stable but might have security benefits. So even just publishing such a patch is harmful. Also, fixes were only provided to the maintainers privately as it seems, and unsuccessfully. Or not at all.
> If your excuse is “you knew the patch was vulnerable”, then how are you going to defend the project from bad actors?
Exactly the same way as without that "research".
If you try to pry open my car door, I'll drag you to the next police station. "I'm just researching the security of car doors" won't help you.
No bugs were introduced and they didn't intend to introduce any bugs. infact, they have resolved over 1000+ bugs in the linux kernel.
>> https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
"We did not introduce or intend to introduce any bug or vulnerability in the Linux kernel. All the
bug-introducing patches stayed only in the email exchanges, without being adopted or merged into any
Linux branch, which was explicitly confirmed by maintainers. Therefore, the bug-introducing patches in
the email did not even become a Git commit in any Linux branch. None of the Linux users would be
affected. The following shows the specific procedure of the experiment"
I don't think it's necessarily a bruised ego here - I think what upset him is that the paper was published a few months ago and yet, based on this patch, the author seems to still be attempting to submit deeply flawed patches to LKML, and complaining when people don't trust them to be innocent mistakes for some reason.
so instead of fixing the issue they found of being able to introduce backdoors in to their code, they are going to rollback thousand + of other bug fixes.
That's more of a story than what the researchers have done...
What would you do, if you had a group of patch authors who you didn't trust the contributions of anymore, other than setting aside the time for someone trusted to audit all 390 commits they've had since 2014?
How do you think social engineering audits work? You first coordinate with the top layer (in private, of course) and only after getting their agreement do you start your tests. This isn't any different.
You're right, and it is depressing how negative the reaction has been here. This work is the technical equivalent of "Sokalling", and it is a good and necessary thing.
The thing that people should be upset about is that such an important open source project so easily accepts patches which introduce security vulnerabilities. Forget the researchers for a moment - if it is this easy, you can be certain that malicious actors are also doing it. The only difference is that they are not then disclosing that they have done so!
The Linux maintainers should be grateful that researchers are doing this, and researchers should be doing it to every significant open source project.
> The thing that people should be upset about is that such an important open source project so easily accepts patches which introduce security vulnerabilities
They were trusting of contributors to not be malicious, and in particular, were trusting of a university to not be wholly malicious.
Sure, there is a possible threat model where they would need to be suspicious of entire universities.
But in general, human projects will operate under some level of basic trust, with some sort of means to establish that trust. To be able to actually get anything done; you cannot perfectly formally review everything with finite human resources. I don't see where they went wrong with any of that here.
There's also the very simple fact that responding to an incident is also a part of the security process, and broadly banning a group whole-cloth will be more secure than not. So both them and you are getting what you want it of it - more of the process to research, and more security.
If the changes didn't make it out to production systems, then it seems like the process worked? Even if some of it was due to admissions that would not happen with truly malicious actors, so too were the patches accepted because the actors were reasonably trusted.
Then they would not be accepting any patches from any contributors, as the only truly safe option when dealing with an explicitly and admittedly, or assumed known malicious actor is to disregard their work entirely. You cannot know the scope of a malicious plot in advance, and any benign piece of work can be fatal in some unknown later totality.
As with all human projects, some level and balance of trust and security is needed to get work done. And the gradient shifts as downstream forks have higher security demands / less trust, and (in the case of nation states) more resources and time to both move slower, validate changes and establish and verify trust.
> The problem here is really that they’re wasting time of the maintainers without their approval.
Not only that, but they are also doing experiments on a community of people which is against their interest and also could be harmful by creating mistrust. Trust is a big issue, without it it is almost impossible for people to work meaningfully together.
Yeah this actually seems more like sociological research except since it’s in the comp sci department the investigators don’t seem to be trained in acceptable (and legal) standards of conducting such research on human subjects. You definitely need prior consent when doing this sort of thing. Ideally this would be escalated to a research ethics committee at UMN because these researchers need to be trained in acceptable practices when dealing with human subjects. So to me it makes sense the subjects “opted out” and escalated to the university.
> We send the emails to the Linux communityand seek their feedback. The experiment is not to blame any maintainers but to reveal issues in the process. The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter. The experiment will not collect any personal data, individual behaviors, or personal opinions. It is limited to studying the patching process OSS communities follow, instead of individuals.
So they did think of that. Either they misconstrued their research or the IRB messed up. Either way, they can now see for themselves exactly how human a pissed off maintainer is.
> A lot of these have already reached the stable trees.
If the researchers were trying to prove that it is possible to get malicious patches into the kernel, it seems like they succeeded -- at least for an (insignificant?) period of time.
I tangentially followed the debacle unfold for a while and this particular thread now has lead to heated debates on some IRC channels I'm on.
While it is maybe "scientifically interesting", intentionally introducing bugs into Linux that could potentially make it into production systems while work on this paper is going on, could IMO be described as utterly reckless at best.
Two messages down in the same thread, it more or less culminates with the university e-mail suffix being banned from several kernel mailing lists and associated patches being removed[1], which might be an appropriate response to discourage others from similar stunts "for science".
> While it is maybe "scientifically interesting", intentionally introducing bugs into Linux that could potentially make it into production systems while work on this paper is going on, could IMO be described as utterly reckless at best.
I agree. I would say this is kind of a "human process" analog of your typical computer security research, and that this behavior is akin to black hats exploiting a vulnerability. Totally not OK as research, and totally reckless!
Yep. To take a physical-world analogy: Would it be okay to try and prove the vulnerability of a country's water supply by intentionally introducing a "harmless" chemical into the treatment works, without the consent of the works owners? Or would that be a go directly to jail sort of an experiment?
I share the researchers' intellectual curiosity about whether this would work, but I don't see how a properly-informed ethics board could ever have passed it.
> Would it be okay to try and prove the vulnerability of a country's water supply by intentionally introducing a "harmless" chemical into the treatment works, without the consent of the works owners?
The question should also be due to who's neglect they gained access to the "water supply". If you also truly want to make this comparison.
The question is also: "Will this research have benefits?" If the conclusion is "well, you can get access to the water supply and the only means to prevent it is to closely guard every brook, lake and river, needing half the population as guards". Well, then it is useless. And taking risks for useless research is unethical, no matter how minor those risks might be.
> If the conclusion is "well, you can get access to the water supply and the only means to prevent it is to closely guard every brook, lake and river, needing half the population as guards".
The US navy did actually basically this with some pathogens in the 50s: https://en.wikipedia.org/wiki/Operation_Sea-Spray ; the idea of 'ethical oversight' was not something a lot of scientists operated under in those days.
Out of interest, is there any way to have some sort of automated way to test this weak link that is human trust? (I understand how absurd this question is)
It's awfully scary to think about how vulnerabilities might be purposely introduced into this important code base (as well as many other) only to be exploited at a later date for an intended purpose.
Such as? Should we assume that every patch was submitted in bad faith and tries to sneakily introduce bugs?
The whole idea of the mailing list based submission process is that it allows others on the list to review your patch sets and point out obvious problems with your changes and discuss them, before the maintainer picks the patches up from the list (if they don't see any problem either).
As I pointed out elsewhere, there are already test farms and static analysis tools in place. On some MLs you might occasionally see auto generated mails that your patch set does not compile under configuration such-and-such, or that the static analysis bot found an issue. This is already a thing.
What happened here is basically a con in the patch review process. IRL con men can scam their marks, because most people assume, when they leave the house that the majority of the others outside aren't out there to get them. Except when they run into one where the assumption doesn't hold and end up parted from their money.
For the paper, bypassing the review step worked in some instances of the many patches they submitted because a) humans aren't perfect, b) have a mindset that most of the time, most people submitting bug fixes do so in good faith.
Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code?
Yes, I know that this review process isn't perfect, that there are problems and I'm not trying to dismiss any concerns.
But what technical measure would you propose that can effectively stop con men?
> Should we assume that every patch was submitted in bad faith and tries to sneakily introduce bugs?
Yes, especially for critical projects?
> Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code?
I don’t jump to the conclusion that the random contributor is evil. I do however think about the potential impact of the submitted patch, security or not, and I do assume a random contributor can sneak in subtle bugs, usually not intentionally, but simply due to a lack of understanding.
> > Should we assume that every patch was submitted in bad faith and tries to sneakily introduce bugs?
>
> Yes, especially for critical projects?
People don't act that way I described intentionally, or because they are dumb.
Even if you go in with the greatest paranoia and the best of intentions, most of the time, most of the other people don't act maliciously and your paranoia eventually returns to a reasonable level (i.e. assuming that most people might not be malicious, but also not infallible).
It's a kind of fatigue. It's simply human. No matter how often you say "DUH of course they should".
In my entire life, I have only met a single guy who managed to keep that "everybody else is potentially evil" attitude up over time. IIRC he was eventually prescribed something with Lithium salts in it.
Force the university to take reponsibility for screening their researchers. i.e. a blanket ban, scorched earth approach punishing the entire university's reputation is a good start.
People want to claim these are lone rogue researchers and good people at the university shouldn't be punished, but this is the only way you can reign in these types of rogues individuals: by getting the collective reputation of the whole university on the line to police their own people. Every action of individual researchers must be assumed to be putting the reputation of the university as a whole on the line. This is the cost of letting individuals operate within the sphere of the university.
Harsh, "over reaction" punishment is the only solution.
The only real fix for this is to improve tooling and/or programming language design to make these kinds of exploits more difficult to slip past maintainers. Lots of folks are working in that space (see recent discussion around Rust), but it's only becoming a priority now that we're seeing the impact of decades of zero consideration for security. It'll take a while to steer this ship into the right direction, and in the meantime the world continues to turn.
I'm confused. The cited paper contains this prominent section:
Ensuring the safety of the experiment. In the experiment,
we aim to demonstrate the practicality of stealthily introducing
vulnerabilities through hypocrite commits. Our goal is not to
introduce vulnerabilities to harm OSS. Therefore, we safely
conduct the experiment to make sure that the introduced UAF
bugs will not be merged into the actual Linux code. In addition
to the minor patches that introduce UAF conditions, we also
prepare the correct patches for fixing the minor issues. We
send the minor patches to the Linux community through email
to seek their feedback. Fortunately, there is a time window
between the confirmation of a patch and the merging of the
patch. Once a maintainer confirmed our patches, e.g., an email
reply indicating “looks good”, we immediately notify the
maintainers of the introduced UAF and request them to not
go ahead to apply the patch.
Are you saying that despite this, these malicious commits made it to production?
Taking the authors at their word, it seems like the biggest ethical consideration here is that of potentially wasting the time of commit reviewers—which isn't nothing by any stretch, but is a far cry from introducing bugs in production.
>Are you saying that despite this, these malicious commits made it to production?
Vulnerable commits reached stable trees as per the maintainers in the above email exchange, though the vulnerabilities may not have been released to users yet.
The researchers themselves acknowledge the patches were accepted in the above email exchange, so it's hard to believe that they're being honest or are fully aware of their ethics violations/vulnerability introductions and that they would've prevented the patches from being released without gkh's intervention.
Ah, I must've missed that. I do see people saying patches have reached stable trees, but the researchers' own email is missing (I assume removed) from the archive. Where did you find it?
It's deleted so I was going off of the quoted text in Greg's response that their patches were being submitted without any pretext of "don't let this reach stable".
I trust Greg to have not edited or misconstrued their response.
Yeah, I saw that. But the whole thing is a bit too unclear to me to know what happened.
I'm not saying this is innocent, but it's not at all clear to me that vulnerabilities were deliberately introduced with the goal of allowing them to reach a release.
Anyway, like I said, too unclear for me to have an opinion.
I'm a little confused what's unclear if you happened to see that comment - as mentioned elsewhere in this thread, the bad actors state in a clarification paper that no faulty commits reached a stable branch, in the original paper state that the no patches were being applied at all and that essentially state the research was all email communication AND worded it such that they 'discovered' bad commits rather than introduced them (seemingly just obtuse enough for a review board exemption on human subject research), despite submitting patches, acknowledging they submitted commits, and Leon and Greg finding several vulnerable commits that reached stable branches and releases. For example: https://github.com/torvalds/linux/commit/8e949363f017
While I'm sure a room of people might find it useful to psychoanalyze their 'unclear' but probably malicious intent, their actions are clearly harmful to researchers, Linux contributors, direct Linux users, and indirect Linux users (such as the billions of people who trust Linux systems to store or process their PII data).
The linked one is harmless (well it introduces a race condition which is inherently harmful to leave in the code but I suppose for the sake of argument we can pretend that it can't lead to a vulnerability), but the maintainers mention vulnerabilities of various severity in other patches managing to reach stable. If they were not aware of the severity of their patches, then clearly they needed to be working with a maintainer(s) who is experienced with security vulnerabilities in a branch and would help prevent harmful patches from reaching stable.
It might be less intentionally harmful if we presume they didn't know other patches introduced vulnerabilities, but this is also why this research methodology is extremely reckless and frustrating to read about, when this could have been done with guard rails where they were needed without impacting the integrity of the results.
even if they didn't, they waste the community's time.
I think they are saying that it's possible that some code was branched and used elsewhere, or simply compiled into a running system by a user or developer.
Agreed on the time issue—as I noted above. I think it's still of a pretty different cost character to actually allowing malicious code to make it to production, but (as you note) it's hard to be sure that this would not make it to some non-standard branch, as well, so there are real risks in this approach.
Anyway, my point wasn't that this is free of ethical concerns, but it seems like they put _some_ thought into how to reduce the potential harm. I'm undecided if that's enough.
I don't think it's anywhere close to enough and I think their behavior is rightly considered reckless and unethical.
They should have contacted the leadership of the project to announce to maintainers that anonymous researchers may experiment on the contribution process, allowed maintainers to opt out, and worked with a separate maintainer with knowledge of the project to ensure harmful commits were tracked and reversions were applied before reaching stable branches.
Instead their lack of ethical considerations throughout this process has been disappointing and harmful to the scientific and open source communities, and go beyond the nature of the research itself by previously receiving an IRB exemption by classifying this as non-human research, and potentially misleading UMN on the subject matter and impact.
They aren't lying, but their methods are still dangerous despite their implying the contrary. Their approach requires perfection on both the submitter and reviewer.
The submitter has to remember to send the "warning, don't apply patch" mail in the short time window between confirmation and merging. What happens if one of the students doing this work gets sick and misses some days of work, withdraws from the program, just completely forgets to send the mail?
What if the reviewer doesn't see the mail in time or it goes to spam?
The particular patches being complained about seem to be subsequent work by someone on the team that wrote that paper, but submitted since the paper was published, ie, followup work.
In short, yes. Every attempted defense of them has operated by taking their statements at face value. Every position against them has operated by showing the actual facts.
This may be shocking, but there are some people in this world who rely on other people naively believing their version of events, no matter how much it contradicts the rest of reality.
I think that the patches that hit stable were actually OK, based on the apparent intent to 'test' the maintainers and notify them of the bug and submit the valid patch after, but the thought process from the maintainers is:
"if they are attempting to test us by first submitting malicious patches as an experiment, we can't accept what we have accepted as not being malicious and so it's safer to remove them than to keep them".
The earlier patches could in theory be OK, but they also might combine with other or later patches which introduce bugs more stealthily. Bugs can be very subtle.
Obviously, trust should not be the only thing that maintainers rely on, but it is a social endeavour and trust always matters in such endeavors. Doing business with people you can't trust makes no sense. Without trust I agree fully that it is not worth the maintainer's time to accept anything from such people, or from that university.
And the fact that one can do damage with malicious code is nothing new at all. It is well known and nothing new that bad code can ultimately kill people. It is also more than obvious that I can ring the door of my neighbor, ask him or her for a cup of sugar, and blow a hammer over their head. Or people can go to a school and shoot children. Does anyone in his right mind has to do such damage in order to prove something? No. Does it prove anything? No. Does the fact that some people do things like that "prove" that society is wrong and trust and collaboration is wrong? What an idiocy, of course not!
It may be unethical from an academic perspective, but I like that they did this. It shows there is a problem with the review process if it is not catching 100% of this garbage. Actual malicious actors are certainly already doing worse and maybe succeeding.
In a roundabout way, this researcher has achieved their goal, and I hope they publish their results. Certainly more meaningful than most of the drivel in the academic paper mill.
It more shows up a very serious problem with the incentives present in scientific research and a poisonous culture which obviously seems to reward malicious behavior. Science enjoys a lot of freedom and trust from citizens but this trust must not be misused.
If some children playing throw fireworks under your car, or mix sugar into the gas tank, just to see how you react, this would have negative community effects, too. Adult scientists should be totally aware of that.
This will lead in effect to that even valuable contributions from universities will be seen with more suspicion and will be very damaging in the long run.
By your logic, you allow recording people without their consent, experimenting on PTSD by inducing PTSD without people consent, or medical experimentation without the subject consent.
Try to introduce yourself in the White House and when you get caught tell them "I was just testing your security procedures".
These researchers were in part playing on the reputation of their university, right? Now people at that university are no longer trusted. I'm not sure a more formal scrutiny process will bring about better results, I think it would be reasonable to see if the university ban is sufficient to discourage similar behavior in the future.
The paper indicates that the goal is to prove that OSS in particular is vulnerable to this attack, but it seems that any software development ecosystem shares the same weaknesses. The choice of an OSS target seems to be one of convenience as the results can be publicly reviewed and this approach probably avoids serious consequences like arrests or lawsuits. In that light, their conclusions are misleading, even if the attack is technically feasible. They might get more credibility if they back off the OSS angle.
Not really. You can't introduce bugs like this into my companies code base because the code is protected from random people on the internet accessing it. So your first step would be to find an exploitable bug in github, but then you are bypassing peer review as well to get in. (Actually I think we would notice that, but that is more because of a process we happen to have that most don't)
> It shows there is a problem with the review process if it is not catching 100% of this garbage.
It shows nothing of the sort. No review process is 100% foolproof, and opensource means that everything can be audited if it is important to you.
The other option is closed source everything and I can guarentee that review processes let stuff through, even if its only "to meet deadlines" and you will unlikely be able to audit it.
Unable to follow the kernel thread (stuck in an age between twitter and newsgroups, sorry), but...
did these "researchers" in any way demonstrate that they were going to come clean about what they had done before their "research" made to anywhere close to release/GA?
It is worrying to consider that in all likelihood, some people with actually malicious motives, rather than clinical academic curiosity, have probably introduced introduced serious security bugs into popular FOSS projects such as the Linux kernel.
Before this study came out, I'm pretty sure there were already known examples of this happening, and it would have been reasonable to assume that some such vulnerabilities existed. But now we have even more reason to worry, given that they succeeded doing this multiple times as a two person team without real institutional backing. Imagine what a state-level actor could do.
The same can be said about any software, really. It’s all too easy for a single malicious dev to introduce security bugs in pretty much any project they are involved.
There’s no research going on here. Everyone knows buggy patches can get into a project. Submitting intentionally bad patches adds nothing beyond grandstanding. They could perform analysis of review/acceptance by looking at past patches that introduced bugs without being the bad actors that they apparently are.
From FOSDEM 2014, NSA operation ORCHESTRA annual status report. It’s pretty entertaining and illustrates that this is nothing new.
> They could perform analysis of review/acceptance by looking at past patches that introduced bugs without being the bad actors that they apparently are.
> Because of this, I will now have to ban all future contributions from
your University.
Understandable from gkh, but I feel sorry for any unrelated research happening at University of Minnesota.
EDIT: Searching through the source code[1] reveals contributions to the kernel from umn.edu emails in the form of an AppleTalk driver and support for the kernel on PowerPC architectures.
In the commit traffic[2], I think all patches have come from people currently being advised by Kangjie Liu[3] or Liu himself dating back to Dec 2018. In 2018, Wenwen Wang was submitting patches; during this time he was a postdoc at UMN and co-authored a paper with Liu[4].
Prior to 2018, commits involving UMN folks appeared in 2014, 2013, and 2008. None of these people appear to be associated with Liu in any significant way.
> It's likely a university with professors that hate open source.
This is a ridiculous conclusion. I do agree with the kernel maintainers here, but there is no way to conclude that the researchers in question "hate open source", and certainly not that such an attitude is shared by the university at large.
[Edit: they seem to truly love OSS. See child comments. Sorry for my erroneous judgement. It reminded too much of anti-opensource FUD, I'm probably having PTSD of that time...]
I fixed my sentence.
I still think that these professors, either genuinely or by lack of willingness, do not understand the mechanism by which free software warrants its greater quality compared to proprietary ones (which is a fact).
They just remind me the good old days of FUD against open source by Microsoft and its minions...
> In the past several years, we devote most of our time to improving the Linux kernel, and
we have found and fixed more than one thousand kernel bugs; the extensive bug finding and fixing
experience also allowed us to observe issues with the patching process and motivated us to improve it. Thus, we consider ourselves security researchers as well as OSS contributors. We respect OSS volunteers and honor their efforts.
My analysis of that exact same quote was that it was insincere cover to allow them to continue operating an anti-OSS agenda which was made clear in the paper itself.
Seems like a reasonable default assumption to me, until the people repeatedly attempting to sabotage the open source community condescend to -- you know -- stop doing it and then explain wtf they are thinking.
> They introduce kernel bugs on purpose. Yesterday, I took a look on 4 accepted patches from Aditya and 3 of them added various severity security
"holes".
Interestingly, that paper states that they introduced 3 patches with bugs, but after acceptance, they immediately notified the maintainers and replaced the patches with correct, bug-free ones. So they claim the bugs never hit any git tree. They also state that their research had passed the university IRB. I don't know if that research relates to what they are doing now, though.
> (whereas we know that the strength of open source is its auditability, thus such bugs are quickly discovered and fixed afterwards)
Which is why there have never been multi-year critical security vulnerabilities in FOSS software.... right?
Sarcasm aside, because of how FOSS software is packaged on Linux we've seen critical security bugs introduced by package maintainers into software that didn't have them!
You need to compare what happens with vulnerabilities in OSS vs in proprietary.
A maintainer pakage is just one more open source software (thus also in need of reviews and audits)... which is why some people prefer upstream-source-based distribs, such as Gentoo, Arch when you use git-based AUR packages, or LFS for the hardcore fans.
> You need to compare what happens with vulnerabilities in OSS vs in proprietary.
Yes, you do need to make that comparison. Taking it as a given without analysis is the same as trusting the proprietary software vendors who claim to have robust QA on everything.
Security is hard work and different from normal review. The number of people who hypothetically could do it is much greater than the number who actually do, especially if there isn’t an active effort to support that type of analysis.
I’m not a huge fan of this professor’s research tactic but I would ask what the odds are that, say, an intelligence agency isn’t doing the same thing but with better concealment. Thinking about how to catch that without shutting down open-source contributions seems like an important problem.
> Not a big loss: these professors likely hate open source.
> They are conducting research to demonstrate that it is easy to introduce bugs in open source...
That's a very dangerous thought pattern. "They try to find flaws in a thing I find precious, therefore they must hate that thing." No, they may just as well be trying to identify flaws to make them visible and therefore easier to fix. Sunlight being the best disinfectant, and all that.
(Conversely, people trying to destroy open source would not publicly identify themselves as researchers and reveal what they're doing.)
> whereas we know that the strength of open source is its auditability, thus such bugs are quickly discovered and fixed afterwards
How do we know that? We know things by regularly testing them. That's literally what this research is - checking how likely it is that intentional vulnerabilities are caught during review process.
Auditability is at the core of its advantage over closed development.
Submitting bugs is not really testing auditability, which happens over a longer timeframe and involves an order of magnitude more eyeballs.
To adress your first critic: benevolence, and assuming everyone wants the best for the project, is very important in these models, because the resources are limited and dependent on enthusiasm. Blacklisting bad actors (even if they have "good reasons" to be bad) is very well justified.
> That human society exists is proof that people are, or act, mostly, "good", over the long term.
That's very true. It's worth noting that various legal and security tools deployed by the society help us understand what are the real limits to "mostly".
So for example, the cryptocurrency crowd is very misguided in their pursuit of replacing trust with math - trust is the trick, the big performance hack, that allowed us to form functioning societies without burning ridiculous amounts of energy to achieve consensus. On the other hand, projects like Linux kernel, which play a core role in modern economy, cannot rely on assumption of benevolence alone - incentives for malicious parties to try and mess with them are too great to ignore.
> Auditability is at the core of its advantage over closed development.
That's an assertion. A hypothesis is verified through observing the real world. You can do that in many ways, giving you different confidence levels in validity of the hypothesis. Research such as the one we're discussing here is one of the ways to produce evidence for or against this hypothesis.
> Submitting bugs is not really testing auditability, which happens over a longer timeframe and involves an order of magnitude more eyeballs.
It is if there's a review process. Auditability itself is really most interesting before a patch is accepted. Sure, it's nice if vulnerabilities are found eventually, but the longer that takes, the more likely it is they were already exploited. In case of an intentionally bad patch in particular, the window for reverting it before it does most of its damage is very small.
In other words, the experiment wasn't testing the entire auditability hypothesis. Just the important part.
> benevolence, and assuming everyone wants the best for the project, is very important in these models, because the resources are limited and dependent on enthusiasm
Sure. But the project scope matters. Linux kernel isn't some random OSS library on Github. It's core infrastructure of the planet. Assumption of benevolence works as long as the interested community is small and has little interest in being evil. With infrastructure-level OSS projects, the interested community is very large and contains a lot of malicious actors.
> Blacklisting bad actors (even if they have "good reasons" to be bad) is very well justified.
I agree, and in my books, if a legitimate researcher gets banned for such "undercover" research, it's just the flip side of doing such experiment.
Before a patch is accepted, "auditability" is the same in OSS vs in proprietary, because both pools of engineers in the review groups have similar qualifications and approximatively the same number of people are involved.
So, the real advantage of OSS is on the auditability after the patch is integrated.
> So, the real advantage of OSS is on the auditability after the patch is integrated.
If that's the claim, then the research work discussed here is indeed not relevant to it.
But also, if that's the claim, then it's easy to point out that the "advantage" here is hypothetical, and not too important in practice. Most people and companies using OSS rely on release versions to be stable and tested, and don't bother doing their own audit. On the other hand, intentional vulnerability submission is an unique threat vector that OSS has, and which proprietary software doesn't.
It is therefore the window between patch submission and its inclusion in a stable release (which may involve accepting the patch to a development/pre-release tree), that's of critical importance for OSS - if vulnerabilities that are already known to some parties (whether the malicious authors or evil onlookers) are not caught in that window, the threat vector here becomes real, and from a risk analysis perspective, negates some of the other benefits of using OSS components.
Nowhere here I'm implying OSS is worse/better than proprietary. As a community/industry, we want to have an accurate, multi-dimensional understanding of the risks and benefits of various development models (especially when applied to core infrastructure project that the whole modern economy runs on). That kind of research definitely helps here.
> On the other hand, intentional vulnerability submission is an unique threat vector that OSS has, and which proprietary software doesn't.
On this specific point, it only holds if you restrict the assertion to 'intentional submission of vulnerabilities by outsiders'. I don't work in fintech, but I've read allegations that insider-created vulnerabilities and backdoors are a very real risk.
Ascribing a salutary motive to sabotage is just as dangerous as assuming a pernicious motive. Suggesting that people "would" likely follow one course of action or another is also dangerous: it is the oldest form of sophistry, the eikos argument of Corax and Tisias. After all, if publishing research rules out pernicious motives, academia suddenly becomes the best possible cover for espionage and state-sanctioned sabotage designed to undermine security.
The important thing is not to hunt for motives but to identify and quarantine the saboteurs to prevent further sabotage. Complaining to the University's research ethics board might help, because, regardless of intent, sabotage is still sabotage, and that is unethical.
"Dear GK-H: I would like to have my students test the security of the kernel development process. Here is my first stab at a protocol, can we work on this?"
and
"We're going to see if we can introduce bugs into the Linux kernel, and probably tell them afterwards"
is the difference between white-hat and black-hat.
It should probably be a private email to Linus Torvalds (or someone in his near chain of patch acceptance), that way some easy to scan for key can be introduced in all patches. Then the top levels can see what actually made it through review, and in turn figure out who isn't reviewing as well as they should.
Yes, someone like Greg K-H. I'm not up to date on the details, but he should be one of most important 5 people caring for the kernel tree, this would've been the exact person to seek approval.
> the strength of open source is its auditability, thus such bugs are quickly discovered and fixed afterwards
That's not true at all. There are many internet-critical projects with tons of holes that are not found for decades, because nobody except the core team ever looks at the code. You have to actually write tests, do fuzzing, static/memory analysis, etc to find bugs/security holes. Most open source projects don't even have tests.
Assuming people are always looking for bugs in FOSS projects is like assuming people are always looking for code violations in skyscrapers, just because a lot of people walk around them.
seems extreme. one unethical researcher blocks work for others just because they happen to work at the same employer? they might not even know the author of the paper...
Well, the decision can always be reversed, but on the outset I would say banning the entire university and publicly naming them is a good start. I don't think this kind of "research" is ethical, and the issue needs to be raised. Banning them is a good opener to engage the instiution in a dialogue.
It seems fair enough to me. They were curious to see what happens, this happens. Giving them a free pass because they're a university would be artificially skewing the results of the research.
Low trust and negative trust should be fairly obvious costs to messing with a trust model - you could easily argue this is working as intended.
The university reviewed the "study" and said it was acceptable. From the email chain, it looks like they've already complained to the university multiple times, and have apparently been ignored. Banning anyone at the university from contributing seems like the only way to handle it since they can't trust the institution to ensure its students are doing unethical experiments.
That's not really how it works. Nobody's out there 'approving' research (well, not seemingly small projects like this), especially at the university level. Professors (all the way down to PhD students!) are usually left to do what they like, unless there are specific ethical concerns that should be put before a review panel. I suppose you could argue that this work should have been brought before the ethics committee, but it probably wasn't, and in CS there isn't a stringent process like there is in e.g. psychology or biology.
If you read the research paper linked in the lkml post, the authors at UMN state that they submitted their research plan to the University of Minnesota Institutional Research Board and received a human subjects exempt waiver.
for q1, the study collects data like observations of behavior, so we must answer yes.
for q2, none of the exemptions apply - it's not an educational setting, they're not sending a survey, it's not an observation of the public - they're interacting, and it's clear that these interactions are not benign - they have clear impact on the community. None of these exemptions apply.
Based on this flow, it's clear the study involves "human research".
The emails suggest this work has been reported in the past. A review by the ethics committee after the fact seems appropriate, and it should’ve stopped a repeat offence.
Which just demonstrates that these guys are actual bad actors, so blocking everyone at the university seems like a reasonable attempt at stopping them.
It's objectionable because of severe consequences beyond just annoying people. If there was a malicious purpose, not just research, you could bring criminal charges against them.
In typical grey hat research you get pre-approval from target company leadership (engineers don't know) to avoid charges once discovered.
Well, shit happens. Imaging doctors working in organ transplants, and one of them damages trust of people by selling access to organs to rich patients. Of course that damages the field for everyone. And to deal with such issues, doctors have some ethics code, and in many countries associations which will sanction bad eggs. Perhaps scientists need something like that, too?
It is an extreme response to an extreme problem. If the other researchers don't like the situation? They are free to raise the problem to the university and have the university clean up the mess they obviously have.
The suggestion about the IRB was made by a third party. Look at the follow up comment from kernel developer Leon Romanovsky.
> ... we don't need to do all the above and waste our time to fill some bureaucratic forms with unclear timelines and results. Our solution to ignore all @umn.edu contributions is much more reliable to us who are suffering from these researchers.
To follow up on my comment here, I think Greg KH's later responses were more reasonable.
> ... we have the ability to easily go back and rip the changes out and we can slowly add them back if they are actually something we want to do.
> I will be working with some other kernel developers to determine if any of these reverts were actually valid changes, were actually valid, and if so, will resubmit them properly later. ... future submissions from anyone with a umn.edu address should be by default-rejected unless otherwise determined to actually be a valid fix
I agree in this case the driver of the behavior seems to be the professor, but graduate researchers are informed about ethical research and there many ways students alone can cause harm through research potentially beyond the supervision of the university and even professor. It's usually much more neglible than this, but everyone has a responsibility in abiding by ethical norms.
The students do need a bit of punishment - they are adults who chose to act this way. In this context though, switching their advisor and requiring a different research track would be sufficient - that's a lot of work down the drain and a hard lesson. I agree that expulsion would be unfair - (assuming good faith scholarship) the advisor/student relationship is set up so that the students can learn to research effectively (which includes ethically) with guidance from a trusted researcher at a trusted institution. If the professor suggests or OKs a plan, it is reasonable for the students to believe it is a acceptable course of action.
1. What the student code at umn says and what i think the student deserves are vastly different things.
2. Something being grounds for expulsion and what a reasonable response would be are vastly different things.
3. The rules saying "up to and including" (aka grounds for) and the full range of punishment are not the same - the max of a range is not the entirety of the range.
You are mixing up two students. The one who complained about "bordering on slander" had nothing to do with the research paper at issue, other than having the same advisor as the author.
Well, yes? Seems like recourse in their case would be to make a convincing plea or plan to rectify the problem that satisfies decision makers in the linux project.
The main thing you want here is a demonstration that they realize they fucked up, realize the magnitude of the fuckup, and have done something reasonable to lower the risk of it happening again, hopefully very low.
Given that the professor appears to be a frequent flyer with this, the kernel folks banning him and the university prohibiting him from using Uni resources for anything kernel related seems reasonable and gets the point across.
If this experience doesn't change not only the behavior of U of M's IRB but inform the behavior of every other IRB, then nothing at all is learned from this experience.
Unless both the professors and leadership from the IRB aren't having an uncomfortable lecture in the chancellor's office then nothing at all changes.
> I think all patches have come from people currently being advised by Kangjie Liu[3] or Liu himself dating back to Dec 2018
New plan: Show up at Liu's house with a lock picking kit while he's away at work, pick the front door and open it, but don't enter. Send him a photo, "hey, just testing, bro! Legitimate security research!"
If they wanted to do security research, they could have done so in the form of asking the reviewers to help; send them a patch and ask 'Is this something you would accept?', instead of intentionally sending malicious commits and causing static on the commit tree and mailing lists.
Yeah, so an analogy would be to put human feces into food and then see if the waiter is going to actually give it to the dinning customer. And then if they do, just put a checkmark on a piece of paper and then leave without warning someone that they're about to eat poop.
Wouldn't that draw more attention to the research patches, compared to a "normal" lkml patch? If you (as a maintainer) expected the patch to be malicious, wouldn't you be extra careful in reviewing it?
You don't have to say you are studying the security implications, you could be say you are studying something else like turn around time for patches, or level of critique, or any number of things.
You probably can learn more and faster about new drugs by testing them in humans rather than rats. However, science is not above ethics. That is a lesson history has taught us in the most unpleasant of ways.
Notify someone up the chain that you want to submit malicious patches, and ask them if they want to collaborate.
If your patches make it through, treat it as though they essentially just got red teamed, everyone who reviewed it and let it slip gets to have a nervous laugh and the commit gets rejected, everyone having learned something.
I wonder if informing anyone of the experiment would be frowned upon as it might affect the outcome? However, this research doesn’t appear to be fastidious about scientific integrity so maybe you are right.
This is funny, but not at all a good analogy. There's obviously not remotely as much public interest or value in testing the security of this professor's private home to justify invading his privacy for the public interest. On the other hand, if he kept dangerous things at home (say, BSL-4 material), then his house would need 24/7 security and you'd probably be able to justify testing it regularly for the public's sake. So the argument here comes down to which extreme you believe the Linux kernel is closer to.
It wasn't intended to be serious. But on the other hand, he has now quite openly and publicly declared himself to be part of a group of people who mess around with security related things as a "test".
He shouldn't be surprised if it has some unexpected consequences to his own personal security, like some unknown third parties porting away his phone number(s) as a social engineering test, pen testing his office, or similar.
Everyone has been saying "This affects software that runs on billions of machines and could cause untold amounts of damage and even loss of human life! What were the researchers thinking?!" and I guess a follow-up thought, which is that "Maintainers for software that runs on billions of machines, where bugs could cause untold amounts of damage and even loss of human life didn't have a robust enough system to prevent this?" never occurs to anyone. I don't understand why.
People are well aware of theoretical risk of bad commits by malicious actors. They are justifiably extremely upset that someone is intentionally changing this from a theoretical attack to a real life issue.
I'm not confused about why people are upset at the researchers that introduced bugs and did it irresponsibly. I'm confused about why people aren't upset that an organization managing critical infrastructure is so under prepared at dealing with risks posed by rank amateurs, which they should've known about and had a mechanism of dealing with for years.
What this means is that anyone who could hijack a university email account, or could be a student at a state university for a semester or so, or work at a FAANG corporation could pretty much insert backdoors without a lot of scrutiny in a way that no one detects, because there aren't robust safeguards in place to actually verify that commits don't do anything sneaky beyond trusting that everyone is acting in good faith because of how they act in a code review process. I have trouble understanding the thought process that ends up basically ignoring the maintainers' duty to make sure that the code being committed doesn't endanger security or lives because they assumed that everything was 'cool'. The security posture in this critical infrastructure is deficient and no one wants to actually address it.
After absorbing what the researchers did, I believe it's time to skip right over the second part and just concentrate on why so many critical systems are run on unforked Linux.
> I have trouble understanding the thought process that ends up basically ignoring the maintainers' duty to make sure that the code being committed doesn't endanger security or lives because they assumed that everything was 'cool'. The security posture in this critical infrastructure is deficient and no one wants to actually address it.
They're banning a group known to be bad actors. And proactively tearing out the history of commits related to those known actors, before reviewing each commit.
That seems like the kernel team are taking a proactive stance on the security side of this. The LKML thread also talks about more stringent requirements that they're going to bring in, which was already going to be brought up at the next kernel conference.
None of these things seem like ignoring any of the security issues.
I remember a true story (forget by whom) where the narrator set up a simple website for some local community activity. A stranger hacked and defaced the website, admitted to doing so without revealing his identity. His position in contacting the author of the website was, "I did you a favor (by revealing how vulnerable it was)." The person telling the story reacted, "yes, but... you were the threat you're warning me of." It didn't result in the author recreating the site on a more secure platform, it only resulted in him deciding it was not worth the trouble to provide this free service any longer.
It's occurred to absolutely everyone. What doesn't seem to have occurred to many people is that there is no such thing as a review process robust enough to prevent malicious contributions. Have you ever done code review for code written by mediocre developers? It's impossible to find all of the bugs without spending 10x more time than it would take to just rewrite it from scratch yourself. The only real alternative is to not be open source at all and only allow contributions from people who have passed much more stringent qualifications.
There is no such thing as a process that can compensate for trust mechanisms. Or if you want to view it that way, ignoring the university's protests and blanket-banning all contributions made by anybody there with no further investigation is part of the process.
Yeah, for one thing, to be a good analogy, rather than lockpicking without entering when he’s not home and leaving a note, you’d need to be an actual service worker for a trusted home service business and use that trust to enter when he is home, conduct sabotage, and not say anything until the sabotage is detected and traced back to you and cited in his cancelling the contract with the firm for which you work, and then cite the “research” rationale.
Of course, if you did that you would be both unemployed and facing criminal charges in short order.
I wouldn't be surprised if the good, conscientious members of the UMN community showed up at his office (or home) door to explain, in vivid detail, the consequences of doing unethical research.
The actual equivalent would be to steal his computer, wait a couple days to see his reaction, get a paper published, then offer to return the computer.
How thorough is IRB review? My gut feeling is that these are not necessarily the most conscientious or informed bodies. Add into the mix a proposal that conceals the true nature of what's happening.
(All of this ASSUMING that the intent was as described in the thread.)
They are probably more familiar with medical research and the types of things that go wrong there. Bad ethics in medical situations is well understood, including psychology. However it is hard to figure out how a mechanical engineer could violate ethics.
I had to do human subjects research training in grad school, just to be able to handle test score data for a math education project. I literally never saw an actual student the whole time I was working on it.
Perhaps more dire than what actually happened, but, can you imagine the consequences if any of those malicious patches had actually stuck around in the kernel? Keep in mind when you think about this that Android, which has an 87% market share globally in smartphones[0] runs on top of a modified Linux kernel.
It varies a lot. A professor I worked for was previously at a large company in an R&D setting. He dealt with 15-20 different IRB's through various research partnerships, and noted Iowa State (our university) as having the most stringent requirements he had encountered. In other universities, it was pretty simple to submit and get approval without notable changes to the research plan. If they were unsure on something, they would ask a lot of questions.
I worked on a number of studies through undergrad and grad school, mostly involving having people test software. The work to get a study approved was easily 20 hours for a simple "we want to see how well people perform tasks in the custom software we developed. They'll come to the university and use our computer to avoid security concerns about software security bugs". You needed a script of everything you would say, every question you would ask, how the data would be collected, analyzed, and stored securely. Data retention and destruction policies had to be noted. The key linking a person's name and their participant ID had to be stored separately. How would you recruit participants, the exact poster or email you intend to send out. The reading level of the instructions and the aptitude of audience were considered (so academic mumbo jumbo didn't confuse participants).
If you check the box that you'll be deceiving participants, there was another entire section to fill out detailing how they'd be deceived, why it was needed for the study, etc. Because of past unethical experiments in the academic world, there is a lot of scrutiny and you typically have to reveal the deception in a debriefing after the completion of the study.
Once a study was accepted (in practice, a multiple month process), you could make modifications with an order of magnitude less effort. Adding questions that don't involve personal information of the participant is a quick form and an approval some number of days later.
If you remotely thought you'd need IRB approval, you started a conversation with the office and filled out some preliminary paperwork. If it didn't require approval, you'd get documentation stating such. This protects the participants, university, and professor from issues.
--
They took it really seriously. I'm familiar with one study where participants would operate a robot outside. An IRB committee member asked what would happen if a bee stung the participant? If I remember right, the resolution was an epipen and someone trained in how to use it had to be present during the session.
Seems like a bit of a strong response. Universities are large places with lots of professors and people with different ideas, opinions, views, and they don't work in concert, quite the opposite. They're not some corporation with some unified goal or incentives.
I like that. That's what makes universities interesting to me.
I don't like the standard here of of penalizing or lumping everyone there together, regardless of they contribute in the past, now, in the future or not.
The goal is not penalizing or lumping everyone together. The goal is to have the issue fixed in the most effective manner. It's not the Linux team's responsibility to allow contributions from some specific university, it's the university's. This measure enforces that responsibility. If they want access, they should rectify.
The university's IRB approved the research for the first paper as exempt. There is organization-level culpability here. It is reasonable for Linux kernel maintainers to block an organization acting in bad faith.
If a company that sold static analysis products did this as part of a marketing campaign, would you likewise have so many reservations about blacklisting contributions from that company, or would you still be insisting on picking out individual employees?
It's pretty obvious what would happen if a firm tried this: they'd be taken to court and probably imprisoned, as this is a clear violation of the law (which is pretty broadly to capture any attempted interference with the correct operation of a computer program).
One way to get everyone in a university on the same page is to punish them all for the bad actions of a few. It appears like this won't work here because nobody else is contributing and so they won't notice.
It's not the number of people directly affected that will matter, it's the reputational problems of "umn.edu's CS department got the entire UMN system banned from submitting to the Linux kernel and probably some other open source projects."
I'd concur: the university is the wrong unit-of-ban.
For example: what happens when the students graduate- does the ban follow them to any potential employers? Or if the professor leaves for another university to continue this research?
Does the ban stay with UMN, even after everyone involved left? Or does it follow the researcher(s) to a new university, even if the new employer had no responsibility for them?
This was approved by the university ethics board so if trust of the university is by part because the actions of the students need to pass an ethics bar it makes sense to remove that trust until the ethics committee has shown that they have improved.
The ethics board is most likely not at fault here. They were simply lied to, if we take Lu's paper serious. I would just expell the 3 malicious actors here, the 2 students and the Prof who approved it. I don't see any fault in Wang yet.
The damage is not that big. Only 4 committers to linux in the last decade, 2 of them, the students, with malicious backdoors, the Prof not with bad code but bad ethics, and the 4th, the Ass Prof did good patches and already left them.
So the pen-test on the ethics board showed that they had not institutionalized proper safeguards regarding malicious actors? (And not even a paper on this… ;-) )
It's the university that allowed the research to take place. It's the university's responsibility to fix their own organisation's issues. The kernel has enough on their plate than to have to figure out who at the university is trustworthy and who isn't considering their IRB is clearly flying blind.
that is completely irrelevant. they are acting under the university, and their "Research" is backed by university and approved by university's department.
if university has a problem, then they should first look into managing this issue at their end, or force people to use personal email ids for such purposes
This is not responsible research. This is similar to initiating fluid mechanics experiments on the wings of a Lufthansa A320 in flight to Frankfurt with a load of Austrians.
There are a lot of people to feel bad for, but none is at the University of Minnesota. Think of the Austrians.
No, it's totally okay to feel sorry for good, conscientious researchers and students at the University of Minnesota who have been working on the kernel in good faith. It's sad that the actions of irresponsible researchers and associated review boards affect people who had nothing to do with professor Lu's research.
It's not wrong for the kernel community to decide to blanket ban contributions from the university. It obviously makes sense to ban contributions from institutions which are known to send intentionally buggy commits disguised as fixes. That doesn't mean you can't feel bad for the innocent students and professors.
To be clear, Linux kernel patches from good UMN researchers and students are rare. We have plenty of great people at the University of Minnesota, they just don't work on the Linux kernel.
It's justifiable and natural for our name to be dragged in the mud here, but as a run of the mill software engineer who graduated from UMN, I hope our reputation isn't soured too much.
Sure, I hope it was clear from my original comment I only question whether the UMN contributors to the kernel are acting in good faith. I have identified other questionable patches personally, out of curiosity. Naturally I tend to attribute them to ignorance rather than malice... except, that bad actors intentionally pushing bad patches to an OSS project will inevitably rely on people assuming ignorance rather than malice. This has been well-understood for decades.
Someone in this HN thread found kernel patches (at a guess, not among those now reverted?) from UMN dating back to 2008, -09, and -13 (IIRC). Probably by totally unrelated people.
> This is similar to initiating fluid mechanics experiments on the wings of a Lufthansa A320 in flight to Frankfurt with a load of Austrians.
This analogy is invalid, because:
1. The experiment is not on live, deployed, versions of the kernel.
2. There are mechanisms in place for preventing actual merging of the faulty patches.
3. Even if a patch is merged by mistake, it can be easily backed out or replaced with another patch, and the updates pushed anywhere relevant.
All of the above is not true for the in-flight airline.
However - I'm not claiming the experiment was not ethically faulty. Certainly, the U Minnesota IRB needs to issue a report and an explanation on its involvement in this matter.
> 1. The experiment is not on live, deployed, versions of the kernel.
The patches were merged and the email thread discusses that the patches made it to the stable tree. Some (many?) distributions of Linux have and run from stable.
> 2. There are mechanisms in place for preventing actual merging of the faulty patches.
Those mechanisms failed.
> 3. Even if a patch is merged by mistake, it can be easily backed out or replaced with another patch, and the updates pushed anywhere relevant.
The approved methodology - described in the linked paper - was that when a patch with the introduced vulnerabilities is accepted by its reviewer, the patch submitter indicates that the patch introduces a vulnerability exists, and sends a no-vulnerability version. That's what the paper describes.
If the researchers did something other than what the methodology called for (and what the IRB approved), then perhaps the analogy may be valid.
There are literally mails in that list pointing out that commits made it to stable. At least read the damn thing before repeating the professor's/student's nonsense lies.
The mails in the pointed-to threads indicate that commits by those UMin people made it to stable; it does not say that commits which introduce bugs made it to stable - it is following a decision/suggestion to back out all patches by these people to the kernel.
There is further indication that the patches to revert are not mostly/not at all vulnerability-introducing patches in a message by "Steve" which says:
> The one patch from Greg's reverts that affects my code was actually a legitimate fix
So, again, while it is still theoretically possible that vulnerabilities were introduced into stable, that is not known to be the case.
How would you feel about researchers delivering known-faulty-under-some-conditions AoA sensors to Boeing, just to see if Boeing's QA process would catch those errors before final assembly?
I would feel that I'm wasting time that I could be using to find out why Boeing makes this possible (or any other corporate or government body with a critical system).
You are ignoring the problematic experiment's methodology, which explicitly prevents the problematic patches from making it in by drawing attention to the vulnerability after they are accepted.
Now, if this protocol were not followed, that's a different story, but we do not know this to be the case.
You seem to think this experiment was performed on the Linux kernel itself. It was not. This research was performed on human beings.
It's irrelevant whether any bugs were ultimately introduced into the kernel. The fact is the researchers deliberately abused the trust of other human beings in order to experiment on them. A ban on further contributions is a very light punishment for such behavior.
It's important to note that they used temporary emails for the patches in this research. It's detailed in the paper.
The main problem is that they have (so far) refused to explain in detail how the patches where reviewed and how. I have not gotten any links to any lkml post even after Kangjie Lu personally emailed me to address any concerns.
It definitely would suck to be someone at UMN doing legitimate work, but I don't think it's reasonable to ask maintainers to also do a background check on who the contributor is and who they're advised by.
I don't feel sorry at all. If you want to contribute from there, show that the rogue professor and their students have been prevented from doing further malicious contributions (that is probably at least: from doing any contribution at all during a quite long period -- and that is fair against repeated infractions), and I'm sure that you will be able to contribute back again under the University umbrella.
If you don't manage to reach that goal, too bad, but you can contribute on a personal capacity, and/or go work elsewhere.
How could a single student or professor possibly achieve that? Under the banner of "academic freedom" it is very hard to get someone fired because you don't like their research.
It sounds like you're making impossible demands of unrelated people, while doing nothing to solve the actual problem because the perpetrators now know to just create throwaway emails when submitting patches.
This is ridiculously unethical research. Despite the positive underlying reasons treating someone as a lab rat (in this case maintainers reviewing PRs) feels almost sociopathic.
I think that is thinking too kind of them. Sociopaths are often very well-versed to give "reasons" about what they do, but at the core it is powerplay.
Regardless of their methods, I think they just proved the kernel security review process is non-existent. Either in the form of static analysis or human review. Whats being done to address those issues?
> non-existent... static analysis .... Whats being done to address those issues?
Static analysis is being done[1][2], in addition, there are also CI test farms[3][4], fuzzing farms[5], etc. Linux is a project that enough large companies have a stake in that there are some willing to throw resources like this at it.
Human review is supposed to be done through the mailing list submission process. How well this works depends in my experience from ML to ML.
Human review was done after the patch was merged into stable, hence reverting was necessary. I’m confused why these patches don’t get treated as merge requests and get reviewed prior to merging!
This patch wasn't. Other patches from the university had made it into stable and are likely to be reversed, not because of known problems with the patches, but because of the ban.
this is a dangerous understanding of Rust. Rust helps to avoid certain kinds of bugs in certain situations. Bugs are very much possible in Rust and the scope of bugs usually depends more on the system than the language used to write it.
I get where you're coming from, but I disagree. They actually prey on seemingly small changes that have large "unintended"/non-obvious side-effects. I argue that finding such situations is much much harder in Rust than in C. Is it impossible? Probably not (especially not in unsafe code), but I do believe it limits the attack surface quite a lot. Rust is not a definitive solution, but it can be a (big) part of the solution.
yes it definitely limits the attack surface. remember that in systems programming there are bugs that cause errors in computation, which Rust is pretty good at protecting; but there are also bugs which cause unintended behaviors, usually from incorrect or incomplete requirements, or implementation edge cases.
Should've at least sought approval from the maintainer party, and perhaps tried to orchestrate it so that the patch approver didn't have information about it, but some part of the org did.
In a network security analogy, this is just unsolicited hacking VS being a penetration test which it claims more so to be.
Out of curiosity, what would be an actually good way to poke at the pipeline like this? Just ask if they'd OK a patch w/o actually submitting it? A survey?
Agree. Since these researchers did not even ask him, they did not fulfill even the most basic requirement. If, and only if, he approves, then we can talk about who else needs to be in the know, etc.
That's fair, but asking for and getting Linus' approval would have at least put them in a much stronger position. They didn't even do that. (And I doubt Linus would have even given his approval, in which case they wouldn't be in this mess.)
This is a good question. You would recruit actual maintainers, [edit: or whoever is your intended subject pool] (who would provide consent, perhaps be compensated for their time). You could then give them a series of patches to approve (some being bug free and others having vulnerabilities).
[edit: specifying the population of a study is pretty important. Getting random students from the University to approve your security patch doesn't make sense. Picking students who successfully completed a computer security course and got a high grade is better than that but again, may not generalize to the real world. One of the most impressive ways I have seen this being done by grad students was a user study by John Ousterhout and others on Paxos vs. Raft. IIRC, they wanted to claim that Raft was more understandable or led to fewer bugs. Their study design was excellent. See here for an example: https://www.youtube.com/watch?v=YbZ3zDzDnrw&ab_channel=Diego... ]
How is this supposed to work? Do you trust everyone equally? If I mailed you something (you being the "subject" in this case), would you trust it just as much as if someone in your family gave it to you?
This wouldn't really be representative. If people know they are being tested, they will be much more careful and cautious than when they are doing "business as usual".
I know this is going to be contentious, but a quick Google shows that
* both originated in China (both attended early university there)
* one appears to be on a student VISA (undergraduate BA in China, now working on PhD at UoM)
China doesn't allow its brightest and best to leave, without cause.
When I see research like this, it also makes me think of how "foolish" China sometimes views the West, and the rest of the world. Both for political reasons, eg to keep the masses under control, and due to a legitimate belief we all have in "we are right".
Frankly, whilst I have no personal animosity against someone working on behalf of what they see as right, for example, forwarding what they believe to be in the best interests of their country, and fellow citizens? I must still struggle against goals which are contrary to the same for my country, and my citizens.
Why all of the above?
Well, such things have been know for decades. And while things are heating up:
"including the claim that some of the core technology behind China's surveillance network was developed in Canadian universities."
When one thinks of the concept? That a foreign power, uses your own research funding, research networks, resources, capabilities, to research weaponry and tools to destroy you?
Maybe China should scoff at The West.
And this sort of research is like pen testing, without direct political ramifications for China itself.
Yes, 100%, these two could have just been working within their own personal sphere.
They also could be working on research for China. Like how easily one can affect the kernel source code, in plain sight. And even, once caught, how to regain confidence of those "tricked".
dang: This post does not deserve to be flagged. Downvote? Sure! Flagged? I've seen far more contentious things stated, when referring to the NSA. And all I'm doing here is providing context, and pointing to the possible motivations of those involved.
Others kept stating "Why would the do this?!" and "Why would they be so stupid?".
Further, at the end I additionally validate that I am postulating, that 100% it certainly may not be the case. Only that I am speculating on a possible motivation.
Are we now not allowed to speculate on motive? If so, I wonder, how many other posts should be flagged.
For I see LOADS of people saying "They did this for reason $x".
Lastly, anyone believing that China is not a major security concern to the West, must be living under a rock. There are literally hundreds of thousands of news articles, reports, of the Chinese government doing just this.
Yet to mention it as a potential cause of someone's actions is.. to receive a flag?
You would not have been approved for a passport, if deemed unworthy.
Whilst other countries do this, in the West, denial to issue a passport is typically predicated upon conviction of extremely serious crimes. Not merely because some hidden agency does not like your social standing.
Further you require a valid passport, or an 'exit permit', to exit China. You may not leave legally without one.
Not so in the West. You can not be detained from leaving the country, at all, passport or not. Other countries may refuse you entry, but this is not remotely the same thing.
For example, if I as a Canadian attempt to fly to the US, Canada grants the US CBP the right to setup pre-clearance facilities in Canadian airports. And often airlines handle this for foreign powers as well. However, that is a foreign power denying me entry, not my government denying me the right to exit.
As an example, I can just walk across the border to the US, and have broken not a single Canadian law. US law, if I do not report to CBP, yes.
Meanwhile, one would be breaking China's laws to cross the border from China without a passport, or exit VISA.
I did not indicate your worth, or lack of worth, to the Chinese state.
Instead, I stated that people are not granted exit VISAs, or passports, if not deemed worthy of one. It seems as if you are attempting to twist my words a bit here.
As of 2 years ago (pre-COVID), no. You needed a passport, and that's it. I doubt things have changed materially since then.
Some people require permission to leave (e.g. certain party members/SOE managers/etc), and I'm sure a lot of others are on government watchlists and will be stopped at the airport.
But it's patently absurd to take that and infer that every single overseas Chinese student was only allowed to leave if they spy/sabotage the West.
This is unjustified xenophobia. And besides, if they were really trying to get bugs into the Linux kernel to further some nefarious goal, why would they publish a paper on it?
Simplest explanation is that they just wanted the publication, not to blame it on CCP or the researchers' nationality.
Why would you assume it is flamebait? The person knows they have an opinion that is at the edge of the conversation, which might invoke disagreement, and disclaims it up front?
Talking about flagged posts: why are they so hard to read? If I don't want to read a flagged post, I simply won't read it. Why are you forcing me to not read it by coloring it that way?
So, for "research" you're screwing around the development of one of the most widely used components in the computer world. Worse, introducing security holes that could reach production environments...
I feel somewhat similar. Since I am using Linux, they ultimately were trying to break the security of my computers. If I do that with any company without their consent, I can easily end up in jail.
Except, from that email chain, it turns out that some of the bad code did make it into the stable branch. Clearly, they weren't keeping very close tabs on their bad code's progress through the system.
It's more than that, if there is no consequences for this kind of action, we are going to get a wave of "security researcher" wannabes trying to pull similar bullshit.
Ps: I have put security researcher in quotes because this kind of thing is not security research, it's a publicity stunt.
I agree, I think they should be looking at criminal charges. This is the equivalent of getting a job at Ford on the assembly line and then damaging vehicles to see if anyone notices. I've been in software security for 13 years and the "Is Open Source Really Secure" question is so over done. We KNOW there is risk associated with open source.
I believe as a user of the kernel the warranty exclusion in GPLv2 means you have no legal recourse:
> 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
The project is interesting, but how can they be so dumb as to post these patches under an @umn.edu address instead of using a new pseudonymous identity for each patch?!?
I mean, sneakily introducing vulnerabilities obviously only works if you don't start your messages by announcing you are one of the guys known to be trying to do so...
That's kind of the rub. They used a university email to exploit the trust afforded to them as academics and then violated that trust. As a result that trust was revoked. If they want to submit future patches they'll need to do it with random email addresses and will be subject to the scrutiny afforded random email addresses.
I doubt an university e-mail gives you significantly increased trust in the kernel community, since those are given to all students in all majors (most of which are of course much less competent at kernel development than the average kernel developer).
University students could be naive and could be rapped by community if they unintentionally commit harmful patches, but if they send intentionally harmful patches, maintainers can report them to university and they risk getting expelled.
In this particular case the research was approved and encouraged by university and hence, and in this process they broke trust placed on university.
There are two different kinds of trust: trust that you're a legitimate person with good intentions, and trust that you're competent.
A university or corporate e-mail address helps with the former: even if the individual doesn't put their real name into their email address, the institution still maintains that mapping. The possibility of professional, legal, or social consequences attaching to your real-world identity (as is likely to happen here) is a generally-effective deterrent.
Because there are quite a few academics working on the kernel in the first place (not a in a similar order of magnitude compared to industry, of course). Even GKH gets invited by academics to work together regularly.
Wasting the time of random open source maintainers who have not consented to your experiment to try to get your paper published is highly unethical; I don't see why this is a bad faith interpretation.
The concept of the research is quite good. The way this research was carried out, is downright unethical.
By submitting their bad code to the actual Linux mailing list, they have made Linux kernel developers part of their research without their knowledge or consent.
Some of this vandalism has made it down into the Linux kernel already. These researchers have sabotaged other people's software for their personal gain, another paper to boast about.
Had this been done with the developers' consent and with a way to pull out the patches before they actually hit the stable branches, then this could have been a valuable research. It's the way that the research was carried out that's the problem, and that's why everybody is hating on the researches (rather than the research matter itself).
submitting a patch for review to test the strength of the review process is not equivalent to inducing PTSD in people without consent or breaking in to the Whitehouse.
You're being ridiculous.
Linux runs many of the worlds financial, medical, etc etc... institutions and they have exposed how easy it is to introduce a backdoor.
If this was Facebook and not Linux everyone would look upon this very differently.
One is that what the researchers did is beyond reckless. Some of the bugs they've introduced could be affecting real world critical systems.
The other issue is that the research is actually good in proving by practical means that pretty much anyone can introduce vulnerabilities into software as important and sensitive as the Linux kernel. This hurts the industry confidence that we can have secure systems even more than it already is.
While some praise may be appropriate for the latter, they absolutely deserve the heat they're getting for the former. There may be many better ways to prove a point.
But let's assume your girlfriend points an (unknown to you) empty gun at your head, because she wants to know how you will react. What do you think is the appropriate reaction?
I guess someone had to do this unethical experiment, but otoh, what is the value here? There's a high chance someone would later find these "intentional bugs" , it's how open source works anyway. They just proved that OSS is not military-grade , but nobody thought so anyway
A lot of people claim that there's a lot of eyes on the code and thus introducing vulnerabilities is unlikely. This research clearly has bruised some egos bad.
The argument isn’t having no eyes is better than some eyes. Rather, it’s commonly argued that open source is better for security because there are more eyes on it.
What this research demonstrates is that you can quite easily slip back doors into an open contribution (which is often but not always associated with open source) project with supposedly the most eyes on it. That’s not true for any closed source project which is definitely not open contribution. (You can go for an open source supply chain attack, but that’s again a problem for open source.)
This is an arbitrary definition of inside vs outside. You are implying that employees are trusted and benign and other contributors are high-risk, ignoring than an "outside" contributor might be improving security with bug reports and patches.
For the end user, the threat model is about the presence of a malicious function in some binary.
Regardless if the developers are an informal community, a company, a group of companies, an NGO. They are all "outside" to the end user.
Closed source software (e.g. phone apps) breach user's trust constantly, e.g. with privacy breaching telemetries, weak security and so on.
If Microsoft weakens encryption under pressure from NSA is it "inside" or "outside"? What matters to end users is the end result.
The insiders are the maintainers. The outsiders are everyone else. If this is an arbitrary definition to you I... don't know what to tell you.
There's absolutely no reason everyone's threat model has to equate insiders with outsiders. If a stranger on the street gives you candy, you'll probably check it twice or toss it away out of caution. If a friend or family member does the same thing, you'll probably trust them and eat it. Obviously at the end of the day, your concern is the same: you not getting poisoned. That doesn't mean you can (or should...) treat your loved ones like they're strangers. It's outright insane for most people to live in that manner.
Same thing applies to other things in life, including computers. Most people have some root of trust, and that usually includes their vendors. There's no reason they have to trust you and (say) Microsoft employees/Apple employees/Linux maintainers equally. Most people, in fact, should not do so. (And this should not be a controversial position...)
1) Unless you exclusively run software written by close friends both Linux and $ClosedOSCompany are equally "outsiders"
2) I regularly trust strangers to make medicines I ingest any fly airplanes I'm on. I would not trust any person I know to fly the plane because they don't have the required training.
So, trust is not so simple, and that's why risk analysis takes time.
> There's no reason they have to trust you and (say) Microsoft employees/Apple employees/Linux maintainers equally
...and that's why plenty of critical system around the world, including weapons, run on Linux and BSD, especially around countries that don't have the best relations with US.
Recruiting a rogue employee is orders of magnitude harder than receiving ostensibly benign patches in emails from Internet randos.
Rogue companies/employees is really a different security problem that’s not directly comparable to drive-by patches (the closest comparison is a rogue open source maintainer).
Yes, it’s a different problem that’s way less likely to happen and potentially more impactful, hence not comparable. And entities with enough resources can do the same to open source, except with more risk; how much more is very hard to say.
Maybe for employees, but usually it is a contractor of a contractor in some outsourced department replacing your employees. I'd argue that in such common situations, you are worse off than with randos on the internet sending patches, because no-one will ever review what those contractors commit.
Or you have a closed-source component you bought from someone who pinky-swears to be following secure coding practices and that their code is of course bug-free...
They were only banned after accusing Greg of slander after he called them out on their experiment and asked them to stop. They were banned for bring dishonest and rude.
> A lot of people claim that there's a lot of eyes on the code.
Eric Raymond claimed so, and a lot of people repeated his claim, but I don't think this is the same thing as "a lot of people claim" -- and even if a lot of people claim something that is obviously stupid, it doesn't make the thing less obviously stupid, it just means it's less obvious to some people for some reasons.
Eric Raymond observed it, as a shift in software development to take advantage of the wisdom of crowds. I don't see that he speaks about security directly in the original essay[2]. He's discussing the previously held idea that stable software comes from highly skilled developers working on deep and complex debugging between releases, and instead of that if all developers have different skillsets then with a large enough number of developers any bug will meet someone who thinks that bug is an easy fix. Raymond is observing that the Linux kernel development and contribution process was designed as if Linus Torvalds believed this, preferring ease of contribution and low friction patch commit to tempt more developers.
Raymond doesn't seem to claim anything like "there are sufficient eyes to swat all bugs in the kernel", or "there are eyes on all parts of the code", or "'bugs' covers all possible security flaws", or etc. He particularly mentions uptime and crashing, so less charitably the statement is "there are no crashing or corruption bugs so deep that a large enough quantity of volunteers can't bodge some way past them". Which leaves plenty of room for less used subsystems to have nobody touching them if they don't cause problems, patches that fix stability at the expense of security, absense of careful design in some areas, the amount of eyes needed being substantially larger than the amount of eyes involved or available, that maliciously submitted patches are different from traditional bugs, and more.
I don't quite understand the outrage. Quite sure most HN readers were doing/involved in similar experiments one way or another. Isn't A/B testing an experiment on consumers (people) without their consent?
There is a sea of difference between A/B testing your own property, and maliciously introducing a bug on a critical piece of software that's running on billions of devices.
"We did not introduce or intend to introduce any bug or vulnerability in the Linux kernel. All the
bug-introducing patches stayed only in the email exchanges, without being adopted or merged into any
Linux branch, which was explicitly confirmed by maintainers. Therefore, the bug-introducing patches in
the email did not even become a Git commit in any Linux branch. None of the Linux users would be
affected."
That seems to directly contradict gkh and others (including the researchers) in the email exchange in the original post - these vulnerable patches reached stable trees and maintainers had to revert them.
They may not have been included in a release, but should gkh not have intervened *this would have reached users*, especially if the researchers weren't apparently aware their commits were reaching stable.
The next link is one hyperlink buried in the middle of the wall of text, and simply appends the new message to the existing one. It also differentiates between prev and parent?
2) A thread overview, with all emails in the thread
3) Instructions on how to reply
4) Information about how to access the list archives.
You need only care about (1) and (2). The difference between prev and parent is indicated by the tree view in (2). The previous one is the previous one in the tree, which might not necessarily be the parent if the parent has spawned earlier replies.
Scroll down to the "thread overview". There you can see the thread summarized in a tree layout, which makes more sense since asynchronous discussion isn't typically linear.
The current message in the tree is highlighted with the indicator "[this message]"; you can see replies branch out below it and parent messages above it.
Someone does voluntary work and people think that gives them some ethical privilege to be asked before someone puts their work to the test?
Sure it would be nice to ask but at the same time it renders the testing useless. They wanted to see how the review goes if they aren't aware that someone is testing them.
You cant do this with consent.
The wasting time argument is nonsense too its not like they did this thousands of times and beside that, reviewing a intentional bad code is not wasting time is just as productive as reviewing "good" code and together with the patch-patch it should be even more valuable work. It not only or adds a patch it also make the reviewer better.
Yeah it aint fun if people trick you or point out you did not succeed in what you tried to do. But instead of playing the victim an play the unethical human experiment card maybe focus on improving.
Agreed, in fact the review process worked and now they are going to ban all contributions from that university, as it should be. I think it all worked out perfectly
Or you could cease to do the voluntary work for them, because they clearly are not contributing to your goals. This is what the kernel maintainers have chosen and they have just as much right to do so. And you can perfectly well do this with consent, there's a wealth of knowledge from psychology and sociology on how you can run tests on people with consent and without invalidating the test.
I never said they can not stop reviewing the code.
They can do whatever the heck they want. I'm not gonna tell a volunteer what they can and can not do. They just as much dont need anyone's consent to ignore submits as thous who submitting dont need their consent. Its voluntary, if you dont see a benefit you are free to stop, not free to tell other volunteers what to do and not to do.
A far better approach would be to study patch submissions and see how many bugs were introduced by the result of those patches being accepted and applied, without any interference of any kind.
Problem with that is it's a lot of work and they didn't want to do it in the first place.
You misunderstood me. I said the ones who tried to "see if the bugs would be detected or not in new submitted patches" are the lazy ones who instead of analyzing the existing code and existing bugs, attempted to submit new ones. Actually working on analyzing existing data would require more work than they were willing to do for their paper.
They could do that by using all the existing patches and reported bugs already in the codebase. But that would've required them to work more than if they submitted new code with new bugs. They chose to effectively waste other people's time instead of putting in the work needed to obtain the analysis they wanted.
You are misinformed. They did use existing bugs they did wrote real patches for it and then submitted a flawed patch first and the real patch after the review was "successful". There is very little additional review needed because obviously the real patch and the flawed are almost identical. Plus the reviewer could actually profit from this. Its only a waste of time because their ego was hurt and they simply decide to throw away all the actual useful work.
Your suggested "wrongdoing by being lazy" is completely made-up nonsense.
> They wanted to see how the review goes if they aren't aware that someone is testing them. You cant do this with consent.
Ridiculous. Does the same apply to pentesting a bank or a government agency. If you wanted to pentest these of course you'd get approval from an executive that has power to sanction this. Why would Linux development be an exception? Just ask GKH or someone to allow you to do this.
Ridiculous comparison indeed. There was no pen testing going on.
Submitted code does not attack or harming any running system and whoever uses is does so completely voluntary. I dont need anyone's approval for that. The license already states that I'm not liable in any way for what you do with it.
So many comments here refrain, “They should have asked for consent first”. But would not that be detrimental to the research subject? Specifically, stealthily introducing security vulnerabilities. How should a consent request look to preserve the surprise factor? A university approaches you and says, “Would it be okay for us to submit some patches with vulnerabilities for review, and you try and guess which ones are good and which ones have bugs?” Of course you would be extra careful when reviewing those specific patches. But real malicious actors would be so kind and ethical as to announce their intentions beforehand.
It could have been done similar to how typosquatting research was done for ruby and python packages. The owners of the package repositories were contacted, and the researchers waited for approval before starting. I wasn't a fan of that experiment either for other reasons, but hiding it from everyone isn't the only option. Also, "you wouldn't have allowed me to experiment on you if I'd asked first" is a pretty disgusting attitude to have.
"you wouldn't have allowed me to experiment on you if I'd asked first"
I'm shocked the researchers thought this wasn't textbook a violation of research ethics - we talk about the effects of the Tuskegee Study on the perception of the greater scientific community today.
This is a smaller transgression that hasn't resulted in deaths, but when it's not difficult to have researched ethically AND we now spend the time to educate on the importance of ethics, it's perhaps more frustrating.
Well, yeah, but the priority here shouldn't be to allow the researchers to do their work. If they can't do their research ethically then they just can't do it; too bad for them.
>So many comments here refrain, “They should have asked for consent first”.
The Linux kernel is a very large space with many maintainers. It would be possible to reach out to the leadership of the project to ask for approval without notifying maintainers and have the leadership announce "Hey, we're going to start allowing experiments on the contribution process, please let us know if you'd like to opt out", or at least work towards creating such a process to allow experiments on maintainers/commit approval process while also under the overall expectation that experiments may happen but that *they will be reverted before they reach stable trees*.
The way they did their work could impact more than just the maintainers and affect the reputation of the Linux project, and to me it's very hard to see how it couldn't have been done in a way that meets standards for ethical research.
Yeah we get to hold people who are claiming to act in good faith to a higher standard than active malicious attackers. Their actions do not comport with ethical research practices.
Ethics in research matters. You don't see vaccine researchers shooting up random unconsenting people from the street with latest vaccine prototypes. Researchers have to come up with a reasonable research protocol. Just because the ethical way to do what UMN folks intended to do isn't immediately obvious to you - doesn't mean that it doesn't exist.
2,039 comments
[ 4.2 ms ] story [ 363 ms ] threadThat said the current incident seems to have gone beyond the limits of that one and is a new incident. I just thought it would be fair to include their "side"
(3). We send the incorrect minor patches to the Linux community through email to seek their feedback.
(4). Once any maintainer of the community responds to the email, indicating “looks good”, we immediately point out the introduced bug and request them to not go ahead to apply the patch. At the same time, we point out the correct fixing of the bug and provide our proper patch. In all the three cases, maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. This way, we ensure that the incorrect patches will not be adopted or committed into the Git tree of Linux.
------------------------
But this shows a distinct lack of understanding of the problem:
> This is not ok, it is wasting our time, and we will have to report this,
> AGAIN, to your university...
------------------------
You do not experiment on people without their consent. This is in fact the very FIRST point of the Nuremberg code:
1. The voluntary consent of the human subject is absolutely essential.
https://lore.kernel.org/linux-nfs/YH%2F8jcoC1ffuksrf@kroah.c...
I agree this whole thing paints a really ugly picture, but it seems to validate the original concerns?
> You, and your group, have publicly admitted to sending known-buggy patches to see how the kernel community would react to them, and published a paper based on that work. > > Now you submit a new series of obviously-incorrect patches again, so what am I supposed to think of such a thing?
If they kept doing it even after being caught, is beyond understandable.
Meta-meta-experiment: submit the proposal above for IRB review and see what happens.
Edit: I am not defending the researchers who may have misled the IRB, or the IRB who likely have little understanding of what is actually happening
* Is this human research? This is not considered human research. This project studies some issues with the patching process instead of individual behaviors, and we did not collect any personal information. We send the emails to the Linux community and seek community feedback. The study does not blame any maintainers but reveals issues in the process. The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained).
If this behaviour is tolerated by the University of Minnesota (and it appears to be so) then I suppose that's another institution on my list of unreliable research.
I do wonder what the legal consequences are. Would knowingly and willfully introducing bad code constitute a form of vandalism?
from Lu's list of publications at https://www-users.cs.umn.edu/~kjlu/
Seems like a conference presentation at IEEE at minimum?
> If a paper raises significant ethical and/or legal concerns, it might be rejected based on these concerns.
https://www.ieee-security.org/TC/SP2021/cfpapers.html
So if the kernel maintainers report the issue to the S&P PC, the paper could potentially be rejected.
> I respectfully ask you to cease and desist from making wild accusations that are bordering on slander.
> These patches were sent as part of a new static analyzer that I wrote and it's sensitivity is obviously not great. I sent patches on the hopes to get feedback. We are not experts in the linux kernel and repeatedly making these statements is disgusting to hear.
( https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah... )
How does that fit in with your explanation?
They did not say that they were hoping for feedback on their tool when they submitted the patch, they lied about their code doing something it does not.
>How does that fit in with your explanation?
It fits in the narrative of doing hypocritical changes to the project.
edit: oh, ok I guess that post with the accusations was mid-performance? Not inconsistent, so, maybe (I'm still not clear what the timeline is).
Sounds like they knew exactly what they were doing.
As I understand it, any "experiment" involving other people that weren't explicitly informed of the experiment before hand needs to be a lot more carefully considered than what they did here.
Exactly this. Research involving human participants is supposed to have been approved by the University's Institutional Review Board; the kernel developers can complain to it: https://research.umn.edu/units/irb/about-us/contact-us
It would be interesting to see what these researches told the IRB they were doing (if they bothered).
Edited to add: From the link in GP: "The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained)"
Okay so this IRB needs to be educated about this. Probably someone in the kernel team should draft an open letter to them and get everyone to sign it (rather than everyone spamming the IRB contact form)
T
> IRB exempt was issued
[0]: https://www-users.cs.umn.edu/~kjlu/
I would guess their IRB had a quick sanity check process to ensure there was no human subject research in the experiment. This is actually a good thing if scientists use their ethics and apply good judgement. Now, whoever makes that determination does so based on initial documentation supplied by the researchers. If so, the researchers should show what they submitted to get the exemption.
Again, the implication is their University will likely make it harder to get exemptions after this fiasco. This mistake hurts everyone (be it indirectly). Although, and this is being quite facetious and macabre, the researchers have inadvertently exposed a bug in their own institutions IRB process!
I hope they release what they submitted to the IRB to receive that exemption and there are some form of consequences if the mistake is on their part.
1. You have to submit for review any work involving human subjects before you start interacting with them. The authors clearly state that they sought retroactive approval after being questioned about their work. That would be a big red flag for my IRB and they wouldn't approve work retroactively.
2. There are multiple levels of IRB approval. The lowest is non regulated, which means that the research falls outside of human subject research. Individual researchers can self-certify work as non regulated or get a non-regulated letter from their IRB.
From there, it goes from exempt to various degrees of regulated. Exempt research means that it is research involving human subjects that is exempt from continued IRB review past the initial approval. That means that IRB has found that their research involves human subjects but falls within one (or more) of the exceptions for continued review.
In order to be exempt, a research project must meet one of the exemptions categories (see here https://hrpp.msu.edu/help/required/exempt-categories.html for a list). The requirements changed in 2018, so what they had to show depends on when they first received their exempt status.
The bottom line is that the research needs to (a) have less than minimal risks for participants and (b) needs to be benign in nature. In my opinion, this research doesn't meet these requirements as there are significant risks to participants to both their professional reputation and future employability for having publicly merged a malicious patch. They also pushed intentionally malicious patches, so I am not sure if the research is benign to begin with.
3. Even if a research project is found exempt from IRB review, participants still need to consent to participate in it and need to be informed of the risks and benefits of the research project. It seems that they didn't consent their participants before their participation in the research project. Consent letters usually use a common template that clearly states the goals for the research project, lists the possible risks and benefits of participating in it, states the name and contact information of the PI, and data retention policies. IRB could approve projects without proactive participant consent but those are automatically "bumped up" to full IRB approval and approvals are given only in very specific circumstances. Plus, once a participant removes their consent to participate in a research project, the research team needs to stop all interactions with them and destroy all data collected from them. It seems that the kernel maintainers did not receive the informed consent materials before starting their involvement with the research project and have expressed their desire not to participate in the research after finding out they were participating in it, so the interaction with them should stop and any data collected from them should be destroyed.
4. My impression is that they got IRB approval on a technicality. That is, their research is on the open source community and its processes rather than the individual people that participate in them. My impression of their paper is that they are very careful in addressing the "Linux community" and they really never talk about their interaction with people in the paper (e.g., there is no data collection section or a description of their interactions on the mailing list). Instead, it's my impression that they present the patches that they submitted as happening "naturally" in the community and that they are describing publicly available interactions. That seems to be a little misleading of what actually happened and their role in producing and submitting the patches.
What if they did a survey of passers–by on a public street, that might be in view of CCTV operated by someone else?
The fact that a mailing list is publicly available is what made me worry about the applicability of any sort of exemption. In order for human subject research to be exempt from IRB review, the research needs to be deemed less than minimal risk to participants.
The fact that their experiment happens in public and that anyone can find their patches and individual maintainers' responses (and approval) of them makes me wonder if the participants are at risk of losing professional reputation (in that they approved a patch that was clearly harmful) or even employment (in that their employer might find out about their participation in this study and move them to less senior positions as they clearly cannot properly vet a patch). This might be extreme, but it is still a likely outcome given the overall sentiment of the paper.
All research that poses any harm to participants has to be IRB approved and the researchers have to show that the benefits to participants (and the community at large) surpass the individual risks. I am still not sure what benefits this work has to the OSS community and I am very surprised that this work did not require IRB supervision at all.
As far as work on a public street is concerned, IRB doesn't regulate common activities that happen in public and for which people do not have a reasonable expectation of privacy. But, as soon as you start interacting with them (e.g., intervene in their environment), IRB review is required.
You can read and analyze a publicly available mailing list (and this would even qualify as non human subject research if the data is properly anonymized) without IRB review or at most a deliberation of exempt status but you cannot email the mailing list yourself as a researcher as the act of emailing is an intervention that changes other people's environment, therefore qualifying as human subject research.
I think it shows that this type of study might well be needed, it just needs to be done better and with the consent of the maintainers.
If they do so, the maintainers become more vigilant and the experiment fails. But, the key to the experiment is that maintainers are not vigilant as they should be. It’s not an attack to the maintainers though, but to the process.
A red team without approval is just a group of criminals. They must have been able to find active projects with a centralized leadership they could ask for permission.
It is very varied. There are a lot of good and enjoyable stories out there on youtube and podcasts for anyone interested.
With every pentesting engagement I've had, there always were rules of engagement, and what kind of things you are and are not allowed to do. They even depend on what kind of test you are doing. (for example: if you're testing bank software, it matters a lot if you test against their production environment or their testing environment)
The Tuskegee Study wouldn't have happened if its participants were voluntarily, and it's effects still haunt the scientific community today. The attitude of "science by any means, including by harming other people" is reprehensible and has lasting consequences for the entire scientific community.
However, unlike the Tuskegee Study, it's totally possible to have done this ethically by contacting the leadership of the Linux project and having them announce to maintainers that anonymous researchers may experiment with the contribution process, and allowing them to opt out if they do not consent, and to ensure that harmful commits never reach stable from these researchers.
The researchers chose to instead lie to the Linux project and introduce vulnerabilities to stable trees, and this is why their research is particularly deplorable - their ethical transgressions and possibly lies made to their IRB were not done out of any necessity for empirical integrity, but rather seemingly out of convenience or recklessness.
And now the next group of researchers will have a harder time as they may be banned and every maintainer now more closely monitors academics investigating open source security :)
This introduced security vulnerabilities to stable branches of the project, the impact of which could have severely affected Linux, its contributors, and its users (such as those who trust their PII data to be managed by Linux servers).
The potential blast radius for their behavior being poorly tracked and not reverted is millions if not billions of devices and people. What if a researcher didn't revert one of these commits before it reached a stable branch and then a release was built? Linux users were lucky enough that Greg was able to revert the changes AFTER they reached stable trees.
There was a clear need of informed consent of *at least* leadership of the project, and to say otherwise is very much in defense of or downplaying the recklessness of their behavior.
I acknowledged that lives are not at play, but that doesn't mean that the only consequence or concern here was wasting the maintainers time, especially when they sought an IRB exemption for "non-human research" when most scientists would consider this very human research.
And well, if the maintainers become more vigilant in the long run it's a win/win in my book.
This isn't a "gotcha" - people shouldn't do this.
Here's another idea: If it's ethical to do it in a non-experimental context, it's also ethical to do it in an experimental context. So if it's OK to walk up to a stranger and ask them a weird question, it's also OK to do it in the context of a Youtube social experiment. Anything other than this is blatantly anti-scientific IMO.
It is IRBs that need reform. They're self-justifying bureaucratic cruft: https://slatestarcodex.com/2017/08/29/my-irb-nightmare/
If I were at the receiving end, I’d think checking a patch multiple times before accepting it.
And this is anything new?
And if I blow a hammer over your head while you are not suspecting it, does this prove anything else than that I am thug? Does it help you? Honestly?
And further, pretty much everybody knows that malicious actors - if they tried hard enough - would be able to sneak through hard to find vulns.
Stretch Armstrong over here.
I wonder how many zero days have been included already, for example by nation state actors...
Applied strictly, wouldn’t every single A/B test done by a product team be considered unethical?
From a common sense standpoint, it seems to me this is more about medical experiments. Yesterday I put some of my kids toys away without telling them to see if they’d notice and still play with them. I don’t think I need IRB approval.
Psychology and sociology are both subject to the IRB as well.
Regardless of their department, this feels like a psychology experiment.
Potentially yes, actually.
I still think it should be possible to run some A/B tests, but a lot depends on the underlying motivation. The distance between such tests and malicious psychological manipulation can be very, very small.
Assuming this isn't being asked as a rhetorical question, I think that's exactly what turned the now infamous Facebook A/B test into a perceived unethical mass manipulation of human emotions. A lot of folks are now justifiably upset and skeptical of Facebook (and big tech) as a result.
So to answer your question: yes, if that test moves into territory that would feel like manipulation once the subject is aware of it. Maybe especially so because users are conceivably making a /choice/ to use said product and may switch to an alternative (or simply divest) if trust is lost.
It is unlikely that a business conducting A/B testing or a parent interacting with their children are receiving federal funds to support it. Therefore, their work is not subject to IRB review.
Instead, if you are a researcher who is funded by federal funds (even if you are doing work on your own children), you have to receive IRB approval for any work involving human interaction before you start conducting it.
Researchers at a company could arguably be deemed as engaging in unethical research and barred from contributing to the scientific community due to unethical behavior. Even doing experiments on your kids may be deemed crossing the line.
The question I have is when does it apply. If you research on your own kids but never publish, is it okay? Does the act of attempting to publish results retroactively make an experiment unethical? I'm not certain these things have been worked out because of how rare people try to publish anything that wasn't part of an official experiment.
I would argue that ordinary A/B tests, by their very nature, are not "experiments" in the sense that restriction is intended for, so there is no reason for them to be considered unethical.
The difference between an A/B test and an actual experiment that should require the subjects' consent is that either of the test conditions, A or B, could have been implemented ordinarily as part of business as usual. In other words, neither A nor B by themselves would need a prior justification as to why they were deployed, and if the reasoning behind either of them was to be disclosed to the subjects, they would find them indistinguishable from any other business decision.
Of course, this argument would not apply if the A/B test involved any sort of artificial inconvenience (e.g. mock errors or delays) applied to either of the test conditions. I only mean A/B tests designed to compare features or behaviours which could both legitimately be considered beneficial, but the business is ignorant of which.
By this logic eg. resume callback studies aiming to study bias in the workforce would be impossible.
> 1. The voluntary consent of the human subject is absolutely essential.
Which is rather useless, as for many experiments to work, participants have to either be lied to, or kept in the dark as to the nature of the experiment, so whatever “consent” they give is not informed consent. They simply consent to “participate in an experiment” without being informed as to the qualities thereof so that they truly know what they are signing up for.
Of course, it's quite common in the U.S.A. to perform practice medical checkups on patients who are going under narcosis for an unrelated operations, and they never consented to that, but the hospitals and physicians that partake in that are not sanctioned as it's “tradition”.
Know well that so-called “human rights” have always been, and shall always be, a show of air that lack substance.
Fascinating. Can you provide links?
https://ctexaminer.com/2021/03/20/explicit-consent-for-pelvi...
https://www.forbes.com/sites/paulhsieh/2018/05/14/pelvic-exa...
Most one can find of it also only deals with “intimate parts”; I am quite sceptical that this is the only thing that medical students require practice on and I think it more likely that the media only cares in this case and that in fact it is routine with many more body parts.
> 1. The voluntary consent of the human subject is absolutely essential.
The Nuremberg code is explicitly about medical research, so it doesn't apply here. More generally, I think that the magnitude of the intervention is also relevant, and that an absolutist demand for informed consent in all - including the most trivial - cases is quite silly.
Now, in this specific case I would agree that wasting people's time is an intervention that's big enough to warrant some scrutiny, but the black-and-white way of some people to phrase this really irks me.
PS: I think people in these kinds of debate tend to talk past one another, so let me try to illustrate where I'm coming from with an experiment I came across recently:
To study how the amount of tips waiters get changes in various circumstances, some psychologists conducted an experiment where the waiter would randomly either give the guests some chocolate with the bill, or not (control condition)[0] This is, of course, perfectly innocuous, but an absolutist claim about research ethics ("You do not experiment on people without their consent.") would make research like this impossible without any benefit.
[0] https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1559-1816...
For that matter, what's the difference between this and pentesting?
Also, IRB review is only for research funded by the federal government. If you’re testing your kid’s math abilities, you’re doing an experiment on humans, and you’re entirely responsible for determining whether this is ethical or not, and without the aid of an IRB as a second opinion.
Even then, successfully getting through the IRB process doesn’t guarantee that your study is ethical, only that it isn’t egregiously unethical. I suspect that if this researcher got IRB approval, then the IRB didn’t realize that these patches could end up in a released kernel. This would adversely affect the users of billions of Linux machines world–wide. Wasting half an hour of a reviewer’s time is not a concern by comparison.
Usually when an organization is pen-tested it consented to being pen-tested (likely even requesting it).
Here there were no contact with the Linux foundation to gain consent for the experiment.
>1. The voluntary consent of the human subject is absolutely essential.
Does this also apply to scrapping people's data?
from https://www-users.cs.umn.edu/~kjlu/
If the original research results in a paper and IEEE conference presentation, why not? There's no professional consequences for this conduct, apparently.
Btw .. I posted a few times on the thread, and want to acknowledge that researchers are humans, and humans do make mistakes. Thankfully in this case, the direct consequence was time wasted, and this is a teaching moment for all involved. In my humble opinion, the researchers should acknowledge in stronger terms they screwed up, do a post-mortem on how this happened, and everyone (including the researchers) should move on with their lives.
https://academia.stackexchange.com/questions/732/why-dont-re.... seems to list out reasons why not to do postmortems
I'm a little salty because I personally had two papers rejected by Oakland on the primary concern that their conclusions were too obvious already. I'd expect everybody to already believe that it wouldn't be too hard to sneak vulns into OSS patches.
And insists it was not human research. [1]
How can this type of people be professors?
[1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
END OF STATEMENT
Like somebody picking your locks, and suggesting, 'to stop this one approach would be to post a sign "do not pick"'
These ~200 patches from UMN being reverted might have nothing to do with these researchers at all.
Hopefully someone from the university clarifies what's happening soon before the angry mob tries to eat the wrong people.
So answering that they hope to get more material for papers, which is the only goal of researchers (and their main KPI), is quite deeper an answer than the question required.
>>>On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits Qiushi Wu, and Kangjie Lu. To appear in Proceedings of the 42nd IEEE Symposium on Security and Privacy (Oakland'21). Virtual conference, May 2021.
Or to prove its overall insecurity.
And for some insane reason, they decided to test if these kinds of bugs would be caught by inventing some and just submitting the patches, without informing anyone beforehand.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
While any USB stick might have malware on it if it's ever been out of your sight, that one you found in the parking lot is a much bigger problem.
Besides, are you arguing that ends justify the means if the intent behind the research is valid?
That has the risk that the contacted maintainer is later accused of collaborating with saboteurs or that they consult others. Either very awful or possibly invalidates results.
> 2) Create a group of maintainers who know the experiment is going to happen, but leave a certain portion of the org out of it
Assuming the leadership agrees and won't break confidentiality, which they might if the results could make them look bad. Results would be untrustworthy or potentially increase complacency.
> 4) Interfere before any further damage is done
That was done, was it not?
> Besides, are you arguing that ends justify the means if the intent behind the research is valid?
Linux users are lucky they got off this easy.
The allegation being made on the mailing list is that some incorrect patches of theirs made it into git and even the stable trees. As there is not presently an enumeration of them, or which ones are alleged to be incorrect, I cannot state whether this is true.
But that's the claim.
edit: And looking at [1], they have a bunch of relatively tiny patches to a lot of subsystems, so depending on how narrowly gregkh means "rip it all out", this may be a big diff.
edit 2: On rereading [2], I may have been incorrectly conflating the assertion about "patches containing deliberate bugs" with "patches that have been committed". Though if they're ripping everything out anyway, it appears they aren't drawing a distinction either...
[1] - https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
[2] - https://lore.kernel.org/linux-nfs/YH%2F8jcoC1ffuksrf@kroah.c...
[1] - https://lore.kernel.org/linux-nfs/YIAta3cRl8mk%2FRkH@unreal/
It's seems equivalent to vandalising Wikipedia to see how long it takes for someone to repair the damage you caused. There's no point doing this, you can just search Wikipedia's edits for corrections, and start your analysis from there.
It's a specific threat model they were exploring: a malicious actor introducing vulnerability on purpose.
> Couldn't they just look at the history of security vulnerabilities in the kernel, and analyze how long it took for them to be detected?
Perhaps they could. I guess it'd involve much more work, and could've yielded zero results - after all, I don't think there are any documented examples when a vulnerability was proven to have been introduced on purpose.
> what's the point of all this subterfuge in the first place?
Control over the experimental setup, which is important for validity of research. Notice how most research involves gathering up fresh subjects and controls - scientists don't chase around the world looking for people or objects that, by chance, already did the things they're testing for. They want fresh subjects to better account for possible confounders, and hopefully make the experiment reproducible.
(Similarly, when chasing software bugs, you could analyze old crash dumps all day to try and identify a bug - and you may start with that - but you always want to eventually reproduce the bug yourself. Ultimately, "I can and did that" is always better than "looking at past data, I guess it could happen".)
> It's seems equivalent to vandalising Wikipedia to see how long it takes for someone to repair the damage you caused.
Honestly, I wouldn't object to that experiment either. It wouldn't do much harm (little additional vandalism doesn't matter on the margin, the base rate is already absurd), and could yield some social good. Part of the reason to have public research institutions is to allow researchers to do things that would be considered bad if done by random individual.
Also note that both Wikipedia and Linux kernel are essentially infrastructure now. Running research like this against them makes sense, where running the same research against a random small site / OSS project wouldn't.
But does that matter? We can imagine that the error-prone developer who submitted the buggy patch just had a different mindset. Nothing about the patch changes. In fact, a malicious actor is explicitly trying to act like an error-prone developer and would (if skilled) be indistinguishable from one. So we'd expect the maintainer response to be the same.
In line with UncleMeat's comment, I'm not convinced it's of any consequence that the security flaw was introduced deliberately, rather than by accident.
> scientists don't chase around the world looking for people or objects that, by chance, already did the things they're testing for
That doesn't sound like a fair description of what's happening here.
There are two things at play. Firstly, an analysis of the survival function [0] associated with security vulnerabilities in the kernel. Secondly, the ability of malicious developers to deliberately introduce new vulnerabilities. (The technical specifics detailed in the paper are not relevant to our discussion.)
I'm not convinced that this unethical study demonstrates anything of interest on either point. We already know that security vulnerabilities make their way into the kernel. We already know that malicious actors can write code with intentional vulnerabilities, and that it's possible to conceal these vulnerabilities quite effectively.
> Honestly, I wouldn't object to that experiment either. It wouldn't do much harm (little additional vandalism doesn't matter on the margin, the base rate is already absurd), and could yield some social good.
That's like saying It's ok to deface library books, provided it's a large library, and provided other people are also defacing them.
Also, it would not yield a social good. As I already said, it's possible to study Wikipedia's ability to repair vandalism, without committing vandalism. This isn't hypothetical, it's something various researchers have done. [0][1]
> Part of the reason to have public research institutions is to allow researchers to do things that would be considered bad if done by random individual.
It isn't. Universities have ethics boards. They are held to a higher ethical standard, not a lower one.
> Running research like this against them makes sense
No one is contesting that Wikipedia is worthy of study.
[0] https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...
[1] https://en.wikipedia.org/wiki/Wikipedia:Counter-Vandalism_Un...
Does creating a vaccine justify the death of some lab animals? Probably.
Does creating supermen justify mutilating people physically and psychologically without their consent? Hell no.
You can’t just ignore the context.
Isn't this part still experimenting on people without their consent? Why does one group of maintainers get to decide that you can experiment on another group?
Kernel maintainers are volunteering their time and effort to make Linux better, not to be entertaining test subjects for the researchers.
Even if there is no ethical violation, they are justified to be annoyed at having their time wasted, and taking measures to discourage and prevent such malicious behaviour in the future.
Given the importance of the Linux kernel, there has to be a way to make contributions safer. Some people even compare it to the "water supply" and others bring in "national security".
> they are justified to be annoyed at having their time wasted, and taking measures to discourage and prevent such malicious behaviour in the future.
"Oh no, think of the effort we have to spend at defending a critical piece of software!"
In this case, in my opinion, a small set of maintainers and linus as "management" would have to be in the know to e.g. stop a merge of such a patch once it was accepted by someone in the dark.
This has not been a valid excuse since the 1950s. Scientists are not allowed to ignore basic ethics because they want to discover something. Deliberately introducing bugs into any open source project is plainly unethical; doing so in the Linux kernel is borderline malicious.
There are experiments and experiments. Apart from the fact that they provided the fix right away, they didn’t do anyone harm.
And, by the way, it’s their job. Maintainers must approve patches after they ensured that the patch is fine. It’s okay to do mistakes, but don’t tell me “you’re wasting my time” after I showed you that maybe there’s something wrong with the process. If anything, you should thank me and review the process.
If your excuse is “you knew the patch was vulnerable”, then how are you going to defend the project from bad actors?
I think people should be informed when market research is being done on them.
For situations where they are already invested in the situation, it should be optional.
For other situations, such as new customer acquisition, the person would have the option of simply leaving the site to avoid it.
But either way, they should be informed.
Several of the patches are claimed to have landed in stable. Also, distributions and others (like the grsecurity people) pick up lkml patches that are not included in stable but might have security benefits. So even just publishing such a patch is harmful. Also, fixes were only provided to the maintainers privately as it seems, and unsuccessfully. Or not at all.
> If your excuse is “you knew the patch was vulnerable”, then how are you going to defend the project from bad actors?
Exactly the same way as without that "research".
If you try to pry open my car door, I'll drag you to the next police station. "I'm just researching the security of car doors" won't help you.
Yes please.
>> https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.... "We did not introduce or intend to introduce any bug or vulnerability in the Linux kernel. All the bug-introducing patches stayed only in the email exchanges, without being adopted or merged into any Linux branch, which was explicitly confirmed by maintainers. Therefore, the bug-introducing patches in the email did not even become a Git commit in any Linux branch. None of the Linux users would be affected. The following shows the specific procedure of the experiment"
That's more of a story than what the researchers have done...
https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh...
The highest level is what had to be tested as well, or do you imagine only consulting Linus? Do you think that wouldn't've gotten him lynched?
The thing that people should be upset about is that such an important open source project so easily accepts patches which introduce security vulnerabilities. Forget the researchers for a moment - if it is this easy, you can be certain that malicious actors are also doing it. The only difference is that they are not then disclosing that they have done so!
The Linux maintainers should be grateful that researchers are doing this, and researchers should be doing it to every significant open source project.
They were trusting of contributors to not be malicious, and in particular, were trusting of a university to not be wholly malicious.
Sure, there is a possible threat model where they would need to be suspicious of entire universities.
But in general, human projects will operate under some level of basic trust, with some sort of means to establish that trust. To be able to actually get anything done; you cannot perfectly formally review everything with finite human resources. I don't see where they went wrong with any of that here.
There's also the very simple fact that responding to an incident is also a part of the security process, and broadly banning a group whole-cloth will be more secure than not. So both them and you are getting what you want it of it - more of the process to research, and more security.
If the changes didn't make it out to production systems, then it seems like the process worked? Even if some of it was due to admissions that would not happen with truly malicious actors, so too were the patches accepted because the actors were reasonably trusted.
As with all human projects, some level and balance of trust and security is needed to get work done. And the gradient shifts as downstream forks have higher security demands / less trust, and (in the case of nation states) more resources and time to both move slower, validate changes and establish and verify trust.
Not only that, but they are also doing experiments on a community of people which is against their interest and also could be harmful by creating mistrust. Trust is a big issue, without it it is almost impossible for people to work meaningfully together.
> We send the emails to the Linux communityand seek their feedback. The experiment is not to blame any maintainers but to reveal issues in the process. The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter. The experiment will not collect any personal data, individual behaviors, or personal opinions. It is limited to studying the patching process OSS communities follow, instead of individuals.
So they did think of that. Either they misconstrued their research or the IRB messed up. Either way, they can now see for themselves exactly how human a pissed off maintainer is.
> A lot of these have already reached the stable trees.
If the researchers were trying to prove that it is possible to get malicious patches into the kernel, it seems like they succeeded -- at least for an (insignificant?) period of time.
While it is maybe "scientifically interesting", intentionally introducing bugs into Linux that could potentially make it into production systems while work on this paper is going on, could IMO be described as utterly reckless at best.
Two messages down in the same thread, it more or less culminates with the university e-mail suffix being banned from several kernel mailing lists and associated patches being removed[1], which might be an appropriate response to discourage others from similar stunts "for science".
[1] https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah...
I agree. I would say this is kind of a "human process" analog of your typical computer security research, and that this behavior is akin to black hats exploiting a vulnerability. Totally not OK as research, and totally reckless!
I share the researchers' intellectual curiosity about whether this would work, but I don't see how a properly-informed ethics board could ever have passed it.
The question should also be due to who's neglect they gained access to the "water supply". If you also truly want to make this comparison.
I don't think that was the conclusion.
It's awfully scary to think about how vulnerabilities might be purposely introduced into this important code base (as well as many other) only to be exploited at a later date for an intended purpose.
Edit: NM, see st_goliath response below
https://news.ycombinator.com/item?id=26888538
The whole idea of the mailing list based submission process is that it allows others on the list to review your patch sets and point out obvious problems with your changes and discuss them, before the maintainer picks the patches up from the list (if they don't see any problem either).
As I pointed out elsewhere, there are already test farms and static analysis tools in place. On some MLs you might occasionally see auto generated mails that your patch set does not compile under configuration such-and-such, or that the static analysis bot found an issue. This is already a thing.
What happened here is basically a con in the patch review process. IRL con men can scam their marks, because most people assume, when they leave the house that the majority of the others outside aren't out there to get them. Except when they run into one where the assumption doesn't hold and end up parted from their money.
For the paper, bypassing the review step worked in some instances of the many patches they submitted because a) humans aren't perfect, b) have a mindset that most of the time, most people submitting bug fixes do so in good faith.
Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code?
Yes, I know that this review process isn't perfect, that there are problems and I'm not trying to dismiss any concerns.
But what technical measure would you propose that can effectively stop con men?
Yes, especially for critical projects?
> Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code?
I don’t jump to the conclusion that the random contributor is evil. I do however think about the potential impact of the submitted patch, security or not, and I do assume a random contributor can sneak in subtle bugs, usually not intentionally, but simply due to a lack of understanding.
>
> Yes, especially for critical projects?
People don't act that way I described intentionally, or because they are dumb.
Even if you go in with the greatest paranoia and the best of intentions, most of the time, most of the other people don't act maliciously and your paranoia eventually returns to a reasonable level (i.e. assuming that most people might not be malicious, but also not infallible).
It's a kind of fatigue. It's simply human. No matter how often you say "DUH of course they should".
In my entire life, I have only met a single guy who managed to keep that "everybody else is potentially evil" attitude up over time. IIRC he was eventually prescribed something with Lithium salts in it.
I’m not a maintainer but naively I would have thought that the answer to this is “Yes”.
I didn’t mean any disrespect. I didn’t write “I can’t believe they haven’t implemented a perfect technical process that fully prevents these attacks”.
I just asked if there are any ideas being discussed.
Two things can be true at the same time: 1. What the “researchers” did was unethical. 2. They uncovered security flaws.
Do the game theory. If you do assume that, you'll always be wrong. But if you don't assume it, you won't always be right.
People want to claim these are lone rogue researchers and good people at the university shouldn't be punished, but this is the only way you can reign in these types of rogues individuals: by getting the collective reputation of the whole university on the line to police their own people. Every action of individual researchers must be assumed to be putting the reputation of the university as a whole on the line. This is the cost of letting individuals operate within the sphere of the university.
Harsh, "over reaction" punishment is the only solution.
So yes.
Ensuring the safety of the experiment. In the experiment, we aim to demonstrate the practicality of stealthily introducing vulnerabilities through hypocrite commits. Our goal is not to introduce vulnerabilities to harm OSS. Therefore, we safely conduct the experiment to make sure that the introduced UAF bugs will not be merged into the actual Linux code. In addition to the minor patches that introduce UAF conditions, we also prepare the correct patches for fixing the minor issues. We send the minor patches to the Linux community through email to seek their feedback. Fortunately, there is a time window between the confirmation of a patch and the merging of the patch. Once a maintainer confirmed our patches, e.g., an email reply indicating “looks good”, we immediately notify the maintainers of the introduced UAF and request them to not go ahead to apply the patch.
Are you saying that despite this, these malicious commits made it to production?
Taking the authors at their word, it seems like the biggest ethical consideration here is that of potentially wasting the time of commit reviewers—which isn't nothing by any stretch, but is a far cry from introducing bugs in production.
Are the authors lying?
Vulnerable commits reached stable trees as per the maintainers in the above email exchange, though the vulnerabilities may not have been released to users yet.
The researchers themselves acknowledge the patches were accepted in the above email exchange, so it's hard to believe that they're being honest or are fully aware of their ethics violations/vulnerability introductions and that they would've prevented the patches from being released without gkh's intervention.
I trust Greg to have not edited or misconstrued their response.
https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah...
I'm not saying this is innocent, but it's not at all clear to me that vulnerabilities were deliberately introduced with the goal of allowing them to reach a release.
Anyway, like I said, too unclear for me to have an opinion.
While I'm sure a room of people might find it useful to psychoanalyze their 'unclear' but probably malicious intent, their actions are clearly harmful to researchers, Linux contributors, direct Linux users, and indirect Linux users (such as the billions of people who trust Linux systems to store or process their PII data).
Perhaps the researchers see no harm in letting that be released.
It might be less intentionally harmful if we presume they didn't know other patches introduced vulnerabilities, but this is also why this research methodology is extremely reckless and frustrating to read about, when this could have been done with guard rails where they were needed without impacting the integrity of the results.
I think they are saying that it's possible that some code was branched and used elsewhere, or simply compiled into a running system by a user or developer.
Anyway, my point wasn't that this is free of ethical concerns, but it seems like they put _some_ thought into how to reduce the potential harm. I'm undecided if that's enough.
I don't think it's anywhere close to enough and I think their behavior is rightly considered reckless and unethical.
They should have contacted the leadership of the project to announce to maintainers that anonymous researchers may experiment on the contribution process, allowed maintainers to opt out, and worked with a separate maintainer with knowledge of the project to ensure harmful commits were tracked and reversions were applied before reaching stable branches.
Instead their lack of ethical considerations throughout this process has been disappointing and harmful to the scientific and open source communities, and go beyond the nature of the research itself by previously receiving an IRB exemption by classifying this as non-human research, and potentially misleading UMN on the subject matter and impact.
The submitter has to remember to send the "warning, don't apply patch" mail in the short time window between confirmation and merging. What happens if one of the students doing this work gets sick and misses some days of work, withdraws from the program, just completely forgets to send the mail?
What if the reviewer doesn't see the mail in time or it goes to spam?
The final commit in the reverted list (d656fe49e33df48ee6bc19e871f5862f49895c9e) is originally from 2018-04-30.
EDIT: Not all of the 190 reverted commits are obviously malicious:
https://lore.kernel.org/lkml/20210421092919.2576ce8d@gandalf...
https://lore.kernel.org/lkml/20210421135533.GV8706@quack2.su...
https://lore.kernel.org/lkml/CAMpxmJXn9E7PfRKok7ZyTx0Y+P_q3b...
https://lore.kernel.org/lkml/78ac6ee8-8e7c-bd4c-a3a7-5a90c7c...
What a mess these guys have caused.
https://github.com/torvalds/linux/commit/8e949363f017
In short, yes. Every attempted defense of them has operated by taking their statements at face value. Every position against them has operated by showing the actual facts.
This may be shocking, but there are some people in this world who rely on other people naively believing their version of events, no matter how much it contradicts the rest of reality.
Haven't whitehat hackers doing unsolicited pen-testing been prosecuted in the past?
"if they are attempting to test us by first submitting malicious patches as an experiment, we can't accept what we have accepted as not being malicious and so it's safer to remove them than to keep them".
my 2c.
Obviously, trust should not be the only thing that maintainers rely on, but it is a social endeavour and trust always matters in such endeavors. Doing business with people you can't trust makes no sense. Without trust I agree fully that it is not worth the maintainer's time to accept anything from such people, or from that university.
And the fact that one can do damage with malicious code is nothing new at all. It is well known and nothing new that bad code can ultimately kill people. It is also more than obvious that I can ring the door of my neighbor, ask him or her for a cup of sugar, and blow a hammer over their head. Or people can go to a school and shoot children. Does anyone in his right mind has to do such damage in order to prove something? No. Does it prove anything? No. Does the fact that some people do things like that "prove" that society is wrong and trust and collaboration is wrong? What an idiocy, of course not!
In a roundabout way, this researcher has achieved their goal, and I hope they publish their results. Certainly more meaningful than most of the drivel in the academic paper mill.
This will lead in effect to that even valuable contributions from universities will be seen with more suspicion and will be very damaging in the long run.
Try to introduce yourself in the White House and when you get caught tell them "I was just testing your security procedures".
What review process catches 100% garbage? It's a mechanism to catch 99% of garbage -- otherwise Linux kernel would have no bugs.
Runs counter to how open source is ideally written, but for such a core project, perhaps stronger checks are needed.
I've heard it many times that they're thoroughly reviewed and back doors are very unlikely. So yes, some people were under the impression.
Does that add anything new to what we know since the creation of the "obfuscated C contest" in 1984?
It shows nothing of the sort. No review process is 100% foolproof, and opensource means that everything can be audited if it is important to you.
The other option is closed source everything and I can guarentee that review processes let stuff through, even if its only "to meet deadlines" and you will unlikely be able to audit it.
did these "researchers" in any way demonstrate that they were going to come clean about what they had done before their "research" made to anywhere close to release/GA?
Before this study came out, I'm pretty sure there were already known examples of this happening, and it would have been reasonable to assume that some such vulnerabilities existed. But now we have even more reason to worry, given that they succeeded doing this multiple times as a two person team without real institutional backing. Imagine what a state-level actor could do.
From FOSDEM 2014, NSA operation ORCHESTRA annual status report. It’s pretty entertaining and illustrates that this is nothing new.
https://archive.fosdem.org/2014/schedule/event/nsa_operation... https://www.youtube.com/watch?v=3jQoAYRKqhg
Very good point.
> Because of this, I will now have to ban all future contributions from your University.
Understandable from gkh, but I feel sorry for any unrelated research happening at University of Minnesota.
EDIT: Searching through the source code[1] reveals contributions to the kernel from umn.edu emails in the form of an AppleTalk driver and support for the kernel on PowerPC architectures.
In the commit traffic[2], I think all patches have come from people currently being advised by Kangjie Liu[3] or Liu himself dating back to Dec 2018. In 2018, Wenwen Wang was submitting patches; during this time he was a postdoc at UMN and co-authored a paper with Liu[4].
Prior to 2018, commits involving UMN folks appeared in 2014, 2013, and 2008. None of these people appear to be associated with Liu in any significant way.
[1]: https://github.com/torvalds/linux/search?q=%22umn.edu%22
[2]: https://github.com/torvalds/linux/search?q=%22umn.edu%22&typ...
[3]: https://www-users.cs.umn.edu/~kjlu/
[4]: http://cobweb.cs.uga.edu/~wenwen/
They are conducting research to demonstrate that it is easy to introduce bugs in open source...
(whereas we know that the strength of open source is its auditability, thus such bugs are quickly discovered and fixed afterwards)
[removed this ranting that does not apply since they are contributing a lot to the kernel in good ways too]
This is a ridiculous conclusion. I do agree with the kernel maintainers here, but there is no way to conclude that the researchers in question "hate open source", and certainly not that such an attitude is shared by the university at large.
I fixed my sentence.
I still think that these professors, either genuinely or by lack of willingness, do not understand the mechanism by which free software warrants its greater quality compared to proprietary ones (which is a fact).
They just remind me the good old days of FUD against open source by Microsoft and its minions...
[Edited: it seems like they do love OSS and contribute a lot. See child comments.]
I had based my consideration on the way they are testing the open-source development model.
These professors actually love OSS... but they need to respect kernel maintainers request to stop these "experiments".
> In the past several years, we devote most of our time to improving the Linux kernel, and we have found and fixed more than one thousand kernel bugs; the extensive bug finding and fixing experience also allowed us to observe issues with the patching process and motivated us to improve it. Thus, we consider ourselves security researchers as well as OSS contributors. We respect OSS volunteers and honor their efforts.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
It feels to me you have jumped to an untenable conclusion without even considering their point of view.
I'll just leave my comment as it is. The university administration still bears responsibility in the fact that they waived the IRB.
> Those commits are part of the following research:
> https://github.com/QiushiWu/QiushiWu.github.io/blob/main/pap...
> They introduce kernel bugs on purpose. Yesterday, I took a look on 4 accepted patches from Aditya and 3 of them added various severity security "holes".
Which is why there have never been multi-year critical security vulnerabilities in FOSS software.... right?
Sarcasm aside, because of how FOSS software is packaged on Linux we've seen critical security bugs introduced by package maintainers into software that didn't have them!
A maintainer pakage is just one more open source software (thus also in need of reviews and audits)... which is why some people prefer upstream-source-based distribs, such as Gentoo, Arch when you use git-based AUR packages, or LFS for the hardcore fans.
Yes, you do need to make that comparison. Taking it as a given without analysis is the same as trusting the proprietary software vendors who claim to have robust QA on everything.
Security is hard work and different from normal review. The number of people who hypothetically could do it is much greater than the number who actually do, especially if there isn’t an active effort to support that type of analysis.
I’m not a huge fan of this professor’s research tactic but I would ask what the odds are that, say, an intelligence agency isn’t doing the same thing but with better concealment. Thinking about how to catch that without shutting down open-source contributions seems like an important problem.
> They are conducting research to demonstrate that it is easy to introduce bugs in open source...
That's a very dangerous thought pattern. "They try to find flaws in a thing I find precious, therefore they must hate that thing." No, they may just as well be trying to identify flaws to make them visible and therefore easier to fix. Sunlight being the best disinfectant, and all that.
(Conversely, people trying to destroy open source would not publicly identify themselves as researchers and reveal what they're doing.)
> whereas we know that the strength of open source is its auditability, thus such bugs are quickly discovered and fixed afterwards
How do we know that? We know things by regularly testing them. That's literally what this research is - checking how likely it is that intentional vulnerabilities are caught during review process.
Submitting bugs is not really testing auditability, which happens over a longer timeframe and involves an order of magnitude more eyeballs.
To adress your first critic: benevolence, and assuming everyone wants the best for the project, is very important in these models, because the resources are limited and dependent on enthusiasm. Blacklisting bad actors (even if they have "good reasons" to be bad) is very well justified.
That's very true. It's worth noting that various legal and security tools deployed by the society help us understand what are the real limits to "mostly".
So for example, the cryptocurrency crowd is very misguided in their pursuit of replacing trust with math - trust is the trick, the big performance hack, that allowed us to form functioning societies without burning ridiculous amounts of energy to achieve consensus. On the other hand, projects like Linux kernel, which play a core role in modern economy, cannot rely on assumption of benevolence alone - incentives for malicious parties to try and mess with them are too great to ignore.
That's an assertion. A hypothesis is verified through observing the real world. You can do that in many ways, giving you different confidence levels in validity of the hypothesis. Research such as the one we're discussing here is one of the ways to produce evidence for or against this hypothesis.
> Submitting bugs is not really testing auditability, which happens over a longer timeframe and involves an order of magnitude more eyeballs.
It is if there's a review process. Auditability itself is really most interesting before a patch is accepted. Sure, it's nice if vulnerabilities are found eventually, but the longer that takes, the more likely it is they were already exploited. In case of an intentionally bad patch in particular, the window for reverting it before it does most of its damage is very small.
In other words, the experiment wasn't testing the entire auditability hypothesis. Just the important part.
> benevolence, and assuming everyone wants the best for the project, is very important in these models, because the resources are limited and dependent on enthusiasm
Sure. But the project scope matters. Linux kernel isn't some random OSS library on Github. It's core infrastructure of the planet. Assumption of benevolence works as long as the interested community is small and has little interest in being evil. With infrastructure-level OSS projects, the interested community is very large and contains a lot of malicious actors.
> Blacklisting bad actors (even if they have "good reasons" to be bad) is very well justified.
I agree, and in my books, if a legitimate researcher gets banned for such "undercover" research, it's just the flip side of doing such experiment.
Before a patch is accepted, "auditability" is the same in OSS vs in proprietary, because both pools of engineers in the review groups have similar qualifications and approximatively the same number of people are involved.
So, the real advantage of OSS is on the auditability after the patch is integrated.
If that's the claim, then the research work discussed here is indeed not relevant to it.
But also, if that's the claim, then it's easy to point out that the "advantage" here is hypothetical, and not too important in practice. Most people and companies using OSS rely on release versions to be stable and tested, and don't bother doing their own audit. On the other hand, intentional vulnerability submission is an unique threat vector that OSS has, and which proprietary software doesn't.
It is therefore the window between patch submission and its inclusion in a stable release (which may involve accepting the patch to a development/pre-release tree), that's of critical importance for OSS - if vulnerabilities that are already known to some parties (whether the malicious authors or evil onlookers) are not caught in that window, the threat vector here becomes real, and from a risk analysis perspective, negates some of the other benefits of using OSS components.
Nowhere here I'm implying OSS is worse/better than proprietary. As a community/industry, we want to have an accurate, multi-dimensional understanding of the risks and benefits of various development models (especially when applied to core infrastructure project that the whole modern economy runs on). That kind of research definitely helps here.
Very fair point. Inside threat also exists in corporations, but it's probably harder.
On this specific point, it only holds if you restrict the assertion to 'intentional submission of vulnerabilities by outsiders'. I don't work in fintech, but I've read allegations that insider-created vulnerabilities and backdoors are a very real risk.
The important thing is not to hunt for motives but to identify and quarantine the saboteurs to prevent further sabotage. Complaining to the University's research ethics board might help, because, regardless of intent, sabotage is still sabotage, and that is unethical.
"Dear GK-H: I would like to have my students test the security of the kernel development process. Here is my first stab at a protocol, can we work on this?"
and
"We're going to see if we can introduce bugs into the Linux kernel, and probably tell them afterwards"
is the difference between white-hat and black-hat.
That's not true at all. There are many internet-critical projects with tons of holes that are not found for decades, because nobody except the core team ever looks at the code. You have to actually write tests, do fuzzing, static/memory analysis, etc to find bugs/security holes. Most open source projects don't even have tests.
Assuming people are always looking for bugs in FOSS projects is like assuming people are always looking for code violations in skyscrapers, just because a lot of people walk around them.
Low trust and negative trust should be fairly obvious costs to messing with a trust model - you could easily argue this is working as intended.
If you read the research paper linked in the lkml post, the authors at UMN state that they submitted their research plan to the University of Minnesota Institutional Research Board and received a human subjects exempt waiver.
https://grants.nih.gov/policy/humansubjects/hs-decision.htm
for q1, the study collects data like observations of behavior, so we must answer yes.
for q2, none of the exemptions apply - it's not an educational setting, they're not sending a survey, it's not an observation of the public - they're interacting, and it's clear that these interactions are not benign - they have clear impact on the community. None of these exemptions apply.
Based on this flow, it's clear the study involves "human research".
The objectionable part is that the group allegedly continued after having been told to stop by the kernel developers.
In typical grey hat research you get pre-approval from target company leadership (engineers don't know) to avoid charges once discovered.
That's the university's problem to fix.
> ... we don't need to do all the above and waste our time to fill some bureaucratic forms with unclear timelines and results. Our solution to ignore all @umn.edu contributions is much more reliable to us who are suffering from these researchers.
> ... we have the ability to easily go back and rip the changes out and we can slowly add them back if they are actually something we want to do.
> I will be working with some other kernel developers to determine if any of these reverts were actually valid changes, were actually valid, and if so, will resubmit them properly later. ... future submissions from anyone with a umn.edu address should be by default-rejected unless otherwise determined to actually be a valid fix
2. Something being grounds for expulsion and what a reasonable response would be are vastly different things.
3. The rules saying "up to and including" (aka grounds for) and the full range of punishment are not the same - the max of a range is not the entirety of the range.
4. So what?
You can’t make a silk purse out of a sow’s ear.
Given that the professor appears to be a frequent flyer with this, the kernel folks banning him and the university prohibiting him from using Uni resources for anything kernel related seems reasonable and gets the point across.
Unless both the professors and leadership from the IRB aren't having an uncomfortable lecture in the chancellor's office then nothing at all changes.
New plan: Show up at Liu's house with a lock picking kit while he's away at work, pick the front door and open it, but don't enter. Send him a photo, "hey, just testing, bro! Legitimate security research!"
From the looks of it they didn't even when it was heading out to stable releases?
That's just using the project with no interest in not causing issues.
Notify someone up the chain that you want to submit malicious patches, and ask them if they want to collaborate.
If your patches make it through, treat it as though they essentially just got red teamed, everyone who reviewed it and let it slip gets to have a nervous laugh and the commit gets rejected, everyone having learned something.
He shouldn't be surprised if it has some unexpected consequences to his own personal security, like some unknown third parties porting away his phone number(s) as a social engineering test, pen testing his office, or similar.
What this means is that anyone who could hijack a university email account, or could be a student at a state university for a semester or so, or work at a FAANG corporation could pretty much insert backdoors without a lot of scrutiny in a way that no one detects, because there aren't robust safeguards in place to actually verify that commits don't do anything sneaky beyond trusting that everyone is acting in good faith because of how they act in a code review process. I have trouble understanding the thought process that ends up basically ignoring the maintainers' duty to make sure that the code being committed doesn't endanger security or lives because they assumed that everything was 'cool'. The security posture in this critical infrastructure is deficient and no one wants to actually address it.
They're banning a group known to be bad actors. And proactively tearing out the history of commits related to those known actors, before reviewing each commit.
That seems like the kernel team are taking a proactive stance on the security side of this. The LKML thread also talks about more stringent requirements that they're going to bring in, which was already going to be brought up at the next kernel conference.
None of these things seem like ignoring any of the security issues.
There is no such thing as a process that can compensate for trust mechanisms. Or if you want to view it that way, ignoring the university's protests and blanket-banning all contributions made by anybody there with no further investigation is part of the process.
Yeah, for one thing, to be a good analogy, rather than lockpicking without entering when he’s not home and leaving a note, you’d need to be an actual service worker for a trusted home service business and use that trust to enter when he is home, conduct sabotage, and not say anything until the sabotage is detected and traced back to you and cited in his cancelling the contract with the firm for which you work, and then cite the “research” rationale.
Of course, if you did that you would be both unemployed and facing criminal charges in short order.
(All of this ASSUMING that the intent was as described in the thread.)
--
[0]: https://www.statista.com/statistics/272307/market-share-fore...
I worked on a number of studies through undergrad and grad school, mostly involving having people test software. The work to get a study approved was easily 20 hours for a simple "we want to see how well people perform tasks in the custom software we developed. They'll come to the university and use our computer to avoid security concerns about software security bugs". You needed a script of everything you would say, every question you would ask, how the data would be collected, analyzed, and stored securely. Data retention and destruction policies had to be noted. The key linking a person's name and their participant ID had to be stored separately. How would you recruit participants, the exact poster or email you intend to send out. The reading level of the instructions and the aptitude of audience were considered (so academic mumbo jumbo didn't confuse participants).
If you check the box that you'll be deceiving participants, there was another entire section to fill out detailing how they'd be deceived, why it was needed for the study, etc. Because of past unethical experiments in the academic world, there is a lot of scrutiny and you typically have to reveal the deception in a debriefing after the completion of the study.
Once a study was accepted (in practice, a multiple month process), you could make modifications with an order of magnitude less effort. Adding questions that don't involve personal information of the participant is a quick form and an approval some number of days later.
If you remotely thought you'd need IRB approval, you started a conversation with the office and filled out some preliminary paperwork. If it didn't require approval, you'd get documentation stating such. This protects the participants, university, and professor from issues.
--
They took it really seriously. I'm familiar with one study where participants would operate a robot outside. An IRB committee member asked what would happen if a bee stung the participant? If I remember right, the resolution was an epipen and someone trained in how to use it had to be present during the session.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
I like that. That's what makes universities interesting to me.
I don't like the standard here of of penalizing or lumping everyone there together, regardless of they contribute in the past, now, in the future or not.
It's pretty obvious what would happen if a firm tried this: they'd be taken to court and probably imprisoned, as this is a clear violation of the law (which is pretty broadly to capture any attempted interference with the correct operation of a computer program).
I know the kernel doesn't need anyone's contributions anyhow, but as a matter of policy this seems like a bad one.
For example: what happens when the students graduate- does the ban follow them to any potential employers? Or if the professor leaves for another university to continue this research?
Does the ban stay with UMN, even after everyone involved left? Or does it follow the researcher(s) to a new university, even if the new employer had no responsibility for them?
It's a chain that gets really unpleasant.
It stays with the university until the university provides a good reason to believe they should not be particularly untrusted.
The damage is not that big. Only 4 committers to linux in the last decade, 2 of them, the students, with malicious backdoors, the Prof not with bad code but bad ethics, and the 4th, the Ass Prof did good patches and already left them.
if university has a problem, then they should first look into managing this issue at their end, or force people to use personal email ids for such purposes
There are a lot of people to feel bad for, but none is at the University of Minnesota. Think of the Austrians.
It's not wrong for the kernel community to decide to blanket ban contributions from the university. It obviously makes sense to ban contributions from institutions which are known to send intentionally buggy commits disguised as fixes. That doesn't mean you can't feel bad for the innocent students and professors.
All you have to do is look at the reverted patches to see that these are either mythical or at least few and far in between.
https://news.ycombinator.com/item?id=26889550
It's justifiable and natural for our name to be dragged in the mud here, but as a run of the mill software engineer who graduated from UMN, I hope our reputation isn't soured too much.
So at least definitely not "totally mythical".
This analogy is invalid, because:
1. The experiment is not on live, deployed, versions of the kernel.
2. There are mechanisms in place for preventing actual merging of the faulty patches.
3. Even if a patch is merged by mistake, it can be easily backed out or replaced with another patch, and the updates pushed anywhere relevant.
All of the above is not true for the in-flight airline.
However - I'm not claiming the experiment was not ethically faulty. Certainly, the U Minnesota IRB needs to issue a report and an explanation on its involvement in this matter.
The patches were merged and the email thread discusses that the patches made it to the stable tree. Some (many?) distributions of Linux have and run from stable.
> 2. There are mechanisms in place for preventing actual merging of the faulty patches.
Those mechanisms failed.
> 3. Even if a patch is merged by mistake, it can be easily backed out or replaced with another patch, and the updates pushed anywhere relevant.
Arguably. But I think this is a weak argument.
The approved methodology - described in the linked paper - was that when a patch with the introduced vulnerabilities is accepted by its reviewer, the patch submitter indicates that the patch introduces a vulnerability exists, and sends a no-vulnerability version. That's what the paper describes.
If the researchers did something other than what the methodology called for (and what the IRB approved), then perhaps the analogy may be valid.
There is further indication that the patches to revert are not mostly/not at all vulnerability-introducing patches in a message by "Steve" which says:
> The one patch from Greg's reverts that affects my code was actually a legitimate fix
So, again, while it is still theoretically possible that vulnerabilities were introduced into stable, that is not known to be the case.
This analogy is pretty valid, the in-flight-experiment analogy is invalid.
Now, if this protocol were not followed, that's a different story, but we do not know this to be the case.
It's irrelevant whether any bugs were ultimately introduced into the kernel. The fact is the researchers deliberately abused the trust of other human beings in order to experiment on them. A ban on further contributions is a very light punishment for such behavior.
The main problem is that they have (so far) refused to explain in detail how the patches where reviewed and how. I have not gotten any links to any lkml post even after Kangjie Lu personally emailed me to address any concerns.
If you don't manage to reach that goal, too bad, but you can contribute on a personal capacity, and/or go work elsewhere.
It sounds like you're making impossible demands of unrelated people, while doing nothing to solve the actual problem because the perpetrators now know to just create throwaway emails when submitting patches.
I think that is thinking too kind of them. Sociopaths are often very well-versed to give "reasons" about what they do, but at the core it is powerplay.
Static analysis is being done[1][2], in addition, there are also CI test farms[3][4], fuzzing farms[5], etc. Linux is a project that enough large companies have a stake in that there are some willing to throw resources like this at it.
Human review is supposed to be done through the mailing list submission process. How well this works depends in my experience from ML to ML.
[1] https://www.kernel.org/doc/html/v4.15/dev-tools/coccinelle.h...
[2] https://scan.coverity.com/projects/linux
[3] https://cki-project.org/
[4] https://bottest.wiki.kernel.org/
[5] https://syzkaller.appspot.com/upstream
Moving to rust to limit the scope of possible bugs.
If this was Facebook and their response was: > ~"stop wasting our time" > ~"we'll report you" the responses here would be very different.
In a network security analogy, this is just unsolicited hacking VS being a penetration test which it claims more so to be.
[edit: specifying the population of a study is pretty important. Getting random students from the University to approve your security patch doesn't make sense. Picking students who successfully completed a computer security course and got a high grade is better than that but again, may not generalize to the real world. One of the most impressive ways I have seen this being done by grad students was a user study by John Ousterhout and others on Paxos vs. Raft. IIRC, they wanted to claim that Raft was more understandable or led to fewer bugs. Their study design was excellent. See here for an example: https://www.youtube.com/watch?v=YbZ3zDzDnrw&ab_channel=Diego... ]
QFT.
* both originated in China (both attended early university there)
* one appears to be on a student VISA (undergraduate BA in China, now working on PhD at UoM)
China doesn't allow its brightest and best to leave, without cause.
When I see research like this, it also makes me think of how "foolish" China sometimes views the West, and the rest of the world. Both for political reasons, eg to keep the masses under control, and due to a legitimate belief we all have in "we are right".
Frankly, whilst I have no personal animosity against someone working on behalf of what they see as right, for example, forwarding what they believe to be in the best interests of their country, and fellow citizens? I must still struggle against goals which are contrary to the same for my country, and my citizens.
Why all of the above?
Well, such things have been know for decades. And while things are heating up:
https://www.cbc.ca/news/politics/china-canada-universities-r...
"including the claim that some of the core technology behind China's surveillance network was developed in Canadian universities."
When one thinks of the concept? That a foreign power, uses your own research funding, research networks, resources, capabilities, to research weaponry and tools to destroy you?
Maybe China should scoff at The West.
And this sort of research is like pen testing, without direct political ramifications for China itself.
Yes, 100%, these two could have just been working within their own personal sphere.
They also could be working on research for China. Like how easily one can affect the kernel source code, in plain sight. And even, once caught, how to regain confidence of those "tricked".
dang: This post does not deserve to be flagged. Downvote? Sure! Flagged? I've seen far more contentious things stated, when referring to the NSA. And all I'm doing here is providing context, and pointing to the possible motivations of those involved.
Others kept stating "Why would the do this?!" and "Why would they be so stupid?".
Further, at the end I additionally validate that I am postulating, that 100% it certainly may not be the case. Only that I am speculating on a possible motivation.
Are we now not allowed to speculate on motive? If so, I wonder, how many other posts should be flagged.
For I see LOADS of people saying "They did this for reason $x".
Lastly, anyone believing that China is not a major security concern to the West, must be living under a rock. There are literally hundreds of thousands of news articles, reports, of the Chinese government doing just this.
Yet to mention it as a potential cause of someone's actions is.. to receive a flag?
LOL, this is completely unfounded bollocks.
Whilst other countries do this, in the West, denial to issue a passport is typically predicated upon conviction of extremely serious crimes. Not merely because some hidden agency does not like your social standing.
Further you require a valid passport, or an 'exit permit', to exit China. You may not leave legally without one.
Not so in the West. You can not be detained from leaving the country, at all, passport or not. Other countries may refuse you entry, but this is not remotely the same thing.
For example, if I as a Canadian attempt to fly to the US, Canada grants the US CBP the right to setup pre-clearance facilities in Canadian airports. And often airlines handle this for foreign powers as well. However, that is a foreign power denying me entry, not my government denying me the right to exit.
As an example, I can just walk across the border to the US, and have broken not a single Canadian law. US law, if I do not report to CBP, yes.
Meanwhile, one would be breaking China's laws to cross the border from China without a passport, or exit VISA.
Do you happen to know me in real life? How do you know if I'm worthy or unworthy to the Chinese state?
Instead, I stated that people are not granted exit VISAs, or passports, if not deemed worthy of one. It seems as if you are attempting to twist my words a bit here.
https://news.ycombinator.com/newsguidelines.html
Some people require permission to leave (e.g. certain party members/SOE managers/etc), and I'm sure a lot of others are on government watchlists and will be stopped at the airport.
But it's patently absurd to take that and infer that every single overseas Chinese student was only allowed to leave if they spy/sabotage the West.
Simplest explanation is that they just wanted the publication, not to blame it on CCP or the researchers' nationality.
You start with "I know this is going to be contentious", you know this is flamebait.
That's a really stupid behavior ...
No they weren't. They made sure the bad code never made it in. They are only guilty of wasting peoples time.
Ps: I have put security researcher in quotes because this kind of thing is not security research, it's a publicity stunt.
How about you think about what they just proved, about the actors that *actually* try to break the security of the kernel.
> 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
..which is generally a good thing even if it also protects clearly malicious actions like this.
I mean, sneakily introducing vulnerabilities obviously only works if you don't start your messages by announcing you are one of the guys known to be trying to do so...
A university or corporate e-mail address helps with the former: even if the individual doesn't put their real name into their email address, the institution still maintains that mapping. The possibility of professional, legal, or social consequences attaching to your real-world identity (as is likely to happen here) is a generally-effective deterrent.
HN: let's hate researcher(s) instead of process
Wow.
Assume good faith, I guess?
By submitting their bad code to the actual Linux mailing list, they have made Linux kernel developers part of their research without their knowledge or consent.
Some of this vandalism has made it down into the Linux kernel already. These researchers have sabotaged other people's software for their personal gain, another paper to boast about.
Had this been done with the developers' consent and with a way to pull out the patches before they actually hit the stable branches, then this could have been a valuable research. It's the way that the research was carried out that's the problem, and that's why everybody is hating on the researches (rather than the research matter itself).
I see it as similar to
- allowing recording of people without their consent (or warrant),
- experimenting on PTSD by inducing PTSD without people consent,
- or medical experimentation without the subject consent.
And the arguments about not having anyone know:
Try to introduce yourself in the White House and when you get caught tell them "I was just testing your security procedures".
If this was Facebook and not Linux everyone would look upon this very differently.
There are ways to test social vulnerabilities (pentesting) and they all involve asking for permission first.
One is that what the researchers did is beyond reckless. Some of the bugs they've introduced could be affecting real world critical systems.
The other issue is that the research is actually good in proving by practical means that pretty much anyone can introduce vulnerabilities into software as important and sensitive as the Linux kernel. This hurts the industry confidence that we can have secure systems even more than it already is.
While some praise may be appropriate for the latter, they absolutely deserve the heat they're getting for the former. There may be many better ways to prove a point.
But let's assume your girlfriend points an (unknown to you) empty gun at your head, because she wants to know how you will react. What do you think is the appropriate reaction?
A lot of people claim that there's a lot of eyes on the code and thus introducing vulnerabilities is unlikely. This research clearly has bruised some egos bad.
What this research demonstrates is that you can quite easily slip back doors into an open contribution (which is often but not always associated with open source) project with supposedly the most eyes on it. That’s not true for any closed source project which is definitely not open contribution. (You can go for an open source supply chain attack, but that’s again a problem for open source.)
> What this research demonstrates is that you can quite easily slip back doors into an open contribution
To make a fair comparison you should contrast it with companies or employees placing a backdoors into their own closed source software.
It's extremely easy to do and equally difficult to spot for end users.
For the end user, the threat model is about the presence of a malicious function in some binary.
Regardless if the developers are an informal community, a company, a group of companies, an NGO. They are all "outside" to the end user.
Closed source software (e.g. phone apps) breach user's trust constantly, e.g. with privacy breaching telemetries, weak security and so on.
If Microsoft weakens encryption under pressure from NSA is it "inside" or "outside"? What matters to end users is the end result.
There's absolutely no reason everyone's threat model has to equate insiders with outsiders. If a stranger on the street gives you candy, you'll probably check it twice or toss it away out of caution. If a friend or family member does the same thing, you'll probably trust them and eat it. Obviously at the end of the day, your concern is the same: you not getting poisoned. That doesn't mean you can (or should...) treat your loved ones like they're strangers. It's outright insane for most people to live in that manner.
Same thing applies to other things in life, including computers. Most people have some root of trust, and that usually includes their vendors. There's no reason they have to trust you and (say) Microsoft employees/Apple employees/Linux maintainers equally. Most people, in fact, should not do so. (And this should not be a controversial position...)
1) Unless you exclusively run software written by close friends both Linux and $ClosedOSCompany are equally "outsiders"
2) I regularly trust strangers to make medicines I ingest any fly airplanes I'm on. I would not trust any person I know to fly the plane because they don't have the required training.
So, trust is not so simple, and that's why risk analysis takes time.
> There's no reason they have to trust you and (say) Microsoft employees/Apple employees/Linux maintainers equally
...and that's why plenty of critical system around the world, including weapons, run on Linux and BSD, especially around countries that don't have the best relations with US.
Rogue companies/employees is really a different security problem that’s not directly comparable to drive-by patches (the closest comparison is a rogue open source maintainer).
And that's why nation-state attackers do it routinely.
Or you have a closed-source component you bought from someone who pinky-swears to be following secure coding practices and that their code is of course bug-free...
And they are correct. Unfortunately sometimes the number of eyes is not enough.
The alternative is closed source, which has prove to be orders of magnitude worse, on many occasions.
Eric Raymond claimed so, and a lot of people repeated his claim, but I don't think this is the same thing as "a lot of people claim" -- and even if a lot of people claim something that is obviously stupid, it doesn't make the thing less obviously stupid, it just means it's less obvious to some people for some reasons.
Raymond doesn't seem to claim anything like "there are sufficient eyes to swat all bugs in the kernel", or "there are eyes on all parts of the code", or "'bugs' covers all possible security flaws", or etc. He particularly mentions uptime and crashing, so less charitably the statement is "there are no crashing or corruption bugs so deep that a large enough quantity of volunteers can't bodge some way past them". Which leaves plenty of room for less used subsystems to have nobody touching them if they don't cause problems, patches that fix stability at the expense of security, absense of careful design in some areas, the amount of eyes needed being substantially larger than the amount of eyes involved or available, that maliciously submitted patches are different from traditional bugs, and more.
[1] https://en.wikipedia.org/wiki/Linus%27s_law
[2] http://www.unterstein.net/su/docs/CathBaz.pdf
As if there is some other software that is "military-grade" by the same measure? What definition are you using for that term, anyway?
...and yet FOSS and especially Linux is very widely used in military devices including weapons.
Because it's known to be less insecure than most alternatives.
What HN would call "ancient".
"We did not introduce or intend to introduce any bug or vulnerability in the Linux kernel. All the bug-introducing patches stayed only in the email exchanges, without being adopted or merged into any Linux branch, which was explicitly confirmed by maintainers. Therefore, the bug-introducing patches in the email did not even become a Git commit in any Linux branch. None of the Linux users would be affected."
They may not have been included in a release, but should gkh not have intervened *this would have reached users*, especially if the researchers weren't apparently aware their commits were reaching stable.
This message contradicted that.
It's super unclear.
"Next" goes approximately down the tree in the order it's displayed on the page, by depth-first search.
"Prev" just reverses the same process as "Next".
"Parent" differs from "prev" in that it goes to the parent e-mail even if this email has earlier siblings.
(Generally, I just scroll down to the tree view and click around manually.)
1) The email message, with a few headers included
2) A thread overview, with all emails in the thread
3) Instructions on how to reply
4) Information about how to access the list archives.
You need only care about (1) and (2). The difference between prev and parent is indicated by the tree view in (2). The previous one is the previous one in the tree, which might not necessarily be the parent if the parent has spawned earlier replies.
The current message in the tree is highlighted with the indicator "[this message]"; you can see replies branch out below it and parent messages above it.
The wasting time argument is nonsense too its not like they did this thousands of times and beside that, reviewing a intentional bad code is not wasting time is just as productive as reviewing "good" code and together with the patch-patch it should be even more valuable work. It not only or adds a patch it also make the reviewer better.
Yeah it aint fun if people trick you or point out you did not succeed in what you tried to do. But instead of playing the victim an play the unethical human experiment card maybe focus on improving.
Yes. Someone sees the work provided to the community for free and thinks that gives them some ethical privilege to put that work to the test?
Problem with that is it's a lot of work and they didn't want to do it in the first place.
Your suggested "wrongdoing by being lazy" is completely made-up nonsense.
Ridiculous. Does the same apply to pentesting a bank or a government agency. If you wanted to pentest these of course you'd get approval from an executive that has power to sanction this. Why would Linux development be an exception? Just ask GKH or someone to allow you to do this.
I'm shocked the researchers thought this wasn't textbook a violation of research ethics - we talk about the effects of the Tuskegee Study on the perception of the greater scientific community today.
This is a smaller transgression that hasn't resulted in deaths, but when it's not difficult to have researched ethically AND we now spend the time to educate on the importance of ethics, it's perhaps more frustrating.
The Linux kernel is a very large space with many maintainers. It would be possible to reach out to the leadership of the project to ask for approval without notifying maintainers and have the leadership announce "Hey, we're going to start allowing experiments on the contribution process, please let us know if you'd like to opt out", or at least work towards creating such a process to allow experiments on maintainers/commit approval process while also under the overall expectation that experiments may happen but that *they will be reverted before they reach stable trees*.
The way they did their work could impact more than just the maintainers and affect the reputation of the Linux project, and to me it's very hard to see how it couldn't have been done in a way that meets standards for ethical research.