It saddens me to see the linked article taken seriously and its political agenda repeated as if the existence of an article advocating Open Source in an Open Source conference gave credence to the idea of mandatory Open Source for life-critical software.
The article "Killed by Code: Software Transparency in Implantable Medical Devices" is weak. It mixes safety and security, which are subtly different beasts. Specifically, it justifies the idea of mandatory Open Source for safety-critical software with security examples. Unfortunately, while there are independent security researchers, I have yet to meet my first independent safety researcher, who looks for flaws in life-critical equipment for fame or money or ego…
The article also completely ignores the fact that DO-178B certified code, as found embedded in civil aircrafts, has a perfect safety track record (although some incidents have, no loss of human life has yet been attributed to software malfunction in a civil aircraft).
I work on static analysis tools for critical software. In this domain, the unavailability of examples is a big pain in the neck. My colleagues and I would love to see more examples of critical embedded code available to try our tools on. But in spite of this, I have to say that I find this article highly unscientific and so tainted by politics that it becomes distasteful.
(oh, and our analysis framework is Open Source. By choice, not by constraints based on unsound reasoning)
NOTE: I merely claim that the article is unscientific, not that it makes a particular unscientific claim such "that opening up software will solve all of it problems". But since apparently I need to be more specific, here is a paragraph straight from the article:
Other public sector agencies, such as the U.S. Navy, the Federal Aviation Administration, the U.S. Census Bureau and the U.S. Patent and Trademark Office have been identified as recognizing the security benefits of publicly auditable source code.20
There is only one reference for the four sources. Let's take a look at that reference:
How does that page justify the above paragraph? You tell me if you find it. This is not how references are supposed to work in a scientific article. The last time I saw this kind of "he said she said" reference, it was in a text trying to justify homeopathic claims.
"We at the Software Freedom Law Center (SFLC) propose an unexplored solution to the software liability issues that are increasingly pressing as the population of IMD-users grows--requiring medical device manufacturers to make IMD source-code publicly auditable."
I defy you to find the place in the article where it makes "unscientific" claims that opening up software will solve all of it problems. It mentions some problems and adds that opening the source code might be worthwhile.
You may call the requirement open up code an "agenda" but given that the public grants a equipment manufacturer a number of privileges (patents, protection from liability, etc), it seems in bounds to ask for a few things in return.
"I have yet to meet my first independent safety researcher, who looks for flaws in life-critical equipment for fame or money or ego…"
You have now, this is what I do for a living.
"The article also completely ignores the fact that DO-178B certified code, as found embedded in civil aircrafts, has a perfect safety track record"
DO-178B is a process, and not a "certification". It's utilization has not produced a "perfect track record" in safety critical systems.
You should know better than to make such grossly incorrect statements. Experts in this industry know for a fact that testing only proves the lack of a "tested for" failure condition, and cannot be relied upon to assure safety.
DO-178B is but one of several criteria by which the Aircraft certification authorities judge an avionics system to be safe.
Software is too complex to be assured by testing alone.
Static analysis is merely a tool, and is not a solution to this problem. You should be well aware of the limitations of Static analysis.
Regulators no longer are provided with access to design and testing data sufficient to fully review a product for safety, and they routinely rely heavily on the manufacturers, thru delegated authority, to regulate themselves.
Design defects can only be eliminated in the application of a transparent review process. Complexity issues in "limited human cognition" and "human bias" are the key causes of life threatening defects in safety critical systems.
Without external review and oversight by "outside consultants" such as myself, life threatening safety defects routinely go undetected.
All safety critical systems will ultimately be "public domain" for this very reason. Just as proposed by the
Software Freedom Law Center.
The FAA does not absolve aircraft manufacturers for liability, as the case with the FDA and Medtronic. (see Varig v. United States) 28 USC 2680(a)
Discretionary function exception
DO-178B software does not "in fact" have a "perfect safety track record".
Turkish Airlines Flight 1951 was a passenger flight which crashed during landing to Amsterdam Schiphol Airport, Netherlands, on 25 February 2009, killing nine passengers and crew including all three pilots.
software defect utilized "invalid altitude data" as valid.
"Tests showed that the Rockwell Collins Enhanced Digital Flight Control System (EDFCS) uses radio altitude values that are characterised as ‘non computed’ (unusable,) whereas this characterisation should have prevented this."
Perhaps there is a misunderstanding. I was speaking of security researchers who look for flaws without having been prompted to do so (the "money" in "fame, money and ego" was the money that you can get when running a well-organized botnet, not the money you get for a honest job well done. Sorry for the confusion). Is this what you do, in the case of safety? If so, this is interesting and I would like to know more. What do you do with your results, publish them? What is your affiliation? Where can I find a list of such publications? Or do you have a contract with someone who is paying you to assess the code under NDA, in which case, where does Open Source come into this model? Would you do your job for free if the code was open? Under a deadline?
> DO-178B is a process, and not a "certification"
I used the words "DO-178B certified". As a Google search will confirm, I am not the first one to use this idiom. Since respectable actors are using it and have gone as far as buying it as Google AdWords, I stand by my use of these words.
> Static analysis is merely a tool, and is not a solution to this problem. You should be well aware of the limitations of Static analysis.
I didn't make any claims one way or the other. It is not unusual to meet people who have strong ideas on the subject, and I avoid this sterile discussion whenever I can.
What always strikes me, when I read reports such as this, is the self-correcting attitude displayed by the relevant authorities. The question, when analyzing an incident, is always, "how do we make sure the same mistake is not made again?". The reason why I think public domain critical code will not happen is that there is no answer to this question
(you can't make it more public domain the next time).
Again, I wish it was different. I just don't think it will happen in the foreseeable future.
> All safety critical systems will ultimately be "public domain" for this very reason.
Do you want to bet this practice has not been adopted in a single domain where it could apply by 2020?
It seems to me, but forgive me if I misunderstood your arguments, that you are saying "Testing is not perfect. Static analysis is not perfect. Therefore, critical code has to be made public domain". I'm not sure it follows. Whatever solution you are rooting for does not necessarily win by default because the other solutions are not perfect. Open source could win if it had something tangible to bring to the table. It wouldn't be automatic, and I am not even sure that the tangible advantages exist. Plane designers have been making software more and more complex because they could. If it turns out they can't complexify it any more, they can also just stop complexifying it. I can assure you that a company such as Boeing or Airbus would rather scale back on the new features used to promote new models than expose their designs to their competitors three years in advance (the code has to be verified before the first flight, right?).
At the risk of repeating myself, I am only describing the situation as I see it, not the way it would be convenient to me for it to be.
4 comments
[ 3.1 ms ] story [ 17.0 ms ] threadThe article "Killed by Code: Software Transparency in Implantable Medical Devices" is weak. It mixes safety and security, which are subtly different beasts. Specifically, it justifies the idea of mandatory Open Source for safety-critical software with security examples. Unfortunately, while there are independent security researchers, I have yet to meet my first independent safety researcher, who looks for flaws in life-critical equipment for fame or money or ego…
The article also completely ignores the fact that DO-178B certified code, as found embedded in civil aircrafts, has a perfect safety track record (although some incidents have, no loss of human life has yet been attributed to software malfunction in a civil aircraft).
I work on static analysis tools for critical software. In this domain, the unavailability of examples is a big pain in the neck. My colleagues and I would love to see more examples of critical embedded code available to try our tools on. But in spite of this, I have to say that I find this article highly unscientific and so tainted by politics that it becomes distasteful.
(oh, and our analysis framework is Open Source. By choice, not by constraints based on unsound reasoning)
NOTE: I merely claim that the article is unscientific, not that it makes a particular unscientific claim such "that opening up software will solve all of it problems". But since apparently I need to be more specific, here is a paragraph straight from the article:
Other public sector agencies, such as the U.S. Navy, the Federal Aviation Administration, the U.S. Census Bureau and the U.S. Patent and Trademark Office have been identified as recognizing the security benefits of publicly auditable source code.20
There is only one reference for the four sources. Let's take a look at that reference:
20 FAQs, Open Source for America, http://opensourceforamerica.org/faq (last visited July 16, 2010).
How does that page justify the above paragraph? You tell me if you find it. This is not how references are supposed to work in a scientific article. The last time I saw this kind of "he said she said" reference, it was in a text trying to justify homeopathic claims.
"We at the Software Freedom Law Center (SFLC) propose an unexplored solution to the software liability issues that are increasingly pressing as the population of IMD-users grows--requiring medical device manufacturers to make IMD source-code publicly auditable."
I defy you to find the place in the article where it makes "unscientific" claims that opening up software will solve all of it problems. It mentions some problems and adds that opening the source code might be worthwhile.
You may call the requirement open up code an "agenda" but given that the public grants a equipment manufacturer a number of privileges (patents, protection from liability, etc), it seems in bounds to ask for a few things in return.
You have now, this is what I do for a living.
"The article also completely ignores the fact that DO-178B certified code, as found embedded in civil aircrafts, has a perfect safety track record"
DO-178B is a process, and not a "certification". It's utilization has not produced a "perfect track record" in safety critical systems.
You should know better than to make such grossly incorrect statements. Experts in this industry know for a fact that testing only proves the lack of a "tested for" failure condition, and cannot be relied upon to assure safety.
DO-178B is but one of several criteria by which the Aircraft certification authorities judge an avionics system to be safe.
Software is too complex to be assured by testing alone.
Static analysis is merely a tool, and is not a solution to this problem. You should be well aware of the limitations of Static analysis.
Regulators no longer are provided with access to design and testing data sufficient to fully review a product for safety, and they routinely rely heavily on the manufacturers, thru delegated authority, to regulate themselves.
Design defects can only be eliminated in the application of a transparent review process. Complexity issues in "limited human cognition" and "human bias" are the key causes of life threatening defects in safety critical systems.
Without external review and oversight by "outside consultants" such as myself, life threatening safety defects routinely go undetected.
All safety critical systems will ultimately be "public domain" for this very reason. Just as proposed by the Software Freedom Law Center.
The FAA does not absolve aircraft manufacturers for liability, as the case with the FDA and Medtronic. (see Varig v. United States) 28 USC 2680(a) Discretionary function exception
DO-178B software does not "in fact" have a "perfect safety track record".
Turkish Airlines Flight 1951 was a passenger flight which crashed during landing to Amsterdam Schiphol Airport, Netherlands, on 25 February 2009, killing nine passengers and crew including all three pilots.
software defect utilized "invalid altitude data" as valid.
"Tests showed that the Rockwell Collins Enhanced Digital Flight Control System (EDFCS) uses radio altitude values that are characterised as ‘non computed’ (unusable,) whereas this characterisation should have prevented this."
http://en.wikipedia.org/wiki/Qantas_Flight_72
ADIRU software defect utilized "invalid data" as valid.
This device was certified to DO178-B Level A, the very highest level of certification, where failure can result in catastrophic loss of aircraft.
1 August 2005, Boeing 777-200 registered 9M-MRG, from Perth to Kuala Lumpur, Malaysia.
use of invalid data as valid
This device was certified to DO178-B Level A, the very highest level of certification, where failure can result in catastrophic loss of aircraft.
Perhaps there is a misunderstanding. I was speaking of security researchers who look for flaws without having been prompted to do so (the "money" in "fame, money and ego" was the money that you can get when running a well-organized botnet, not the money you get for a honest job well done. Sorry for the confusion). Is this what you do, in the case of safety? If so, this is interesting and I would like to know more. What do you do with your results, publish them? What is your affiliation? Where can I find a list of such publications? Or do you have a contract with someone who is paying you to assess the code under NDA, in which case, where does Open Source come into this model? Would you do your job for free if the code was open? Under a deadline?
> DO-178B is a process, and not a "certification"
I used the words "DO-178B certified". As a Google search will confirm, I am not the first one to use this idiom. Since respectable actors are using it and have gone as far as buying it as Google AdWords, I stand by my use of these words.
> Static analysis is merely a tool, and is not a solution to this problem. You should be well aware of the limitations of Static analysis.
I didn't make any claims one way or the other. It is not unusual to meet people who have strong ideas on the subject, and I avoid this sterile discussion whenever I can.
I made clear in which sense I meant "perfect safety record". There were no casualties in Qantas' Flight 72. The report at http://www.onderzoeksraad.nl/docs/rapporten/Rapport_TA_ENG_w... is new to me. Thanks for the reference.
What always strikes me, when I read reports such as this, is the self-correcting attitude displayed by the relevant authorities. The question, when analyzing an incident, is always, "how do we make sure the same mistake is not made again?". The reason why I think public domain critical code will not happen is that there is no answer to this question (you can't make it more public domain the next time).
Again, I wish it was different. I just don't think it will happen in the foreseeable future.
> All safety critical systems will ultimately be "public domain" for this very reason.
Do you want to bet this practice has not been adopted in a single domain where it could apply by 2020?
It seems to me, but forgive me if I misunderstood your arguments, that you are saying "Testing is not perfect. Static analysis is not perfect. Therefore, critical code has to be made public domain". I'm not sure it follows. Whatever solution you are rooting for does not necessarily win by default because the other solutions are not perfect. Open source could win if it had something tangible to bring to the table. It wouldn't be automatic, and I am not even sure that the tangible advantages exist. Plane designers have been making software more and more complex because they could. If it turns out they can't complexify it any more, they can also just stop complexifying it. I can assure you that a company such as Boeing or Airbus would rather scale back on the new features used to promote new models than expose their designs to their competitors three years in advance (the code has to be verified before the first flight, right?).
At the risk of repeating myself, I am only describing the situation as I see it, not the way it would be convenient to me for it to be.
Pascal