I worked at a medical imaging company in Toronto in the 80's and 90's. Therac-25 was very much top of mind for all companies in our industry and changed everything from how we did Design, Development and QA to how the FDA reviewed and approved devices running our software.
It was one of the standard examples in my introductory computer science classes, along with the Ariane overflow and the inaccurate system clock on some missile defence system that got a bunch of soldiers killed in Iraq when someone missed the scheduled restart hack.
> The Patriot missile battery at Dhahran had been in operation for 100 hours, by which time the system's internal clock had drifted by one-third of a second. Due to the missile's speed this was equivalent to a miss distance of 600 meters.
I went to an engineering school in Canada for Electrical/Computer Engineering and this was an example of engineering failure we studied in a course about software quality assurance and engineering ethics. Not sure if its widely used as a case study outside of Canada, but it really stuck with me. Especially when designing critical systems.
There are two types of reactions to the Therac-25 story.
1) We need to put a lot of process in place to review and test and ensure we have proper interlocks in place.
2) They just didn't have enough talent on the team. It couldn't happen here.
You want some people with reaction 1 working on a safety critical system.
In my field, we use this kind of example to tell people that testing sucks or is not enough and that we need to formally/mathematically prove that software is free of bugs for critical systems.
If you think like #3 you're one of the engineers that should work on safety critical software. Recognizing that you may produce errors I think is akin to the Dunning-Kruger effect. It shows me you know enough that you don't know enough, which is far preferable to people who don't know they don't know and thus think they're good.
This is why Quality is such an awesome field. It's all about how a group of smart, skilled, but fundamentally fallible people can build a product with an arbitrarily high level of safety. If it's done properly, no one individual should be able to create a bug that can cause a failure like this.
4) I don't work on anything safety critical, I just work on (millions of dollars per second | stores of personal information) systems. I don't need to worry.
For many systems, the lack of you working on it might hurt someone.
For example, imagine you work in a bank on their mobile apps. You decide that implementing automatic bill payments for a certain type of account is too much technical effort and not worth it.
Hundreds of thousands of people with that account now have a subpar experience. They have to manually pay bills, and many will forget, leading to arrears, bankruptcy, payday loans, and prison.
Your decision not to implement that feature to make the lives of your customers better indirectly led to ruining the lives of tens or hundreds of people, who otherwise would have led a good life.
From an ethical point of view, unknowingly ruining a life due to an action or due to inaction, is pretty similar.
> The truth is that any software program will probably contain one error for every 500 lines of code. The Therac-25's software program, relatively crude by today's standards, probably contained 101000 lines of code. At one error for every 500 lines, that works out to the possibility of twenty errors.
Look at the rest of the document. It is almost clear from the errors you can find throughout the document that this "report" was posted from some original hard copy run through some form of OCR.
I think this is a particular example of it. Why the number of LOC is 101000?
Maybe the OCR interpreted a bad comma from scanning as a "1", in which case the number would have read 10,000 LOC - and thus "twenty errors".
As another glaring example of this OCR issue, look for this line:
> "There were [.4ECL] people sitting in our offices telling us it [the Therac-25] couldn't hurt anybody when they knew it could."
"[.4ECL]"?
It should read "[AECL]" - but the scan or the copy it was scanned from probably had a small break on the arm of the "A" - which the OCR interpreted as ".4"
You can find other small anomalies like this elsewhere in the document if you look for them. They kinda bugged me, but I could tell from the pixels what was going on.
The author, Barbara Rose Wade, has a website (a copy of story with the same artifacts is there), and a twitter account. Might ask if the has a copy or scan on the original. (I'm not on Twitter.)
I remember hearing about this case in college. My professor's point was don't trust software alone. Always include hardware safe guards when it comes to safety.
Since I started really playing with hardware/software integration (hobby level) a couple of decades ago, and also knowing what I learned from various efforts I've been involved in at employers to "bandaid patch" in security (ie, logins, passwords, access control, etc) - and how it never works out, I've since advocated for the implementation of security controls first, whether implementing a hardware and/or software solution, and one of those controls, if hardware is involved, should be a physical "all-stop" cut-the-power control, manifestly separate from any "software" (whether implemented in hardware logic or otherwise).
Design the security/safety system up front, for the task of the system being designed, for all potential failure modes - and then implement the first, and test the hell out of it, and document it thoroughly.
Unfortunately, this tends to get shot down, until it is needed, and then it gets bandaid-patched in place, because usually at the point it is needed, a whole refactor of the code base is usually called for, and we can't have that, so "just fix it" becomes the call of duty.
And it never works out properly. Things get missed, security is effectively out the window, until something happens, with the refrain of "why did this occur" and of course nobody wants to take the blame for "I told you so".
Unfortunately, even implementing such a security system first will only get you so far, and it is still possible that something might be missed, and potentially be the cause of a safety critical issue. But hopefully, most if not all of those extreme events can be caught first in the system, by some part of it, and what's left will be relatively minor.
When I was doing industrial controls in the 80's, I read a paper on the topic. The most important point was safety is a primary design goal. You can't add it in later. Because security and safety concerns have pervasive effects on design decisions.
> Always include hardware safe guards when it comes to safety.
What does that actually mean? Maybe in the Therac-25 case the maximum dosage could somehow have been limited by a hardware dial in addition to software. But what form would hardware safeguards take for the millions of decisions software makes every second on a starting rocket? And considering hardware has far higher failure rates than no-moving-parts software, wouldn't any system that lets hardware dominate software decrease safety?
I don't know exactly but the previous generation model had the hardware interlock that would have prevented this problem so it is possible.
From Wikipedia:
"One, when the operator incorrectly selected X-ray mode before quickly changing to electron mode, which allowed the electron beam to be set for X-ray mode without the X-ray target being in place. A second fault allowed the electron beam to activate during field-light mode, during which no beam scanner was active or target was in place.
Previous models had hardware interlocks to prevent such faults, but the Therac-25 had removed them, depending instead on software checks for safety."
Perhaps just a dose integrator with an alarm limit? There's probably a total dose above which you could likely say "this is a mistake".
A dose integrator would be a detector + an integrator independent of the therapeutic delivery software. If it goes over limit, it shuts the system down.
Hmm - that's not good. That said, once I read the article title, my first thought was "Therac-25"? A quick skim before I read it confirmed that suspicion.
...and I have to add that I'm one of those "informally educated software engineers" - though I've never touched or worked on any kind of safety-critical systems (and I would decline such a position if offered, to be honest). I've been doing SWE for over 25 years now, and the only "formal schooling" I've had in the field was a couple of community college classes I took to learn C/C++ 2+ decades ago. Everything else has either been learned thru my employment, or at home. Prior to my first "professional" position in 1992, all I had was a high school degree and an associates from a trade school in "computer electronics".
But even I know about the Therac-25, and it's always in the back of my head.
The fairly recent change in the industry surrounding automated testing, test case programming, CI, etc - all of that which more or less fell out of the whole "agile" movement - has led to a vast improvement in software, imho. Yet bugs still persist, despite all of that and more.
We have to do better - I don't know how, or if it's even possible (from what I understand, it's not mathematically possible, except in maybe certain trivial examples, to validate all possible states of code - maybe I'm mistaken on that).
It's not about clickbait in this case. It's more like not giving away the answer to a puzzle. The puzzle itself is a good thing. It jigs us out of the mode in which we expect everything to be instantly explained. I can't prove it, but I believe that when operating in that mode, we are more likely to have predictable responses.
Also, there's a historical aspect to this. We don't need, and needn't presume, to rewrite the titles of classic articles.
I'm a firm believer in providing adequate and sufficient contex. Difficult enough given HN's 80 character subject field -- I've wordsmithed numerous submissions to fit. I've also stuck with numerous poor original titles with gritted teeth knowing HN's policies, sometimes addressing the ambiguity in a clarifying comment. And I've noted clickbait innumerous emails to HN, some of which you agree with, some not.
And yes, this is art not science: aimed at effect.
The Therac case study has acquired a recognition the contemporaneous article (and title) wouldn't have experienced -- the company name was then the more notable signifier. AECL has been far eclipsed by its most notorious product. Today, "Therac-25" should lead, on the same basis as "AECL" did in the original.
I'm generally a follower of Jacob Nielsen on microcontent:
Well-written, short text fragments presented out of supporting context can provide valuable information and nudge web users toward a desired action.
Mrtimer J. Adler does not fully condemn content concealment in book tables of contents, but his lips are clearly pursed, and nose wrinkled:
It used to be a common practice, especially in expository works ... to write very full tables of contents, with the chapters or parts broken down into many subtitles indicative of the topics covered. Milton, for example, wrote more or less lengthy headings, or "Arguments," as he called them, for each book of Paradise Lost. Gibbon published his Decline and Fall of the Roman Empire with an extensive analytical table of contents for each chapter. Such summaries are no longer common.... [P]ublishers have come to feel that a less revealing table of contents is more seductive than a completely frank and open one. Readers, they feel, will be attracted to a book with more or less mysterious chapter titles-they will want to read the book to find out what the chapters are about. Even so, a table of contents can be valuable.
-- How to Read a Book, p. 33.
The same argument applies to article titles at HN.
You've made the case (and commented on comunity failings) for defusing titles on hot-button subjects. That's valid.
This isn't that circumstance.
Pandering and information concealment as a deliberate policy, again not the case for the article as initially written, in the cotext of its time, is overtly manipulative, to no real gain.
> The hospital staff, physicist Fritz Hager, and his technician, who had worked the machine in both accidents, stayed at the console long after everybody else had gone home for the weekend, typing and retyping the prescription into the computer console, determined to re-create Malfunction 54.
Thank God for the curious minds of this world, the people who have he patience and dedication to reproduce bugs.
Yes, it seems that this hospital staff were the testers that AECL should have had themselves. A great tester is curious, imaginative, dedicated, patient and worth a good salary.
I wonder about the increasing use of x-rays in non-medical applications. Industrial applications, or even with the public- like the backscatter machines that TSA and police forces use. Therac-25 seems to be the only story we have where something went wrong - but what are the chances a bad thing could happen, or already happened, in another machine?
TSA employees are not permitted to wear dosimeters[1], which makes it hard to detect these things. Scientific American, the most established magazine in the US, is among respected bodies that have recommended dosimeters for accountability reasons.[2]
There are also extremely infrequent (and sometimes rigged) studies done on the equipment, and they often find dangerous conditions. For example:
>Doses for some of the baggage screeners exceeded the maximum dose for the public.... Some EDS machines were not well maintained (i.e., they had bent curtain rods and missing curtain flaps).... Most EDS machines emitted low levels of radiation; a few exceeded regulatory limits.[2]
>> but what are the chances this could happen, or already happened, in another place?
Zero. The TSA machines are not limited by software. They will catch fire themselves before killing people. Therac-25 was designed to destroy tissues, cancer tissues but human tissue nevertheless. It is like comparing an m-16 to a rubber band gun. They are both guns, but they are fundamentally different in terms of capabilities with the former.
All xray backscatter imaging devices were removed by the TSA and replaced with millimeter backscatter. The downside is those xray backscatter devices went to prisons for use.
>All xray backscatter imaging devices were removed by the TSA
At the airports. They still exist in prisons, in covert trucks and other mobile units owned by TSA and law enforcement, and potentially in other venues of public transportation, stadiums and arenas, etc.
X-rays are also in INCREASING use at airports, since they are in the process now of replacing the carry-on bag scanners with CT scanners. A CT scan gives a 3D image of your luggage which means you don't have to take out your laptop. Unfortunately, a CT scan also uses 10X more radiation than a regular x-ray.
1. Xray scanning should be outlawed for public safety use in humans. You want to scan humans? You must use millimeter wave, and the cost is on you.
2. Governance. It's perfectly acceptable to use CT scanners for non-human surveillance (carry on bags, checked luggage, etc), but you must ensure life safety is enforced. If you don't, steep penalties up to and including jail time.
Call your reps, run for office, vote for people who care about this.
Up here in Canada I'm happy to walk through a metal detector as part of the security screening... and then board on my plane to a destination I hope doesn't include the US to avoid the hassle of American security theater.
When I was still in VT I was happy to see that our third tier airport managed to hang on the noticeably more effective metal detectors long after other areas phased them out, sadly they've now been swapped out.
TSA Officers aren't permitted to wear dosimeters on the job.
PRetty damning, if you ask me.
Here's their fationale from teh FAQ in the linked article [1]
"Q: Why aren’t your officers permitted to wear dosimeters?
A: There is a really good reason for this. The emissions from our X-ray technology are well below the requirements that would require their routine usage. To help reassure passengers and employees that the technology is safe, however,health physicists with the U.S. Army have been conducting area dosimeter surveys at multiple airports nationwide."
This is at best massively flawed reasoning by the TSA, and at worst a deliberate misleading rationale.
1) While the NORMAL dosage is of course well below any level that would make a dosimeter useful, a dose from a malfunction or bent/broken/missing shield panel could easily exceed dosimeter detection thresholds.
2) Spot checks are nice, but unless they are very frequent, they could easily miss a malfunction and leave it overdosing agents and people for a long time. Even if inspections are every 6 months at every airport station, that will leave a malfunction overdosing people for on average, three months.
3) Detecting, identifying, and repairing a malfunctioning system could be done much sooner and at lower cost by using dosimeters, either on employee volunteers or just hanging them on the chairs & workstation surrounds.
It is this kind of frankly stupid stick-your-head-in-the-sand reasoning that breeds distrust of governments and the companies that supply them.
I'll give them the benefit of the doubt and say this is poorly worded. TSA officers also aren't permitted to hygrometers on the job. Why? Because there's no point to wearing a hygrometer as a TSA officer. Unlike with the dosimeter, that doesn't need to be spelled out.
That doesn't mean they aren't allowed to perform dosimeter readings at all, it just means it's not acceptable to carry this equipment around all the time.
> While the NORMAL dosage is of course well below any level that would make a dosimeter useful, a dose from a malfunction or bent/broken/missing shield panel could easily exceed dosimeter detection thresholds.
Do you actually know that these could be harmful, or is it just a suspicion?
TSA officers may not be offered hygrometers, but I doubt if there is anything explicitly preventing them from carrying their own hygrometer - actively preventing officers from carrying dosimeters seems more likely to be tied to prevent any long term lawsuits. We get bombarded by all sorts of radiation throughout the day, describing a malignant skin mole to a back-scatter vs. the sun isn't something you could do without evidence. The collection of such evidence is specifically being prevented.
So I'd assume this is more like coal miner operators in the 1800s planning ahead to protect themselves from the eventual outbreak of black lung.
I won't give them the benefit of the doubt on poor wording (they have plenty of time and resources to get it right).
As far as we know, there is no general ban on wearing instrumentation of any other on the job, unless it is somehow clumsy and would interfere with their duties or the appearance of their uniform.
In contrast, the simplest dosimeter is typically a very small patch of photographic paper in an envelope opaque to visual light but transparent to harder radiation (towards the gamma end of the spectrum). It is like an inch square, or like a typical nametag, easily worn in a shirt pocket, no more visible to anyone else than a credit card in the shirt pocket.
The only plausible reason to forbid this is to prevent any kind of whistleblower event.
Of course I do not know the actual levels of radiation inside each machine or all of their possible failure modes. However, we do know
1) high-energy radiation dosage is cumlulative -- each day's dose adds up, so if you have 1/100th of a critical dose each day in a workzone, in less than four months, you should be banned from that work zone. This is the way it works in many facilities dealing with radiation sources, and everyone wears dosimeters which are checked, and people are reassigned when the total dosage exceeds the prescribed level.
2) There are medical limits on how many x-ray and CT-Scan procedures are allowed for patients, again due to the cumulative nature of radiation exposure. Even with the very low-doses of modern machines, these are a concern and are tracked (I know this from the experiences of several close family members).
3) The TSA machines do produce sufficient radiation in their beam to fog ASA800 and higher film. This is definitely a non-trivial dose. Tho obviously not lethal in a single dose (e.g., if you rode through the machine), it is significant.
If an interior shielding panel is bent or somehow mis-installed during manufacturing, maintenance, or repair, it is entirely plausible that people working regularly in the area of the leaking radiation could be overexposed.
So, no, this is not even close to mere suspicion or Luddite "ooh radiation bad" -- and yes, we do know that these could be harmful, especially over extended periods.
I hope you are not an employer with that attitude.
> As far as we know, there is no general ban on wearing instrumentation of any other on the job, unless it is somehow clumsy and would interfere with their duties or the appearance of their uniform.
Who is "we"? Aren't you just pleading ignorance here? What do really you know about what TSA agents can and can't wear? Anything can "interfere" with the appearance of a uniform, including a "small photographic patch".
> The only plausible reason to forbid this is to prevent any kind of whistleblower event.
Then why do they do allow third party monitoring?
Having further looked into this, the page you posted is outdated and they ended up handing out dosimeters themselves:
The result was negative. Of course you can't prove a negative, maybe there's radiation somewhere else where they didn't look...
> 2) There are medical limits on how many x-ray and CT-Scan procedures are allowed for patients, again due to the cumulative nature of radiation exposure.
> 3) The TSA machines do produce sufficient radiation in their beam to fog ASA800 and higher film. This is definitely a non-trivial dose.
" TSA claims the “backscatter machines” cannot produce more than 0.005 millirem of radiation per scan. In comparison, the agency says, a chest x-ray exposes patients to 10 millirem of radiation, and the maximum recommended exposure to radiation from man-made sources is 100 millirem per year."
That would mean that if you had a machine that was completely unshielded, you would have to perform 2000 scans to get the equivalent of a chest X-Ray. Compare that to the radiation exposure of your average flight attendant.
>> Anything can "interfere" with the appearance of a uniform, including a "small photographic patch".
Not if it is inside your pocket. Get real.
>> Then why do they do allow third party monitoring?
Obviously because they can control when it is done as well as how the results are distributed, or not distributed.
>> " TSA claims the “backscatter machines” cannot produce more than 0.005 millirem of radiation per scan. In comparison, the agency says, a chest x-ray exposes patients to 10 millirem of radiation, and the maximum recommended exposure to radiation from man-made sources is 100 millirem per year."
>> That would mean that if you had a machine that was completely unshielded, you would have to perform 2000 scans to get the equivalent of a chest X-Ray
1) this assumes that the "0.005 millirem per scan" is the total output of the machine, including all radiation absorbed by the lensing & shielding. We do not know if this figure is the total generation output, or only the total amount put to the target & receiving sensors.
2) one thing we do know is that the radiation will NOT be scattered evenly; there will be lobes and hotspots and cold spots.
3) even if we assume .005millirem/scan, at the rate of two/minute, that's 1000 minutes to get a chest x-ray, or 16 hours, basically two work days. In LESS THAN ONE MONTH, they are over the yearly dose.
More importantly, this is only counting the backscatter machines that are specifically designed for super-low doses as they are specifically irradating people.
This does NOT look at the luggage scanners, either for carry-ons or for checked luggage, which likely have much higher outputs.
Finally, even if your arguments are correct and there is truly no risk, there should be no problem at all for officers to wear their own dosimeters -- the results would merely be putting their minds at ease that they are not getting exposed.
Thus, we are back to the only reason to prevent officers from wearing dosimeters is an attempt to control the information and prevent whistleblowers and/or lawsuits.
Again, I sincerely hope you are not an employer with this attitude.
> Obviously because they can control when it is done as well as how the results are distributed, or not distributed.
Let's apply some Occam's Razor here.
I am led to believe they know that these machines are harmful and therefore they are carefully orchestrating all testing of these devices by, among others, all of the following parties: "U.S. Army Public Health Command, the Food and Drug Administration’s (FDA) Center for Devices and Radiological Health (CDRH), the National Institute of Standards and Technology (NIST), and the Johns Hopkins University Applied Physics Laboratory (APL)".
That's all done to prevent some sort of mass lawsuit, which might cost the United States some money.
The simpler explanation is that TSA rules are simply really strict, because that's just how TSA rules are.
The rules don't have to make sense. There's a whole lot about the TSA and its procedures that doesn't make sense.
> We do not know if this figure is the total generation output, or only the total amount put to the target & receiving sensors.
I'm content with the description given that these machines "cannot produce" more than that. You can do your own research on the matter.
> 3) even if we assume .005millirem/scan, at the rate of two/minute, that's 1000 minutes to get a chest x-ray, or 16 hours, basically two work days. In LESS THAN ONE MONTH, they are over the yearly dose.
...assuming no shielding whatsoever. You would hope that after two months without shielding on the device, someone might notice. Even then, such exposure would be well below the occupational limit for radiation workers (5000 millirem):
>>I am led to believe they know that these machines are harmful and therefore they are carefully orchestrating...
That's not even close to the simplest explanation. Far simpler is that they are careless, clueless, and paranoid, and don't want anyone finding out something bad on their watch. Or they just don't want to allow anyone to get concerned.
>> The rules don't have to make sense.
Unless we're living in a Kafka or Vonnegut book, they are supposed to have actual reasons and make sense.
Calling out one particular type of measuring device, which happens to measure the one plausible type of hazard in that workplace, yeah, that's suspicious.
>> .assuming no shielding whatsoever.
Does not assume no shielding whatsoever, only a plausible hotspot or focus. The point is that the potential dosing is in the range of values that could cause a hazard, and even at 10% of that value, would be overdosing within a year. It is not like it is totally off the scale (as are many of the public's fears).
>> You would hope that after two months without shielding on the device, someone might notice.
Yes, you would hope that. However, if you've ever read reports on industrial issues, you'll find that far worse and more noticeable hazards go unnoticed/unfixed for YEARS at a time before they get around to killing someone. And this sort of low-level radiation is particularly pernicious, as it is completely unnoticeable until after it is too late.
The simple solution is, assuming that these machines are largely safe (which I generally expect), is to say "you can buy dosimeters on your own dime and wear them if you like, just keep them out of sight in your pocket while on duty. If anyone gets any unexpectedly high readings, we'll investigate, but remember since you haven't got a chain of custody around the meter, it'll be only the beginning of an investigation, not immediate acceptance of your results."
Ye, it may turn out to be bit more work, but it shows you're on the side of worker safety, and on the road to getting things fixed. Mostly, it'll quell any fear and likely cause no one to wear one.
With the rules as it is, I'd be damn sure to sneak dosimeters into my clothes while working there.
the staff at Princess Margaret Hospital in Toronto had decided to... measure all doses of radiation in the beam and, in a fraction of a second, stop excessive doses before they could reach the patient
Astonishing this basic safety precaution was not already in use. Do modern devices do this?
> She verified everything else and turned on the beam. The machine stopped and the computer screen flashed "Malfunction 54," a mysterious message not even mentioned in the Therac-25 manual.
> [...] the Therac-25 typically issued up to four error messages a day. It did so by displaying "Malfunction" plus a number, from 1 through 64. No explanation was offered by the computer nor was there any reference to the malfunction codes in the operator's manual. Technicians could, in most cases, bypass the irritating malfunctions simply by pressing the "p" key, for "proceed." Doing so became a matter of habit.
Even if you're not doing safety-critical systems, there are many lessons to be learned just from this bit alone.
Use human-readable error messages, and make sure they are understandable to the users/operators of your software (not just developers). Including a specific error code is fine -- it can disambiguate similar errors, and make it very easy to do a 'find all' in source to go directly to the offending spot -- but that doesn't help your user.
If you're going to allow users to bypass an error, make it deliberate. The best way to do this is debatable, but through years of bad software, users have been trained to dismiss popup messages by pressing OK or 'yes' or 'proceed'.
When you show an important error, make sure it's different from informational messages or warnings. Showing users routine warnings and messages is fatiguing and basically just trains them to always press the 'p' key.
> According to a computer system's analysis of FDA documents, the computer would not accept new information on a particular phase of treatment (in the case of both Tyler accidents, changing the x-ray mode to electron mode) if the technician made the changes within eight seconds after reaching the end of the prescription data. That's what Malfunction 54 meant.
Consider if they had actually tried to show an error message for this (or even document what it meant). Something like "Data on screen does not reflect actual parameters being used. Proceed?" Hopefully at some point during development, someone would question that: "why are we allowing them to proceed in this case?" "If we can detect that is the case, why even show this as an error, instead of just updating the parameters?"
"why even show this as an error?" is one of my favorite questions. It's the UI equivalent of "make illegal states unrepresentable".
> Use human-readable error messages, and make sure they are understandable to the users/operators of your software (not just developers). Including a specific error code is fine -- it can disambiguate similar errors, and make it very easy to do a 'find all' in source to go directly to the offending spot -- but that doesn't help your user.
Yes. Remember, however, that much software needs to be localized, even if the initial developer didn't think to. Error codes are much better than hard-coded English-language error messages; map them to a message at the last possible moment.
A cobalt machine killed a patient just a few days ago, in Voronezh, Russia. The patient was crushed between the movable treatment table and the collimator. As far as I can tell from the limited information available on the web, the problem was not software but rather a stuck manual switch.
That's kinda like being killed by jacked up car crushing person working on the car underneath. Nothing to do with radiation or software. With a static device where operator does not position the jack, the cause would be lack of mechanical maintenance or material failure (unlikely).
The failure mode of the switch contributed to this accident, and a switch which does not exhibit this failure mode should have been used if possible. If not possible, then the switch should have been moved through a test actuation before reaching a position where the failure mode causes the patient to be killed. This test actuation should be performed each time the table is moved. Finally, the pathway that the switch passes through should be periodically cleared of debris and the switch tested on a regular basis. If the test actuation fails(indicating switch failure) then the table should reset to a safe position.
As an aside, every time I step on the treadmill at the gym I check that the emergency stop button works and that the incline down and speed down buttons work. I have been stuck on a treadmill which I couldn't stop or slow down before.
All mechanical devices will fail eventually. While a test actuation might reduce the risk somewhat - its not going to be a sure thing. Remember the table didn't kill the last patient that stepped on it - why would anyone suspect the switch would fail now.
Mechanical failures need to be dealt with differently than software. Parts must be replaced before they fail. Even if they appear to work correctly. Software doesn't wear out so its a whole different ball game.
Sounds like there is an obvious design failure - similar to garage or elevator doors the table shouldn't move and push up beyond some force threshold (patient weight plus say 200N of force).
This article/site keeps coming back to HN often and I cannot stop reading it. It reads like a novel - very sad but very good writing. I hope one day we will see the movie about the issue described.
I think this story is not as much a story about a software bug as it is a story about off hand accident management. If AECL had earlier been more humble and determined to find the errors maybe they would have? It seems jaw dropingly ignorantly handled. Why did it have to take so much time and several people dead before serious effort was put in to find the issues?
80 comments
[ 5.3 ms ] story [ 116 ms ] threadhttps://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha...
> The Patriot missile battery at Dhahran had been in operation for 100 hours, by which time the system's internal clock had drifted by one-third of a second. Due to the missile's speed this was equivalent to a miss distance of 600 meters.
There are two types of reactions to the Therac-25 story.
1) We need to put a lot of process in place to review and test and ensure we have proper interlocks in place.
2) They just didn't have enough talent on the team. It couldn't happen here.
You want some people with reaction 1 working on a safety critical system.
Ethics and responsible software development was a good unit.
3) I'd never work on life critical medical (or anything else) software, I know I fuck up and even the best can (and I'm not).
I simply wouldn't want it on my conscience that an easily preventable error (in hindsight) hurt someone.
For example, imagine you work in a bank on their mobile apps. You decide that implementing automatic bill payments for a certain type of account is too much technical effort and not worth it.
Hundreds of thousands of people with that account now have a subpar experience. They have to manually pay bills, and many will forget, leading to arrears, bankruptcy, payday loans, and prison.
Your decision not to implement that feature to make the lives of your customers better indirectly led to ruining the lives of tens or hundreds of people, who otherwise would have led a good life.
From an ethical point of view, unknowingly ruining a life due to an action or due to inaction, is pretty similar.
200 errors I think.
Look at the rest of the document. It is almost clear from the errors you can find throughout the document that this "report" was posted from some original hard copy run through some form of OCR.
I think this is a particular example of it. Why the number of LOC is 101000?
Maybe the OCR interpreted a bad comma from scanning as a "1", in which case the number would have read 10,000 LOC - and thus "twenty errors".
As another glaring example of this OCR issue, look for this line:
> "There were [.4ECL] people sitting in our offices telling us it [the Therac-25] couldn't hurt anybody when they knew it could."
"[.4ECL]"?
It should read "[AECL]" - but the scan or the copy it was scanned from probably had a small break on the arm of the "A" - which the OCR interpreted as ".4"
You can find other small anomalies like this elsewhere in the document if you look for them. They kinda bugged me, but I could tell from the pixels what was going on.
;)
EDIT: My own mistakes fixed...sigh
https://twitter.com/BarbaraWadeRose
Design the security/safety system up front, for the task of the system being designed, for all potential failure modes - and then implement the first, and test the hell out of it, and document it thoroughly.
Unfortunately, this tends to get shot down, until it is needed, and then it gets bandaid-patched in place, because usually at the point it is needed, a whole refactor of the code base is usually called for, and we can't have that, so "just fix it" becomes the call of duty.
And it never works out properly. Things get missed, security is effectively out the window, until something happens, with the refrain of "why did this occur" and of course nobody wants to take the blame for "I told you so".
Unfortunately, even implementing such a security system first will only get you so far, and it is still possible that something might be missed, and potentially be the cause of a safety critical issue. But hopefully, most if not all of those extreme events can be caught first in the system, by some part of it, and what's left will be relatively minor.
What does that actually mean? Maybe in the Therac-25 case the maximum dosage could somehow have been limited by a hardware dial in addition to software. But what form would hardware safeguards take for the millions of decisions software makes every second on a starting rocket? And considering hardware has far higher failure rates than no-moving-parts software, wouldn't any system that lets hardware dominate software decrease safety?
From Wikipedia:
"One, when the operator incorrectly selected X-ray mode before quickly changing to electron mode, which allowed the electron beam to be set for X-ray mode without the X-ray target being in place. A second fault allowed the electron beam to activate during field-light mode, during which no beam scanner was active or target was in place.
Previous models had hardware interlocks to prevent such faults, but the Therac-25 had removed them, depending instead on software checks for safety."
A dose integrator would be a detector + an integrator independent of the therapeutic delivery software. If it goes over limit, it shuts the system down.
The previous title noted that this was bout the Therac-25.
...and I have to add that I'm one of those "informally educated software engineers" - though I've never touched or worked on any kind of safety-critical systems (and I would decline such a position if offered, to be honest). I've been doing SWE for over 25 years now, and the only "formal schooling" I've had in the field was a couple of community college classes I took to learn C/C++ 2+ decades ago. Everything else has either been learned thru my employment, or at home. Prior to my first "professional" position in 1992, all I had was a high school degree and an associates from a trade school in "computer electronics".
But even I know about the Therac-25, and it's always in the back of my head.
The fairly recent change in the industry surrounding automated testing, test case programming, CI, etc - all of that which more or less fell out of the whole "agile" movement - has led to a vast improvement in software, imho. Yet bugs still persist, despite all of that and more.
We have to do better - I don't know how, or if it's even possible (from what I understand, it's not mathematically possible, except in maybe certain trivial examples, to validate all possible states of code - maybe I'm mistaken on that).
It's good when not every title is completely obvious. It makes readers work a little and gets the brain out of internet reflex mode.
https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
Also, there's a historical aspect to this. We don't need, and needn't presume, to rewrite the titles of classic articles.
And yes, this is art not science: aimed at effect.
The Therac case study has acquired a recognition the contemporaneous article (and title) wouldn't have experienced -- the company name was then the more notable signifier. AECL has been far eclipsed by its most notorious product. Today, "Therac-25" should lead, on the same basis as "AECL" did in the original.
I'm generally a follower of Jacob Nielsen on microcontent:
Well-written, short text fragments presented out of supporting context can provide valuable information and nudge web users toward a desired action.
https://www.nngroup.com/articles/microcontent-how-to-write-h...
Mrtimer J. Adler does not fully condemn content concealment in book tables of contents, but his lips are clearly pursed, and nose wrinkled:
It used to be a common practice, especially in expository works ... to write very full tables of contents, with the chapters or parts broken down into many subtitles indicative of the topics covered. Milton, for example, wrote more or less lengthy headings, or "Arguments," as he called them, for each book of Paradise Lost. Gibbon published his Decline and Fall of the Roman Empire with an extensive analytical table of contents for each chapter. Such summaries are no longer common.... [P]ublishers have come to feel that a less revealing table of contents is more seductive than a completely frank and open one. Readers, they feel, will be attracted to a book with more or less mysterious chapter titles-they will want to read the book to find out what the chapters are about. Even so, a table of contents can be valuable.
-- How to Read a Book, p. 33.
The same argument applies to article titles at HN.
You've made the case (and commented on comunity failings) for defusing titles on hot-button subjects. That's valid.
This isn't that circumstance.
Pandering and information concealment as a deliberate policy, again not the case for the article as initially written, in the cotext of its time, is overtly manipulative, to no real gain.
Please don't do that.
Thank God for the curious minds of this world, the people who have he patience and dedication to reproduce bugs.
Discussion of another good article on the Therac-25 (with pictures of the machine) [2]
[1] https://news.ycombinator.com/item?id=7992548
[2] https://news.ycombinator.com/item?id=12201147
2015: https://news.ycombinator.com/item?id=9643054
2014: https://news.ycombinator.com/item?id=7257005
2010: https://news.ycombinator.com/item?id=1143776
> there have been complaints in the high-tech community that software documentation is hampering competitiveness
TSA employees are not permitted to wear dosimeters[1], which makes it hard to detect these things. Scientific American, the most established magazine in the US, is among respected bodies that have recommended dosimeters for accountability reasons.[2]
There are also extremely infrequent (and sometimes rigged) studies done on the equipment, and they often find dangerous conditions. For example:
>Doses for some of the baggage screeners exceeded the maximum dose for the public.... Some EDS machines were not well maintained (i.e., they had bent curtain rods and missing curtain flaps).... Most EDS machines emitted low levels of radiation; a few exceeded regulatory limits.[2]
[1]https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-... [2]https://www.scientificamerican.com/article/airport-screeners... [3]https://www.cdc.gov/niosh/hhe/reports/pdfs/2003-0206-3067.pd...
Zero. The TSA machines are not limited by software. They will catch fire themselves before killing people. Therac-25 was designed to destroy tissues, cancer tissues but human tissue nevertheless. It is like comparing an m-16 to a rubber band gun. They are both guns, but they are fundamentally different in terms of capabilities with the former.
At the airports. They still exist in prisons, in covert trucks and other mobile units owned by TSA and law enforcement, and potentially in other venues of public transportation, stadiums and arenas, etc.
X-rays are also in INCREASING use at airports, since they are in the process now of replacing the carry-on bag scanners with CT scanners. A CT scan gives a 3D image of your luggage which means you don't have to take out your laptop. Unfortunately, a CT scan also uses 10X more radiation than a regular x-ray.
1. Xray scanning should be outlawed for public safety use in humans. You want to scan humans? You must use millimeter wave, and the cost is on you.
2. Governance. It's perfectly acceptable to use CT scanners for non-human surveillance (carry on bags, checked luggage, etc), but you must ensure life safety is enforced. If you don't, steep penalties up to and including jail time.
Call your reps, run for office, vote for people who care about this.
When I was still in VT I was happy to see that our third tier airport managed to hang on the noticeably more effective metal detectors long after other areas phased them out, sadly they've now been swapped out.
https://www.cbc.ca/news/canada/hamilton/radiation-treatment-...
http://indico.ictp.it/event/a12208/session/65/contribution/4...
PRetty damning, if you ask me.
Here's their fationale from teh FAQ in the linked article [1]
"Q: Why aren’t your officers permitted to wear dosimeters?
A: There is a really good reason for this. The emissions from our X-ray technology are well below the requirements that would require their routine usage. To help reassure passengers and employees that the technology is safe, however,health physicists with the U.S. Army have been conducting area dosimeter surveys at multiple airports nationwide."
This is at best massively flawed reasoning by the TSA, and at worst a deliberate misleading rationale.
1) While the NORMAL dosage is of course well below any level that would make a dosimeter useful, a dose from a malfunction or bent/broken/missing shield panel could easily exceed dosimeter detection thresholds.
2) Spot checks are nice, but unless they are very frequent, they could easily miss a malfunction and leave it overdosing agents and people for a long time. Even if inspections are every 6 months at every airport station, that will leave a malfunction overdosing people for on average, three months.
3) Detecting, identifying, and repairing a malfunctioning system could be done much sooner and at lower cost by using dosimeters, either on employee volunteers or just hanging them on the chairs & workstation surrounds.
It is this kind of frankly stupid stick-your-head-in-the-sand reasoning that breeds distrust of governments and the companies that supply them.
[1] https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-...
That doesn't mean they aren't allowed to perform dosimeter readings at all, it just means it's not acceptable to carry this equipment around all the time.
> While the NORMAL dosage is of course well below any level that would make a dosimeter useful, a dose from a malfunction or bent/broken/missing shield panel could easily exceed dosimeter detection thresholds.
Do you actually know that these could be harmful, or is it just a suspicion?
So I'd assume this is more like coal miner operators in the 1800s planning ahead to protect themselves from the eventual outbreak of black lung.
As far as we know, there is no general ban on wearing instrumentation of any other on the job, unless it is somehow clumsy and would interfere with their duties or the appearance of their uniform.
In contrast, the simplest dosimeter is typically a very small patch of photographic paper in an envelope opaque to visual light but transparent to harder radiation (towards the gamma end of the spectrum). It is like an inch square, or like a typical nametag, easily worn in a shirt pocket, no more visible to anyone else than a credit card in the shirt pocket.
The only plausible reason to forbid this is to prevent any kind of whistleblower event.
Of course I do not know the actual levels of radiation inside each machine or all of their possible failure modes. However, we do know
1) high-energy radiation dosage is cumlulative -- each day's dose adds up, so if you have 1/100th of a critical dose each day in a workzone, in less than four months, you should be banned from that work zone. This is the way it works in many facilities dealing with radiation sources, and everyone wears dosimeters which are checked, and people are reassigned when the total dosage exceeds the prescribed level.
2) There are medical limits on how many x-ray and CT-Scan procedures are allowed for patients, again due to the cumulative nature of radiation exposure. Even with the very low-doses of modern machines, these are a concern and are tracked (I know this from the experiences of several close family members).
3) The TSA machines do produce sufficient radiation in their beam to fog ASA800 and higher film. This is definitely a non-trivial dose. Tho obviously not lethal in a single dose (e.g., if you rode through the machine), it is significant. If an interior shielding panel is bent or somehow mis-installed during manufacturing, maintenance, or repair, it is entirely plausible that people working regularly in the area of the leaking radiation could be overexposed.
So, no, this is not even close to mere suspicion or Luddite "ooh radiation bad" -- and yes, we do know that these could be harmful, especially over extended periods.
I hope you are not an employer with that attitude.
Who is "we"? Aren't you just pleading ignorance here? What do really you know about what TSA agents can and can't wear? Anything can "interfere" with the appearance of a uniform, including a "small photographic patch".
> The only plausible reason to forbid this is to prevent any kind of whistleblower event.
Then why do they do allow third party monitoring?
Having further looked into this, the page you posted is outdated and they ended up handing out dosimeters themselves:
https://www.livescience.com/36079-airport-security-screening...
The result was negative. Of course you can't prove a negative, maybe there's radiation somewhere else where they didn't look...
> 2) There are medical limits on how many x-ray and CT-Scan procedures are allowed for patients, again due to the cumulative nature of radiation exposure.
> 3) The TSA machines do produce sufficient radiation in their beam to fog ASA800 and higher film. This is definitely a non-trivial dose.
According to: https://www.radiologybusiness.com/topics/business-intelligen...
" TSA claims the “backscatter machines” cannot produce more than 0.005 millirem of radiation per scan. In comparison, the agency says, a chest x-ray exposes patients to 10 millirem of radiation, and the maximum recommended exposure to radiation from man-made sources is 100 millirem per year."
That would mean that if you had a machine that was completely unshielded, you would have to perform 2000 scans to get the equivalent of a chest X-Ray. Compare that to the radiation exposure of your average flight attendant.
Not if it is inside your pocket. Get real.
>> Then why do they do allow third party monitoring?
Obviously because they can control when it is done as well as how the results are distributed, or not distributed.
>> " TSA claims the “backscatter machines” cannot produce more than 0.005 millirem of radiation per scan. In comparison, the agency says, a chest x-ray exposes patients to 10 millirem of radiation, and the maximum recommended exposure to radiation from man-made sources is 100 millirem per year."
>> That would mean that if you had a machine that was completely unshielded, you would have to perform 2000 scans to get the equivalent of a chest X-Ray
1) this assumes that the "0.005 millirem per scan" is the total output of the machine, including all radiation absorbed by the lensing & shielding. We do not know if this figure is the total generation output, or only the total amount put to the target & receiving sensors.
2) one thing we do know is that the radiation will NOT be scattered evenly; there will be lobes and hotspots and cold spots.
3) even if we assume .005millirem/scan, at the rate of two/minute, that's 1000 minutes to get a chest x-ray, or 16 hours, basically two work days. In LESS THAN ONE MONTH, they are over the yearly dose.
More importantly, this is only counting the backscatter machines that are specifically designed for super-low doses as they are specifically irradating people.
This does NOT look at the luggage scanners, either for carry-ons or for checked luggage, which likely have much higher outputs.
Finally, even if your arguments are correct and there is truly no risk, there should be no problem at all for officers to wear their own dosimeters -- the results would merely be putting their minds at ease that they are not getting exposed.
Thus, we are back to the only reason to prevent officers from wearing dosimeters is an attempt to control the information and prevent whistleblowers and/or lawsuits.
Again, I sincerely hope you are not an employer with this attitude.
Let's apply some Occam's Razor here.
I am led to believe they know that these machines are harmful and therefore they are carefully orchestrating all testing of these devices by, among others, all of the following parties: "U.S. Army Public Health Command, the Food and Drug Administration’s (FDA) Center for Devices and Radiological Health (CDRH), the National Institute of Standards and Technology (NIST), and the Johns Hopkins University Applied Physics Laboratory (APL)".
That's all done to prevent some sort of mass lawsuit, which might cost the United States some money.
The simpler explanation is that TSA rules are simply really strict, because that's just how TSA rules are.
https://www.quora.com/Are-TSA-screeners-permitted-to-wear-ra...
The rules don't have to make sense. There's a whole lot about the TSA and its procedures that doesn't make sense.
> We do not know if this figure is the total generation output, or only the total amount put to the target & receiving sensors.
I'm content with the description given that these machines "cannot produce" more than that. You can do your own research on the matter.
> 3) even if we assume .005millirem/scan, at the rate of two/minute, that's 1000 minutes to get a chest x-ray, or 16 hours, basically two work days. In LESS THAN ONE MONTH, they are over the yearly dose.
...assuming no shielding whatsoever. You would hope that after two months without shielding on the device, someone might notice. Even then, such exposure would be well below the occupational limit for radiation workers (5000 millirem):
https://ehs.stanford.edu/manual/radiation-protection-guidanc...
> This does NOT look at the luggage scanners, either for carry-ons or for checked luggage, which likely have much higher outputs.
Fair enough, but you picked an article that was specifically concerned with those backscatter devices.
That's not even close to the simplest explanation. Far simpler is that they are careless, clueless, and paranoid, and don't want anyone finding out something bad on their watch. Or they just don't want to allow anyone to get concerned.
>> The rules don't have to make sense. Unless we're living in a Kafka or Vonnegut book, they are supposed to have actual reasons and make sense.
Calling out one particular type of measuring device, which happens to measure the one plausible type of hazard in that workplace, yeah, that's suspicious.
>> .assuming no shielding whatsoever. Does not assume no shielding whatsoever, only a plausible hotspot or focus. The point is that the potential dosing is in the range of values that could cause a hazard, and even at 10% of that value, would be overdosing within a year. It is not like it is totally off the scale (as are many of the public's fears).
>> You would hope that after two months without shielding on the device, someone might notice.
Yes, you would hope that. However, if you've ever read reports on industrial issues, you'll find that far worse and more noticeable hazards go unnoticed/unfixed for YEARS at a time before they get around to killing someone. And this sort of low-level radiation is particularly pernicious, as it is completely unnoticeable until after it is too late.
The simple solution is, assuming that these machines are largely safe (which I generally expect), is to say "you can buy dosimeters on your own dime and wear them if you like, just keep them out of sight in your pocket while on duty. If anyone gets any unexpectedly high readings, we'll investigate, but remember since you haven't got a chain of custody around the meter, it'll be only the beginning of an investigation, not immediate acceptance of your results."
Ye, it may turn out to be bit more work, but it shows you're on the side of worker safety, and on the road to getting things fixed. Mostly, it'll quell any fear and likely cause no one to wear one.
With the rules as it is, I'd be damn sure to sneak dosimeters into my clothes while working there.
Astonishing this basic safety precaution was not already in use. Do modern devices do this?
I guess nobody thought about Murphy's Law.
> [...] the Therac-25 typically issued up to four error messages a day. It did so by displaying "Malfunction" plus a number, from 1 through 64. No explanation was offered by the computer nor was there any reference to the malfunction codes in the operator's manual. Technicians could, in most cases, bypass the irritating malfunctions simply by pressing the "p" key, for "proceed." Doing so became a matter of habit.
Even if you're not doing safety-critical systems, there are many lessons to be learned just from this bit alone.
Use human-readable error messages, and make sure they are understandable to the users/operators of your software (not just developers). Including a specific error code is fine -- it can disambiguate similar errors, and make it very easy to do a 'find all' in source to go directly to the offending spot -- but that doesn't help your user.
If you're going to allow users to bypass an error, make it deliberate. The best way to do this is debatable, but through years of bad software, users have been trained to dismiss popup messages by pressing OK or 'yes' or 'proceed'.
When you show an important error, make sure it's different from informational messages or warnings. Showing users routine warnings and messages is fatiguing and basically just trains them to always press the 'p' key.
> According to a computer system's analysis of FDA documents, the computer would not accept new information on a particular phase of treatment (in the case of both Tyler accidents, changing the x-ray mode to electron mode) if the technician made the changes within eight seconds after reaching the end of the prescription data. That's what Malfunction 54 meant.
Consider if they had actually tried to show an error message for this (or even document what it meant). Something like "Data on screen does not reflect actual parameters being used. Proceed?" Hopefully at some point during development, someone would question that: "why are we allowing them to proceed in this case?" "If we can detect that is the case, why even show this as an error, instead of just updating the parameters?"
> Use human-readable error messages, and make sure they are understandable to the users/operators of your software (not just developers). Including a specific error code is fine -- it can disambiguate similar errors, and make it very easy to do a 'find all' in source to go directly to the offending spot -- but that doesn't help your user.
Yes. Remember, however, that much software needs to be localized, even if the initial developer didn't think to. Error codes are much better than hard-coded English-language error messages; map them to a message at the last possible moment.
Sorry for linking the daily fail, but the only articles I can find in English seem to be in UK tabloids: https://www.dailymail.co.uk/news/article-7066289/Cancer-pati...
по-русски: https://dni.ru/regions/2019/5/23/424405.html
As an aside, every time I step on the treadmill at the gym I check that the emergency stop button works and that the incline down and speed down buttons work. I have been stuck on a treadmill which I couldn't stop or slow down before.
Mechanical failures need to be dealt with differently than software. Parts must be replaced before they fail. Even if they appear to work correctly. Software doesn't wear out so its a whole different ball game.
Sounds like there is an obvious design failure - similar to garage or elevator doors the table shouldn't move and push up beyond some force threshold (patient weight plus say 200N of force).