Tell HN: Duty of care
I was once asked to remove a hardware safety mechanism from a medical device [1] and replace it with a software only function. Technical first, then we'll discuss wetware.
The process was started by, and results displayed on, an embedded Windows PC [2]. The PC didn't control any safety critical operations. Hardware peripherals had their own sensors, firmware, and safety mechanisms. You could incinerate the PC at any point and be fine.
There was a Big Red Button acting as a physical power cut-out. Request was to remove the physical button and rely only on a software (touch screen). Death was a remote harm, but moderate injury / user stuck in device / panic attack was plausible. And it's Windows, so duh.
Managers didn't contest my logic, but the request came down from none other than the Chairman of the Board of Directors of a publicly traded company, who was also a major shareholder and took a personal interest in the project. He ran over every layer of management until he ran into my skinny ass. I'm the runt of the litter that would have been pecked to death in the nest if his parents weren't so amazing.
I said no way. Not going to happen. Not going to do it. But I never hinted at resignation. They were going to have to fire me or go around me while I take discoverable notes.
That Big Red Button is still there to this day.
"Never doubt that a small group of thoughtful, committed, citizens can change the world. Indeed, it is the only thing that ever has." - Margaret Mead
Some suggestions for people in similar circumstances:
1. Live within your means and have savings. Hard to stand your ground if you're living paycheck to paycheck.
2. Suggest reasonable alternatives and keep cool.
3. Document. Even if it's just an email to yourself. Who, what, where, when, why, and how much. Update it as the situation evolves. Stick to raw facts you have direct knowledge of.
4. Research and cite applicable regulations or similar situations. Bring it up with company legal / regulatory affairs staff. Email them after the meeting with a summary of what you discussed.
5. Bypass chain of command. Make sure people calling the shots have been given your data.
6. Don't resign. Just put your tools down.
7. ---- I've never had to cross this line, but theoretically….
8. Sign nothing that looks like a liability release or NDA (often included in severance offers) if they terminate you.
9. Contact government regulators. OSHA, FDA, FAA, etc.
10. Nuclear option is go to the press, but I don't suggest that unless you can afford to bring a lawyer and love a shit show. You probably already signed an NDA before this point. But personally, if it involved risk of death, I'd do it.
References:
[0] https://twitter.com/LdnmaSentinel/status/1589822045592096768
[1] The studious among you will note the similarities to the Therac-25: https://en.wikipedia.org/wiki/Therac-25
[2] At one point we had requirements for hardware peripherals that only had Windows drivers. Otherwise would have gone Linux or some RTOS.
Edits: formatting
54 comments
[ 4.9 ms ] story [ 128 ms ] threadSorry if that’s a dumb question.
The Big Boss Man hated the way it looked. He once spent 30 minutes telling a graphics designer what a worthless piece of sh* he was where everyone on the floor could hear. Corporate equivalent of a terrorist.
> Request was to remove the physical button and rely only on a software (touch screen). Death was a remote harm, but moderate injury / user stuck in device / panic attack was plausible.
Good on you for not removing it.
I work sometimes in mineral processing (above ground | below ground) with conveyor belts, crushers, etc.
Big easy physical power cutouts are everywhere:
Conveyors have "rip cords" running alongside, pull them and you can stop a 30 tonne/hour conveyor maybe before someone drawn into the rollers is crushed to death or maimed.
Crushers | screens | most big equipment has single OFF knife switch with "gang plates" that take multiple padlocks - if you and two others work inside a crusher you each put a lock (three in total) through the gang plates and nobody gets to restart the machin until everybody is out and has removed their padlock.
Big physical switches save lives.
- Richard P. Feynman
The solution is to talk to arrange to talk to the boss directly, even if you’re several levels away in the hierarchy. Sometimes it’s not worth it, but in a situation like this it clearly is. 9 times out of 10 a situation like this can be resolved with a sit down over a coffee.
Most other professional engineering disciplines have clear, codified rules of ethics. If the Chairman of a public company told a building engineer "Can you get rid of a few of these bolts on this beam here? I don't like the way they look." he would rightfully be told to get bent by literally nearly any structural engineer. And it's not because structural engineers have vastly greater morals or thicker backbones. It's because they know they could, and should, lose their engineering license (not to mention be personally liable) if something were to go awry due to a clear violation of engineering standards.
Kudos to you for sticking up for what was right, but it's still an overall process failure that this situation required you to have this backbone instead of being able to fall back on "What you're asking me to do is a clear violation of professional standards that could cause me to lose my license."
A single week lecture on this topic could move the needle.
Or regulators like FDA could demand to see corporate training materials given to software "engineers" (I concur on the quotes) on how to promote product safety.
Good on you for taking it as an elective, but it's weird to me that any degree in any topic can be accredited without having a mandatory ethics course, let alone a degree in a science/engineering field.
For an anecdotal example, I have been told that standing up for ethics in software engineering would have negative consequences on my performance (implied: bonus, promotions, career). I have left that company, and they probably found someone more agreeable to replace me.
I don't know how we can force ethics onto companies, especially large corporations who receive thousands of applicants into engineering roles each week. There do not seem to be good incentives for engineers to be overly concerned with the overall ethical impact of their work. And so there are many engineers who won't ready to replace those who would.
Obviously this doesn't solve all the problems, but it works as well as any solution I can think of.
The caveat when it comes to software is that coming to a consensus of what the standard procedures and policies should be would be nearly impossible. If and when software joins the licensed engineering fields, a lot of people are going to be very upset at whatever the requirements end up being.
Perhaps licensure could be an effective solution, but one that is not very empathetic to engineers. Maybe some kind of a government-owned ethics controller/body to handle unethical software would be more just for engineers. Although it could also be very inefficient.
For egregious ethical violations, the whistleblower act provides a remedy.
Not feasible to ask this for every developer, but for safety critical systems it should be mandatory.
This is exactly my point. The same is true for other engineering disciplines, too, but by codifying their ethical responsibilities (and, in some cases, assigning liability to engineers who forego those responsibilities), other professional engineering organizations help to ensure a bare minimum for what is and isn't allowed in their profession.
I'll give another example that has much less dire consequences than the OP's. 90% of "scarcity marketing", e.g. "Act now! There are only 2 rooms left!" or even "8 other people are looking at this property!", is complete and total bullshit. I've even seen A/B tests where they developers were like "yeah, the data here is not real, we just want to see if it has an effect." Why is this even in the realm of acceptability? There is no gray area here - it's not just a "dark pattern". It is 100% outright lying. Yet I never heard someone stand up in those product reviews (myself included, so I'm no hero either) and say "How can we spend so much time on our 'company values' when this is obviously bullshit and slimy?"
I wish there were a "software engineering code of conduct" that said that outright lying to end users is verboten, and that software developers can be held personally liable if they are aware of the lie and still implement it.
For OP's case case, I'd assume/hope a professional engineer would need to sign off on this change or be able to tell the VP to f off.
This was before the device was submitted to the FDA for approval. Chance they would have rejected it. A fallback position for a developer not willing to Play It Hard would be to make sure the risk docs submitted to FDA document this failure path. There is a whole process you are supposed to go through to analyze risks / harms and whether your controls are adequate.
How can we offer that guarantee as a profession when hardware, drivers, OSes and even libraries that we use change and shift without our control and sometimes consent?
Critical systems (or at least the critical subsystems) must not allow any unapproved/untested changes. Hence why it's better for safety controls to be hardware/firmware and not part of a general-purpose OS.
It's one reason why you see separate payment terminal hardware on self-checkout kiosks. The payment hardware is more tightly controlled whereas they can modify the kiosk much faster.
There are also RTOS (real time OS) that offer execution time guarantees. Used in aerospace.
Point being; the software industry is deeply dysfunctional; and the best people who aren't steeped in software every day can do to influence their decision making process is to trust and listen to the people who are. Software engineering is as much Engineering Software as it is Navigating The Absolute Dysfunction of our Software Landscape; actually now that I type that, I think its significantly more the latter.
I don't know if anyone is trying to fix this, or at least do better. I really hope so. I've been on major teams at three systemically critical internet companies and its not getting better; its getting worse. It scares the living daylight out of me that the biggest thing holding everything together is PagerDuty.
Fly-by-wire in airplanes is pretty good, but it still has its problems, and a lot of verification work goes into it, probably more than embedded Windows.
[0] https://www.acm.org/code-of-ethics [1] https://www.ieee.org/about/corporate/governance/p7-8.html
I once designed a hardware state machine out of TTL (no CPU, except a network interface that could only passively read and transmit the state) to ensure that a pressure vessel was safe (and this was ahead of the mechanical safety devices). Regularly someone would ask why we had this large device and didn't just replace it with a tiny MPU.
Also, companies have some discretion in what changes they re-submit to the FDA and what the decide to just document internally. There are guidelines, like if you change the OS you better resubmit, but there is a wide swath of gray area. The FDA is not looking at every code/hardware change request. They audit your process.
Edit: typos
And had your management not backed down, and tried to push this through as a Letter to File, you'd have been well-within your rights to blow the whistle to FDA. There's not a FDA employee alive that wouldn't go "hold on there chief" to a device change of this magnitude.
Engineers have some power to prevent harm. Use it when it matters.
First, there's your story. To describe it as vague would be an understatement. What was the medical device? If the device does something important for the patient, wouldn't a big red button also be dangerous? You wouldn't want it turned off accidentally. Also the machine needs a button or switch to turn on, right? So why not have that component be the physical switch? And if the kill switch is on the touch screen, wouldn't that take up valuable pixels that could be used for other purposes? It would also make accidental triggering more common.
And you claim the button removal request was from the Chairman of the Board of Directors of a different company, who was a major shareholder? The board of the company you worked at, the C-level executives, and management all caved to him but you didn't, and you kept your job? That makes no sense. Someone with that much influence should have no problem getting an individual contributor let go.
Then there's you. Your Twitter account was created in September of 2017, but either you never tweeted until September of 2022 or your tweets auto-delete. Maybe you changed your username because neither archive.org nor archive.is have any history for your Twitter handle.
Your domain[1] was registered on September 22nd of this year, and it contains some general-sounding activism-du-jour. One might even call it "current thing". If anything it seems quite contradictory. Liberty does not march abroad... except when fighting a proxy war in Ukraine? OK, I guess.
I know my comment is controversial, but your post is exactly what I would write if my goal was to create perfect HN bait. It seems much more likely that someone lied on the internet than that things actually happened as described.
1. https://ldnma.com/
Your due diligence data is largely correct and I complement your research. I repurposed this twitter account from one I created in 2017 and used as read-only in the intervening time. Also worth pointing out my HN account is quite new.
Wasn't great to cross the wires of my advocacy and technology, but I used the account I had. Didn't expect my twitter comment to be noticed.
I didn't promote my blog at all in this post, but since you mention it, if you read it more carefully I think you will find I take care to distinguish the Ukraine conflict from prior US interventions. Namely, one is a defense of a democracy which invited us whereas others (Iraq, Afghanistan) were not. US troops are not marching into Ukraine. It's even less direct support than the French provided the US revolutionary war.
I don't take issue with your questioning except - the presumption that someone is lying before they have even had a chance to respond to your information.
Edit: removed redundant sentence
Always keep in mind that when shit hits the fan they throw your name very quickly and you are going to stand in court to defend yourself. That's what happens when you work on devices which can kill people.
I learnt that they cannot reproach you to stay ethical and they know it is very difficult to make someone do something which goes against his values, it is easier to find someone else.
But that's not how things work in most medium-large companies. I was asked to implement some telemetry in our client applications that I considered was going too far (clearly designed to get more information than was strictly required for feature function). I said no, manager said ok and assigned the task to John. John said sure boss and come end of year also got a promo. The telemetry is still there, now piped into a "personalization" feature; all above board (ToS-wise) of course.