> When Rajkumar has raised concerns with those Tesla employees about autopilot's technical limitations, the response is they have to "wash their hands of it" because "it's a business decision."
I've wondered about Tesla's engineering management before, and this would support that it is inadequate. Engineers cannot "wash their hands" on safety critical projects. This is potentially a serious ethical lapse: the health, safety, and welfare of the public is the foremost concern:
If engineers' judgment is overruled under circumstances
that endanger life or property, they shall notify their
employer or client and such other authority as may be
appropriate.[0]
> In another anecdote recounted by two sources, Musk was told that the sensors used for Tesla's self-parking feature might have difficulty recognizing something as small as a cat. Musk is said to have responded that given how slow the car moves in this parking mode, it would only be dangerous to "a comatose cat."
Outrageous, a baby can be the size and of the speed of a "comatose cat". Hysterical example aside, a comatose cat demands no less consideration with respect to risk analysis. Handwaving that the the feature is of no safety concern, rather than submit to the rigors of risk management, is an absolutely improper response.
But is it really risk analysis if any negative eventuality, no matter how unlikely or infrequent, is a deal-breaker? Risk analysis is exactly that - figuring out what the risks are, their likelihoods and potential impacts, and then figuring out which ones are worth taking and which are not.
So the salient fact remains the same: deaths with autonomous driving are less than deaths without it. Isn't it unethical to prevent such a feature from going out into the world when it could be saving lives, due to concerns about some unlikely / infrequent eventuality or driver error? Is it sound risk analysis to allow hundreds, thousands of people to die when you have a feature that could save their lives?
Cars are inherently unsafe, and yet we let people drive them. Clearly we already (ethically) accept some risk in our products.
> Clearly we already (ethically) accept some risk in our products.
We do, when the risk is managed. A key feature of risk management is, after identification of potential risks, the minimization of the occurance probability or the severity to within an acceptable range.
But we as engineers have to recognize that the socialization of technology is squishy and illogical.
Even if we're correct, and that dispassionate analysis will move society forward, not everyone will agree. And they will have strong arguments, made stronger if we aren't contextually aware.
Demonization of tech is very possible. If we want driverless cars and a host of other things in our lifetimes, we have to be approach the socialization with some sensitivity.
Brashness is fine, but risky. If it's just a website going down in flames, no one cares. But cars kill people. Brashness will be rewarded with regulations. Possibly shortsighted regulations that do more long term harm than good, because politics.
Today, drivers (and mechanical failures) kill people and we have a social structure to deal with that. Tomorrow, cars will kill people in situations where drivers would not have, and where everything is operating according to spec.
That's a big change, which not everyone is ready for. And snotty tech billionaires will not be sympathetic characters in the ensuing press/political discourse.
I'm not sure about your engineer "cannot wash their hands" comment. Let's reframe it about seatbelts.
Seatbelts are a safety innovation, which save lives. They also kill and injure people.
It's not clear to me that an engineer working on making seatbelts work better is the best person to decide when they reach the point that the benefits outweigh the drawbacks. It's a totally different skill set.
> It's not clear to me that an engineer working on making seatbelts work better is the best person to decide when they reach the point that the benefits outweigh the drawbacks.
I agree that they're not the best person, but they are an active participant in the process. As a participant, they have an ethical duty to raise their concerns. Consider that an engineer "working on making seatbelts work better" is (arguably) the organizational expert on seatbelt operations. It is perfectly appropriate for engineering management to address the raised concerns through risk management and for the engineer to accept this decision because it was through a defined PLM process.
When, as in the quote, the justification for accepting the dismissal of concerns is that "it was a business decision", it raises red flags for me.
6 comments
[ 1.9 ms ] story [ 25.4 ms ] threadI've wondered about Tesla's engineering management before, and this would support that it is inadequate. Engineers cannot "wash their hands" on safety critical projects. This is potentially a serious ethical lapse: the health, safety, and welfare of the public is the foremost concern:
> In another anecdote recounted by two sources, Musk was told that the sensors used for Tesla's self-parking feature might have difficulty recognizing something as small as a cat. Musk is said to have responded that given how slow the car moves in this parking mode, it would only be dangerous to "a comatose cat."Outrageous, a baby can be the size and of the speed of a "comatose cat". Hysterical example aside, a comatose cat demands no less consideration with respect to risk analysis. Handwaving that the the feature is of no safety concern, rather than submit to the rigors of risk management, is an absolutely improper response.
[1] https://www.nspe.org/resources/ethics/code-ethics
So the salient fact remains the same: deaths with autonomous driving are less than deaths without it. Isn't it unethical to prevent such a feature from going out into the world when it could be saving lives, due to concerns about some unlikely / infrequent eventuality or driver error? Is it sound risk analysis to allow hundreds, thousands of people to die when you have a feature that could save their lives?
Cars are inherently unsafe, and yet we let people drive them. Clearly we already (ethically) accept some risk in our products.
We do, when the risk is managed. A key feature of risk management is, after identification of potential risks, the minimization of the occurance probability or the severity to within an acceptable range.
But we as engineers have to recognize that the socialization of technology is squishy and illogical.
Even if we're correct, and that dispassionate analysis will move society forward, not everyone will agree. And they will have strong arguments, made stronger if we aren't contextually aware.
Demonization of tech is very possible. If we want driverless cars and a host of other things in our lifetimes, we have to be approach the socialization with some sensitivity.
Brashness is fine, but risky. If it's just a website going down in flames, no one cares. But cars kill people. Brashness will be rewarded with regulations. Possibly shortsighted regulations that do more long term harm than good, because politics.
Today, drivers (and mechanical failures) kill people and we have a social structure to deal with that. Tomorrow, cars will kill people in situations where drivers would not have, and where everything is operating according to spec.
That's a big change, which not everyone is ready for. And snotty tech billionaires will not be sympathetic characters in the ensuing press/political discourse.
Seatbelts are a safety innovation, which save lives. They also kill and injure people.
It's not clear to me that an engineer working on making seatbelts work better is the best person to decide when they reach the point that the benefits outweigh the drawbacks. It's a totally different skill set.
I agree that they're not the best person, but they are an active participant in the process. As a participant, they have an ethical duty to raise their concerns. Consider that an engineer "working on making seatbelts work better" is (arguably) the organizational expert on seatbelt operations. It is perfectly appropriate for engineering management to address the raised concerns through risk management and for the engineer to accept this decision because it was through a defined PLM process.
When, as in the quote, the justification for accepting the dismissal of concerns is that "it was a business decision", it raises red flags for me.