HN Explorer

Ask HN: Are there real-world examples of moral failures of software engineers?

60 points by objectiveariel ↗ HN
Consider the following projects:

* Facebook's Free Basics initiative in India: http://www.bbc.com/news/technology-35169226

* The Stingray phone surveillance device: https://en.wikipedia.org/wiki/Stingray_phone_tracker

* North Korea's Red Star operating system: https://media.ccc.de/v/32c3-7174-lifting_the_fog_on_red_star_os

* Hacking Team's surveillance software (sold to countries with a poor human rights track record): https://theintercept.com/2015/07/07/leaked-documents-confirm-hacking-team-sells-spyware-repressive-countries/

Do you think that the software engineers who consented to work on the above projects acted ethically?

76 comments

[ 5.1 ms ] story [ 143 ms ] thread
SideburnsOfDoom
How about the engineers who wrote this code? https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal

I can't put all the blame on the software engineers involved: the decision to do this surely lies higher up the chain. But did anyone of the coders object or blow the whistle beforehand? Or were they all fine with "just following orders"?

Now that more is known, have any come forward or even given an anonymous interview about it afterwards? AFAIK,no.

bakhy
good point. although this actually crosses the line from "just unethical" to straight up "illegal", which makes the question perhaps less interesting. developers who wrote that code are accessories in crime. and the defense of "just following orders" might only be considered for soldiers, not for civilian workers.
pc86
It's typically not even considered and excuse for soldiers, is it? Just because someone may be immune from civil prosecution does not mean there are not equally (or more) severe consequences in the military justice system.
ni-hil
I find it hard to believe you could be punished in the military system if you followed your superior orders.

Edit: my bad, in fact you can, if the order is illegal, and it has been unsuccessfully used as a legal defense in hundreds of cases. But, considering what you risk when you don't follow orders, it seems like a tought decision to take. If you don't follow orders, you are done and will probably be punished. And if you follow and you get caught it's on you and you will have to assume full responsability...

Here is more info on this subject: http://usmilitary.about.com/cs/militarylaw1/a/obeyingorders....

bakhy
i think it is, but to some very limited extent. it is not desirable for soldiers to question commands too much. even the most human rights obeying, peace loving government typically does not want its soldiers to think too much. (i think that may even be part of the reason why they have a separate court system, no?)
pjc50
This isn't the military. One of the side effects of corporate "personhood" is that the crime was committed by Volkswagen. Sale of illegal products is not usually attributed to any individual in the process, unless the company is a single person.
bakhy
your idea actually sounds more like the military ;) since, according to that vision, individual people nor their personal responsibility do not even exist there.

can a person really just knowingly take part in any illegal activity, and as long as they're not the only employee, the company is to blame and they are free? even if that were true, and i doubt it is, it would need immediate changing.

icebraining
It's not certainly not always true, and the recent Yates memo shows that, in the US, the DOJ is actually pushing for more individuals convictions.

That said, it may be true in VW's case, I don't know.

pjc50
This depends on a lot of things, such as what the illegal activity is, whether they were told to do it, whether they knew it was illegal, and so on.

In this case, it's a product sale regulation. It's not illegal to manufacture non-compliant engines, but it's illegal to sell them. So in a world of individual-only responsibility, that would make the individual sales agents who performed the transactions with customers liable. I don't think that's an improvement.

(This answer may change if the sales agents know or should have known that the product was not legal to sell; this definitely applies to e.g. sales of alcohol to minors)

bakhy
thank you for the clarification! one issue still remains - a developer writing that software knows that it will end up in cars which will be sold. even if they will not sell them themselves, they are certainly in the know about the illegal practice. if not before, than at least when those cars start getting sold, the developer knows laws were broken. not blowing the whistle at that point must make the developer accessory to the crime. i think that is inescapable. or, should be..
[deleted]
(comment deleted)
sqrt17
Apparently, Volkswagen's emissions regulating code was built to Volkswagen's orders by Bosch, who then delivered the code with an explicit note that that code does not fulfill the requirements to be installed in production cars.

See here for a more complete story: https://media.ccc.de/v/32c3-7331-the_exhaust_emissions_scand...

sliverstorm
They told Bosch it was for testing and development, though Bosch was suspicious. So Bosch engineers were hardly complicit
[deleted]
(comment deleted)
ethbro
There's a lot of ethical failure in failed implementations.

I realize these kind of projects are big consulting firms' bread and butter, but if you as an engineer continue to work on a project that's being managed with an eye towards "Failure is okay, because our lawyers wrote the contract to cover that contingency" then that's pretty scummy from an ethical perspective.

https://en.wikipedia.org/wiki/Virtual_Case_File

Edit: On the flipside, I feel a lot more comfortable when employers I've worked for have sat down with customers and had the "Look, this just isn't working out. We recommend you cancel this project and we tie off our relationship, because it's not going to end well for either of us if we continue" talk.

jerf
Errr, define "ethically".

I'm not trying to trap you. I'm willing to work with whatever definition you specify, and I won't try to play semantic games with the definition if it's at least close enough to something specific to work with. I'm not asking for a universality claim. But without some specification of what you mean the question is vague to the point of unanswerability.

The North Korean programmers may well have truly believed in what they were doing. A utilitarian may well truly believe that even if Facebook Free Basics isn't a perfect program, it's a net good for the participants. The "surveillance software" can be seen as just a tool and whether the tool makers are responsible for its misuse is ethically debatable. (And let me be clear I mean that literally, not as an attempt to rhetorically state a position. I could write a coherent argument both ways.) After all, many people even in free countries end up calling for strict regulation of corporations and that same "surveillance software" is pretty much the way you instantiate such regulation, so, is it really clear that it's intrinsically evil?

And again let me emphasize the point I'm making here is just the width of possible arguments about ethics that can be made. My previous paragraph is itself ethically incoherent, inasmuch I'm not even trying to take a consistent stand overall, but merely trying to highlight the most obvious problem per issue where ill-defined "ethics" makes it hard to even debate the matter.

RodericDay
Why? Defining ethics is extremely difficult, but volunteering our intuitions via examples helps us get closer and closer to narrowing what it is exactly that we're talking about.

It's mighty hard to pin down a universal definition for "art", "love", and even "game"- and yet we use these words regularly and mostly very successfully to communicate.

Discussing "ethics" is difficult too, but I am unconvinced by arguments of the "it's obviously all subjective" variety.

[deleted]
(comment deleted)
[deleted]
(comment deleted)
objectiveariel
> I'm willing to work with whatever definition you specify, and I won't try to play semantic games with the definition if it's at least close enough to something specific to work with.

My question is precisely asking you to define that.

OliverJones
Ethics is a slippery subject. Different people have different ethical stances, so if I were to say "that's unethical" to you, you'd be likely to misunderstand me.

One common ethical stance is utilitarianism. This stance purports to optimize collective welfare. I don't know any NSA people, but it seems likely that they share a utilitarian approach to ethical decision making, and they define happiness as security from evildoers. (I'm speculating about their stance, not claiming it to be my stance.) Selfish interest is part of utilitarianism.

For example, if you have a utilitarian stance you might choose to refrain from having sex with strangers to protect your health and theirs.

Another ethical stance is deontology. In this stance you refrain from having sex with strangers because it's externally defined as wrong. For example, "Do not commit adultery" shows up in one common collection of externally defined rules. If you say to someone, "that's illegal" you're speaking from a deontological stance.

A third stance is altruism. When operating in that stance, people value the welfare of others above their own welfare. Many who donate blood do so from an altruistic stance.

Real people have a combination of actual stances. And for most of us, our real, operative stances are often not quite aligned with what we say our stances are. That's just reality. Almost nobody completely walks their talk on this stuff.

An engineer who works through the night to repair a critical defect probably has a combination of ethical attitudes. Trying to make users happy is altruism. It may also be deontological -- they're violating their quality agreements. It may also be selfish and utilitarian: losing face, losing revenue and getting fired are to be avoided. All that is fine.

Life is harder when different people have contradictory stances. The life and death of Aaron Swartz is a tragic example of that.

Immanuel Kant proposed the "categorical imperative". (Oversimplifying) he suggests that we should live and behave ourselves the way we WISH everybody would live and behave. Professional codes of ethics attempt to employ the idea of the categorical imperative to create a shared ethical stance.

Codes of ethics are helpful precisely because of the slipperiness of ethics. Good codes of ethics offer a common language. And they serve to convert various ethical stances into deontological stances--written external collections of rules to follow. They make it easier for us to predict each others' ethical behavior.

So, a plea: when questions of ethics are up for grabs, let's be explicit about our own ethical stances and generous when trying to interpret other peoples' stances.

dreen
It is subjective, but you can argue that any device ever made with intention to kill or harm people had people involved who knew what they were doing and created it anyway. Everything from Gas chambers in concentration camps (made supposedly by Siemens or Bosch), bombs, guns, even swords.

And then there is Java EE...

Cyph0n
How about the extensive surveillance software developed by the NSA (PRISM et. al.)? That's probably the biggest example of ethical and moral failure in modern history.
collyw
And the one person who spoke out about it had to flee the country.
edanm
I think that most people disagree with you about that. So it's hard to call the people working on that immoral.
Cyph0n
I call them immoral because I think the work they are doing simply is immoral. Nothing "hard" about that.
threesixandnine
Why would software engineers be any different than other people? As in any field you will find the whole spectrum of humans...
ryandrake
I've always thought we could benefit from having a Hippocratic Oath for software developers and taking it seriously. The tough part would be agreeing what kinds of projects are and are not ethical to work on.

I personally have quit a job in the past because (among sever other important reasons) I felt the projects' primary application (surveillance) was not something I wanted to be associated with. I have also chosen not to voluntarily participate in the patent process for any software I've designed (foregoing those slimy patent "bonuses" other engineers seem to like to gobble up). But not every software developer shares my particular set of ethics. How would you come up with a definitive list of what does and does not violate the "Software Hippocratic Oath"?

gorbachev
My Comp.Sci. degree actually included one. All graduating students swore an oath to not use our skills for evil (paraphrased). It's been while since that happened, so I don't remember the specifics any more, but I do remember thinking what a great thing to pass onto new software engineers.
[deleted]
(comment deleted)
[deleted]
(comment deleted)
Lawtonfogle
Even the Hippocratic Oath is really hard to follow in corner cases? What counts as harm. What counts as an entity that can be harmed? Are all entities treated as equal in regards to the value of their harm?
J_Darnley
No, no, yes (I assume people from NK wrote it), no.

Add to that list everyone who works for a social network company or who works for an ad company.

drdeca
What about social networks that aren't ad supported ? ( e.g. donation funded )

So, do you think providing a social networking site is inherently unethical, or just information selling advert stuff?

I ask because I've been kinda working on an alternative to a particular one, and if you think that is unethical, I'd like to know why, in case your reasons why are convincing.

(because I don't want to do something unethical, unless not doing it would be morally worse than doing it.)

J_Darnley
These other social networks ("they really exist, trust me") probably don't try to actively spy on the entire world and follow every person across the entire internet. They probably don't try to hoard content into their silo and try to keep the users locked in. Clearly I am describing the big three here.

Now a not ad funded network may not do these things. That removes my largest complaints about them. It is these things which make them so grossly unethical.

As for your project I wish you luck. You will need it.

[deleted]
(comment deleted)
SuperKlaus
How about anybody and everybody working on surveillance software for the NSA?
dsacco
There are software engineers who follow some definition of ethics, and there are software engineers who ignore some definition of ethics - just like every other profession in the world.

What do you hope to achieve by finding real world examples? People do Bad Things for some definition of bad all the time, no matter their profession, but good luck trying to make any claim to absolute wrongdoing without the discussion devolving into semantics.

It will be very difficult for you to ascribe a moral position to any real world example because the real world isn't binary. To a first approximation, debates about ethics are usually won by vindicating the majority in-group's opinion about something they disagree with.

As you can tell, I'm basically saying this discussion isn't productive. You'll probably get some trendy answers like "NSA" or something to do with "surveillance" but I'll bite - how about the blackhat hackers who are employed by organized crime to de-anonymize "problem people" so those people can be found and "dealt with"? But that's just my opinion.

One of the biggest problems with this sort of thing is that no matter how powerfully you might believe someone working at e.g. the NSA is doing something evil, that individual likely feels just as powerfully that they are working to increase the net good in the world. In fact, they probably have an objectively coherent argument in favor of their position.

bcg1
Its nearly impossible to answer your question, because the examples you give all have an element of "its bigger than one person".

In all of those cases, it it is conceivable that programmers could easily justify the "larger purpose" in their own moral/ethical framework. This a problem inherent in all collective efforts.

"There's always the same amount of good luck and bad luck in the world. If one person doesn't get the bad luck, somebody else will have to get it in their place. There's always the same amount of good and evil, too. We can't eradicate evil, we can only evict it, force it to move across town. And when evil moves, some good always goes with it. But we can never alter the ratio of good to evil. All we can do is keep things stirred up so neither good nor evil solidifies. That's when things get scary. Life is like a stew, you have to stir it frequently, or all the scum rises to the top."[1]

[1]: http://www.goodreads.com/quotes/12020-there-s-always-the-sam...

bakhy
although i would personally consider all of the above unethical to some degree, i think i could understand someone thinking differently about Facebook Free Basics, Stingray, and Red Star. someone might put more weight to giving the poor any "internet", than giving them proper, neutral internet. Stingrays are used by legitimate law enforcement agencies, although i don't know much about how and under what conditions. and Red Star, well, all of that is legal there and not complying with the Kim can cost NK citizens their heads, so i'd cut a lot of slack there.

but investing your work in a company which sells surveillance equipment to dictators is downright disgusting. people are being tortured and killed because their governments managed to track them down thanks to that company. that seems like complete moral ambivalence at work. even if one could have somehow rationalized enabling their own government to spy, upon finding out who the company sells to, they must quit. and perhaps also leak everything.

Grue3
Let the one without sin cast the first stone.
AnimalMuppet
That means don't condemn others. It does not mean "don't bother cleaning up your own act". It doesn't even mean "don't make moral judgments about what other people do".
Grue3
No, that means everybody is a "sinner". It can be argued that anyone who writes closed-source software is unethical (see Stallman et al). But open source programmers are themselves unethical, being privileged enough to give away their work for free which drives down salaries of less fortunate coders. There's simply no programmer who can be described as completely ethical, and your helloworld.c is the Original Sin.
icebraining
Why is lowering the salaries of other coders unethical?
burner_1000
I heard a rumour (well people were talking in abstract terms about "the secret project" so I'd say 90%+ true) that a certain french owned software consultancy was building systems for the Saudi government to monitor homosexuals and journalists. Mass surveillance FTW!