"This was a couple of software engineers who put this in for whatever reason," - I'm not buying it. What would be their motive compared say to VW's motive? VW has a heck of a lot more motive for falsifying emissions than do "a couple of software engineers." Even if you were to argue the engineers were somehow fearful for their jobs that's still VW's problem because they're the ones who created that work environment instilling that fear.
I really expected VW to do better in handling this. Some business analysts are saying VW may be out of business within five years time. Based off this testimony I say good riddance.
They will blame some small man to avoid prosecution of higher staff. I bet this was done off the records - so the Software Engineer manager have no proof this was ordered or suggested by upper management. At worst they will get 1-2 engineers arrested while they will move quietly to other companies that will be willing to use their experience and connections.
Of course. Fall guy 101. Not some tinfoilness, but expedient politics of surviving corporate death pits.
The sort of shops where most every discussion is double top-secret: happens in-person, quietly, behind closed doors and the email trail is kept to strictly vague coordination details, at most, since it's subject to discovery. Oh and information is hoarded, because details create weaknesses.
Arrested? Remember that cars don't pollute, drivers do or something like that. At the end of the day, what the engineers in the trenches did was design a device that is illegal to sell in some other country. It is correct that the blame goes to a place higher up, where the decision to sell was made. Which is what is actually happening at Volkswagen proper. The surprising part is that the VW USA CEO seems to be completely out of the loop about this.
From what I know of the car industry, coding standards are very low. I don't think any car companies would look good if their code was audited by professional software engineers in other fields.
Then you don't know the motor industry very much as it has very strict coding standards like ISO 26262, the code has to be audited internally and by certified 3rd parties with every release to meet the development cycle requirements of the standards.
The industry also has specific coding standard for every language that is used in embedded systems like MISRA-C
https://en.wikipedia.org/wiki/MISRA_C
Today every thing you do with your car from using breaks to deploying air-bags is done through the computer, there's no way in hell that the motor industry would be releasing poor code "intentionally" because it would cost them billions in liabilities.
And you're not familiar with what happened when a 3rd party (Michael Barr) was allowed to FINALLY look inside Toyota's engine code and determine the cause of the "Unintended Acceleration" problem:
"Barr checked the [Toyota] source code against MISRA’s 2004 edition and found 81,514 violations."
The model year 2005 Camry in question from the trial would have been made available in 2004, and the electronic throttle control system was designed prior to 2004 (IIRC, the ETC design was carried forward from the previous model). How does it makes sense to apply MISRA-C:2004 in this case, especially when the Toyota ETC in question was admittedly not designed under MISRA-C:2004 nor prior MISRA-C revisions?
It is inaccurate to claim Michael Barr determined the cause of Toyota's unintended acceleration: he proposed a possible failure mode that was persuasive to a lay jury. His proposed failure mode didn't leave a DTC, so there's no way to actually know if his proposed failure mode actually happened in this case (or in any other).
I remember when this was posted on HN before, what I always wondered about is the discrepancy between the violations that NASA found and Barr.
NASA found 7000 and change Barr found over 81,000, I'm not sure whats the size of the code that was reviewed but that's allot, even if you count the fact that half of them might be silly like naming convention violations that still leaves quite an odd number for actual violations especially when considering that the code for an embedded system can't be that huge.
And the 10,000 global variables thing, well 10,000 sounds like a huge number but there wasn't any mention if they were a) necessary, b) implemented correctly, and c) out of how many variables in total? if the code they've tested has say 1 million variables then well....
And how they described global variables is also weird, they claimed that every software within the system can access them to me this sounds that your stereo or the rain sensor for the wipers can override data for the breaks, but that might be true only if everything is running within a single application.
If the breaks run as an independent application well then only every function within the break software (depending on the language and how global variables are implemented) could potentially access those variables.
To me pretty much that entire explanation seem to be constructed to sound worse than what it is, which is why I think that expert witnesses should only be allowed to be appointed and called by the court it self (as in by the judge so they and the jury could understand the technical details better) and not by either the defense or the prosecution.
> there wasn't any mention if they were a) necessary,
They were not. It is never necessary to have 10,000 global variables. If you have 10,000 global variables you are systematically Doing It Wrong. (That's not to say it's uncommon. From my experience I strongly suspect that it's a very common practice.) As one of the witnesses put it, quite correctly IMO:
"And in practice, five, ten, okay, fine. 10,000, no, we're done. It is not safe, and I don't need to see all 10,000 global variables to know that that is a problem,” Koopman testified.
Trying to justify this is like trying to justify not having a bug tracker. Which, BTW, they didn't.
> b) implemented correctly,
There's no way to check 10,000 global variables to see if they're implemented correctly, and that's the entire point.
> and c) out of how many variables in total?
That is completely irrelevant.
> they claimed that every software within the system can access them to me this sounds that your stereo or the rain sensor for the wipers can override data for the breaks,
This was the ECU, it only controls the engine, not the radio or the wipers. It has tasks for stuff like monitoring the engine and wheel speeds, the accelerator and brake pedal positions and controlling the fuel injection. And all of those tasks were only an extra header file and a typo away from stomping on one of the 10,000 global variables belonging to another task, but that isn't even the issue. The issue is that there is no way to trace the flow of data in the system because it's completely unstructured.
> but that might be true only if everything is running within a single application.
Welcome to the world of embedded RTOSs, where everything is running within a single application. It's not a PC. "Applications" aren't a thing. There's certainly no memory protection.
> Welcome to the world of embedded RTOSs, where
> everything is running within a single application.
> There's certainly no memory protection.
Generally, I agree, even though there have been quite a few chips already that have a primitive "Memory Protection Unit" that typically only distinguishes two processor states and you'd be able to protect the working data of e.g. the scheduler and your watchdog from the rest of the code running on your system. Which is much better than nothing!
But in the end the important consequence is: Having everything in one address space encourages people to violate the concepts that memory protection between tasks on your PC typically enforces, e.g. tasks reading from, and writing to other tasks' data structures without proper synchronization; instead of using proper IPC mechanisms like queues. Which is probably what these 10'000 global variables were used for.
And 2 of my friends worked for BetterPlace (RIP) and it took years for Nissan to accept the code which had to be rewritten to comply and certified by an external 3rd party.
And that code wasn't even "mission critical" in terms of road safety.
I have seen both very good code and very bad code. Standards are in place for making it good but it really depends the most on who's implementing it and the SW team culture.
And, supposing this is true, would you actually be surprised by this? This is a car company, not a CI and code-review obsessed software company in Silicon Valley.
> This is a car company, not a CI and code-review obsessed software company in Silicon Valley.
I read that as:
> This is a company whose products are high-velocity 1500kg chunks of steel zooming around in public, not a company that sells an intangible virtual good/service
---
Yes, I know software is historically poor with car companies, but we should still expect better. Let's not just lower our standards because of cynicism :-)
I found products made by car companies to be in general more reliable than those made by software companies (either in Silicon Valley or not).
I trust my life to my car (software included) every day. I wouldn't do that with any software of my smartphone, not even the OS.
Doesn't it blow your mind that there are no "standards of engineering" or whatever for software? There's no licensure body for software engineers who build software running your car, and therefore no accountability on a personal level.
When an engineer builds a bridge, she has to personally sign off on the bridge, saying it's safe, and is risking not only her professional career, but I think she can also be jailed and held criminally liable if the bridge kills people due to negligence.
It blows my mind, at least, that no such thing exists for software.
If you could have perfectly safe software, without requiring individual engineers to be licensed, would that be acceptable to you?
I'm not asking if you think such a thing would be possible or not - I'm asking if you would accept an alternate means of getting what I think we both want.
The big problem with all those standards, frankly, is that you've gotta pay money to actually evaluate them.
There aren't any guarantees that they'll be useful, that they'll match the modern development processes in your language, that they'll fit your problem domain, etc.
Those standards are there primarily to make the publisher a buck--not to represent the codified wisdom of up-to-date practitioners in a field.
Until we've got a truly open-source standard for people to code against, we should stop wringing our hands about these things.
Then you don't know the motor industry very much as it has very strict coding standards like ISO 26262, the code has to be audited internally and by certified 3rd parties with every release to meet the development cycle requirements of the standards.
http://www.ldra.com/en/software-quality-test-tools/group/by-....
The industry also has specific coding standard for every language that is used in embedded systems like MISRA-C https://en.wikipedia.org/wiki/MISRA_C
And a German one and from experience dealing with German companies software isn't regarded I the same light as what they would consider "proper engineering"
If that's the case, the execs are even more at fault for not having proper controls in place to prevent rogue employees from jeopardizing the entire company.
The code might even have been outsourced. I'm not trying to disparage outsourced developers, I know what I might do if the user story was something like "Needs to pass emissions when attached to test system."
It shouldn't matter. Another test process should have caught incorrect code. The car should be tested to the specs and the specs should reflect the regulations.
Remember the Toyota code, or at least the review, that was leaked a while back? I don't find it surprising at all that it would be possible to sneak through some stuff that no one knows what it does. And in that kind of environment, people generally don't mess with stuff they don't know what it does, cause they don't know what it'll break.
Exactly. The most devastating part of all of this for me (a US citizen) is that the German CEO resigns in disgrace and there are ongoing criminal investigations in Germany, but in the USA the CEO holds fast and blames engineers.
It seems like the German government is more competent that the US at enforcing laws against corporations, even when those laws are USA laws...
*EDIT: This comment is poorly worded. What I mean to say is, Germany is doing a better job at holding PEOPLE AT THE TOP responsible, whereas the USA is predictably allowing CxOs to plead ignorance and hide behind the corporate veil. Meanwhile in Germany, http://www.wsj.com/articles/german-prosecutors-open-investig...
You realize this is the exact opposite of what the rest of the world is saying, right? Is this the first article you've read on the matter? I've been following it with great interest, and many others have expressed shame that it's the US that is catching a European company cheating, and threatening prosecution.
This all happened because of the work of the West Virginia lab, plus CARB and EPA's investigation, and the EPA's refusal to issue an emissions certification to 2016 diesel VW passenger cars which has led to likely tens of thousands of un-sellable cars sitting on US dealer lots.
The German government has been shown to have looked the other way repeatedly and to be complicit in fighting on behalf of the automakers instead of the environment.
The European emissions and mileage tests have been shown to be an utter and complete sham.
The story you're peddling is really the exact opposite of reality. Take a look around at the other stories that have been posted on HN on the matter.
actually the european started this asking for help since they had the equipement for static test, but not the equipement to fool the defeating device into thinking it was actually on a road.
Good call. It wasn't European governments, and it wasn't an attempt to punish automakers for malfeasance, though.
The ICCT (an NGO) that appears to be based on Europe, commissioned the West Virginia study to try to figure out how more stringent US Diesel standards were being met, so they could encourage automakers to meet the same standards in Europe.
"In 2013, his organization [the ICCT] commissioned a study of VW diesels by West Virginia University after questions were raised about European diesel emissions standards and whether European vehicles were emitting too much nitrogen oxides linked to smog. Testers looked at three diesel cars for the U.S. market: a 2012 VW Jetta, a 2013 VW Passat and a BMW X5.
The group expected the cars they tested would perform better than those in Europe, because U.S. regulations are tougher. They were surprised the two Volkswagens had significantly higher-than-expected emissions, while the BMW performed well. They were so certain they had done something wrong that they tested the cars two more times with similar results."
This was an engineering problem. There may be an assumption that while there are about a dozen people in the US who work for VW in an engineering capacity, the US part of the group is in essence purely a marketing operation.
Exactly. I don't buy this one bit. If those engineers feared for their jobs because the cars wouldn't pass, I bet those engineers could have left VW and found better jobs. Nobody needs that kind of stress.
This is probably a little reactionary, but can we just stop for a second and admire the difference in American CEO's versus his European counterpart? The latter resigns immediately as a way of accepting he is ultimately responsible, the former points fingers. This is a blatant example of why the tide of opinion is turning against those with the most wealth.
The Staatsanwaltschaft Braunschweig is leading the criminal investigation and they said specifically that they are not after Martin Winterkorn, the press reports.
It's pretty close to the end. Fines will be levied against the Volkswagen company, not to him personally. His stock options will become worthless but he's still set for life. I think in this area, at least, China does it right. Shoot the guy as a warning to the others.
The cars were designed in Germany, weren't they? (If not, excuse my shock to find we've been subjected to misleading advertising.) I doubt the engineers reported to the USA guy.
Wow, that's amazing. I would never have thought it was sensible to make all those tiny changes from one market to another. I guess it's kind of like the changes they make between model years...
No way could it be just a couple of software engineers. The engine is a system, and its the engine that has to pass the emissions tests. So that means the whole engine development team had to be in on it, including their technical management. They in turn would not have done such a thing if they had not believed that senior management wanted it, and it is the responsibility of senior management to make it clear what they want.
When this CEO of a company with 600k employees says "it's some software engineers", I assume he might actually be honestly referring to what we would call engineering managers, perhaps with hundreds of indirect reports, 2-3 levels down from the CEO.
I find this plausible...
(Particularly since these hearings seem designed to score political points rather than to find out what actually happened.)
However plausible it may be, it is still not excusable.
The company should have processes in place that can verify the claims it makes. Plausible deniability is really just a claim of neglect.
Edit: I'm being rate-limited, so I'll respond to the points below right here:
>What processes should be in place, exactly, to do what?
How about driving a vehicle in the manner most of their customer's do and examining the emissions?
>Quarterly polygraph tests performed by all engineering managers
How would that help exactly? Polygraph is demonstrably unreliable.
>to assure top management they haven't broken any laws?
It's mostly irrelevant how assured top management is about anything. What matters, in this case, is how far out of spec the vehicles Volkswagon produced are. That is what needed to be verified: specs vs. facts.
Let's assume the individial he is talking about is a manager of a group of hundreds of engineers working in this area. This means this guy's direct manager most likely would not be in a position to inspect the work of people 2-3 levels down for any irregularities.
What processes should be in place, exactly, to do what? Quarterly polygraph tests performed by all engineering managers to assure top management they haven't broken any laws?
At the end of the day, that manager is still responsible for everything that happens under them. That's the justification for giving them the big paychecks, isn't it?
All you need are strong whistleblowing protections and mandatory independent investigation of any claims made.
Do you really think of the dozens of people who had to be aware of it none of them would have thought "this is really dodgy" and wanted to push it up the chain?
The (now resigned) CEO prided himself on being an engineer and in depth knowledge. I don't believe he didn't know how VW was able to market clean diesel engines in the US without having a clean diesel emission system for them.
His big thing was to be the largest car maker and he needed to juice US sales to get there. He had to have spent a lot of time on US strategy. The shitty part is that he succeeded--they became #1 (this year!) and he has a huge pension to retire on.
"Winterkorn’s professional career began in 1977 as a specialist assistant in the research division "Process Engineering" at Robert Bosch GmbH. From 1978 to 1981 he headed the refrigerant compressor development group "Substances and Processes" at Robert Bosch GmbH and Bosch-Siemens-Hausgeräte GmbH.
In 1981 Winterkorn joined AUDI AG as assistant to the Member of the Board for Quality Assurance. Two years later, he assumed responsibility for "Measuring Technology/Sampling and Test Laboratory" at Audi. At the beginning of 1988, he was made departmental head of "Central Quality Assurance", and in 1990 Head of Audi Quality Assurance.
In 1993 Winterkorn became Head of "Group Quality Assurance" at Volkswagen AG and was appointed General Representative of Volkswagen AG in March 1994."
These would be his formative years. It seems he focused on processes and quality rather than actual technology, IMO.
fact is the whole report chain is rotten. who wrote the code, who told him so, and indirectly most engineers that had a reasonable knowledge to be in a position of suspecting a foul play.
you can't get this running for 6 years without widespread cooperation, or at least willfully turning a blind eye.
those engines ran loads of miles on benches before being put into cars, and it's a large group running multiple cross tests on many parameters including fuel consumption and efficiency and operating temperature, which should give anyone within the sector a good rough idea of how an engine performs emission wise.
Just because VW might have more motive to do this doesn't automatically mean that they're lying when they say it was the fault of individual engineers. Guilt is not determined by "who has the most motive", it's determined by who actually did it.
Now it may very well be true that VW is lying and it was a corporate decision. But it's not completely implausible that this really was the decision of a few individual employees.
Other commenters here are expressing surprise that engineers could "slip through arbitrary code" to do this. But isn't that kind of the job of software engineers, to write code to do things? I would hope they have code review going on at VW, but code review doesn't always catch everything, or maybe several engineers who were working together would review each other's code.
Like NASA, auto companies like VW produce a TON of documentation on all phases of design and engineering. This is especially true when it comes to regulation-sensitive processes like emissions testing.
As someone who has worked in a comparably highly regulated industry (pharma) I can state with full confidence that Mr Horn is a lying bastard.
> Like NASA, auto companies like VW produce a TON of documentation on all phases of design and engineering. This is especially true when it comes to regulation-sensitive processes like emissions testing.
Bingo! A lot of HN is acting like "it could slip through the processes we have at work so I can see how it could happen here" without realising that the processes, documentation, and independent review requirements are 100x more involved and thorough for critical car firmware than they are for a basic software shop.
Yes and no. Remember the Toyota firmware fracas and the bizarre coding practices (100s of global variables anyone?) that were eventually revealed. Adding heavyweight process doesn't necessarily solve more problems than it creates.
I'm not sure emissions controls are considered "critical firmware" in the same sense as systems that could actually kill the occupants of the vehicle if they fail. Or rather, I hope they aren't.
"But it's not completely implausible that this really was the decision of a few individual employees."
Oh, I fully believe it was the decision of a few individual employees. I just don't believe those individuals were the actual coders. Coders will code to the spec they are given; problems are far more likely to lie in the spec layer.
It is possible it will turn out the coders should have known better. But it isn't even that hard for me to create plausible circumstances in which the coders didn't have any reason to believe they were doing anything wrong. Remember, we're only seeing the end-results, but it's not hard to imagine a combination of specifications where the programmers are assured each little step is legal and OK, and a combination of other optimizations combining together to produce this result, with no one person quite understanding the interactions.
And just to be clear, I work a lot in security, on the defense side. I'm familiar with taking the possibility that coders will be actively hostile seriously. But I'm also very familiar with seeing five intelligent people each acting according to their own best info and with their own best interests at heart interacting to produce blithering stupidity. It's not always as simple as it looks when you're just looking at the outcome.
Right. And those tests and regulations exist in the interest of public health, not product benchmarking. So the issue is bigger than lying about some performance metric to compare more favorable to competitors. And it's also not a domain that's auxiliary—this is engine technology, the very heart of the company and the product. And good god, they built an entire brand campaign around it.
I really do feel this is analogous to some large pharmaceutical finally admitting that they cheated clinical testing on an important drug. You can't just pin that back to a couple of "rouge scientists".
Even if you look at it from a corporate politics standpoint, a software engineer will much sooner say "car's not passing emissions? Not my problem -- take it back to engineering".
VW will not be out of business in 5 year. Nobody will remember this in 5 years. All the major automakers have manufacturing scandals in their past that resulted in actual deaths, not just fudging some arbitrary emissions standard. They are almost all still in business.
Probably, possibly, statistically contributing -- yeah maybe. But other automakers have made decisions that directly, indisputably, really killed people. Usually to save a couple of dollars on a part.
"Some people have made the wrong decisions in order to get away with something"
Some people; he doesn't actually say it was the programmers. Both statements are perfectly consistent with the cheating being orchestrated from the top. This looks like a very careful attempt to mislead without actually risking a visit to Club Fed.
I've always wondered how something like this plays out legally: Can the engineers be somehow held accountable for writing something they were forced / asked to by their managers?
This isn't a direct comparison, but after World War II it was decided clearly that soldiers are not protected by saying "my boss told me to do it" when the commit war crimes or other atrocities. This is now called the Nuremberg defense and it holds no water at all.
The same principle applies here, in a general sense - but it's sufficiently different in practice, I wonder how it would be treated too.
Engineers aren't solely responsible, but... Did you software guys not take engineering ethics classes? You have a responsibility to society to consider health and safety of the public, even if it means losing your job.
We "software guys" aren't required to do anything at all aside from know how to make software. I can't even imagine how to change that, the tools to learn to make software are federated widely and universities have nothing like a monopoly on them.
Most 'software guys' have academic backgrounds in computer science, which typically doesn't include anything like engineering ethics. An outright 'Software Engineering' degree might include something like that, but it's definitely emphasized far less than it would be in something like mechanical or civil engineering.
My school had "Social Implications of Computing" but it was more focused on research ethics, reference citing, IP, privacy, and security type issues.
Also, since we are CS and not Professional Engineers, we did not have any ethical ceremony like Order of the Engineer.
Some type of Professional Ethics course is part of the ACM model curriculum for computer science (https://www.acm.org/education/curricula-recommendations), but I think how that's fulfilled varies widely across universities.
This is kind of tricky, as I'm assuming that, unlike say a hard engineer that works on a Power Plant or a Bridge, they're not licensed as a Professional Engineer.
In that case the organization would be responsible to a degree, but so would engineers that sign off on something like this. The coder shouldn't solely be responsible, but if you sniff something dirty in the water imo it's your moral responsibility to raise something, or resign. I hope that if I'm in a similar position (probably won't be because I don't work on anything that will kill a person if poorly engineered) that I would react appropriately.
That said, this guy is probably full of crap and is just throwing whoever he can under the bus to preserve whatever stock value is left.
By that reasoning we can order soldiers to torture if we simply fragment the process enough. One to strap in the left wrist, one to strap in the right wrist, one to switch on the gas to the blowtorch, etc.
Yes, they can. Most engineers (in the US) have to take an ethics course that covers this. Just because your boss says do something, doesn't give you a pass.
Definitely had to take an ethics course when I went to Waterloo (in Ontario, Canada). I believe it's a requirement for a university's program to receive Engineering accreditation. At any rate, all engineers took it.
My job title is "Software Engineer," but I have a Computer Science degree. I use the title because that's what people call it, but it's not an engineering discipline. I had to take some ethics-related course work as part of my general education requirements, but this mostly involved reading some philosophy that vaguely related to ethics.
You don't have to have a software engineering degree or an EIT/PE to be a "software engineer". I know plenty of people who have vaguely related degrees like mathematics or statistics that never had to take such courses. And of course there are many autodidacts in the field who studied god knows what.
These are worrying times for VW if the new CEO to say something like this. Many VW engineers will be considering leaving if VW is going find a couple of engineers as scapegoats, and not be back them up.
Presumably there are a couple of engineers who deserve some of the blame, but wow, what better way to say "Our culture is poison". Both because they're throwing independent contributors under the bus, and because he's saying the culture is such that the engineers would think it's OK.
To be fair, I think he might not have meant it to come out that way.
EDIT: Looking again at the article, it seems like the US CEO isn't even aware of this: "The German automaker has suspended 10 senior managers, including three top engineers, as part of its internal investigation." So VW doesn't necessarily have a poisonous culture. But they do have a US CEO who doesn't prepare when he testifies to Congress.
Unfortunately, we may never know the truth in the scenario. The way that automotive world works with the multi-tier supplier chain, anyone who is not close to the source, will never know where the real blame is.
Behind the scenes in VW's HQ in Wolfsburg, it must be chaos.
I'm assuming the development process for embedded auto software like this would be pretty rigorous, meaning it would be fairly difficult to sneak in code like this.
I also assume that testing of emissions software like this would involve a lot of lab/road testing with the vehicles against the EPA test profile and therefore a lot of people would be involved. Suggesting this is the stealthy work of just a couple of software engineers is very hard to believe.
Can anyone who's worked on embedded software in the automotive environment comment?
The investigations regarding Toyotas(?) unexplained/unintended acceleration suggest otherwise. The code was apparently pretty bad, halfheartedly reviewed and just unmaintainable. While there the software seemed to have (a huge number of) errors put in by mistake, of course most independent reviews should also spot intentional mischief.
I don't claim that VW had comparably crappy code, but "industry practice" likely wouldn't have caught it either way.
Even if this is true, and I doubt that it is, it's not really that important. The real question is why people in positions of responsibility didn't prevent it from happening.
"On tonight's episode of Punked: watch as we add a second floor and backyard porch to this man's bungalow while he's away for the weekend! You won't believe his reaction!"
How is Volkswagen's US head, who lives and works in the US, able to issue proclamations with such certainty about the non-existence of private conversations and directions taking place among the engineering staff and management in Germany, where he spends almost no time?
It is as if a US diplomat stationed in New Zealand used his authority to declare that Bush had never authorized any torture, and it was rogue individual interrogators who did it. How would he know given he was on the other side of the world when the conversations would have taken place.
I would assume he is speaking on behalf of Volkswagen, not testifying as an individual on his own experiences. Volkswagen's current story is that it was a couple rogue engineers, so he is conveying that. Congress isn't going to get the CEO in to testify, so it's reasonable to assume he is speaking as a proxy.
Exactly. If Mr Horn is certain of what he says, where's the proof? To make such allegations, any court would require that VW produce historical evidence of this misbehavior. Yet he offers none.
In the absence of such evidence, the responsibility for misdeed falls to the corporation's most senior executives -- for failing to create and maintain a process that should have captured and revealed such misdeeds, thereby PRODUCING THE MISSING AUDIT TRAIL.
This ridiculous story is not going to hold and they know it so watch for higher ups trying to divest. I mean Bosch told them it would be illegal to use the way they were intending. I surely hope someone is watching who at the company is selling shares.
How could some engineers cheat the entire VW corporation?
Germans are known for their ridiculously meticulous standards and testing. VW should have many in-house and external-independent labs performing tests.
The engineers who write the ECU code have a great distance from the mechanics who test every engine and car batch from their production line.
How did they never tested and engine entirely; power, emissions, consumption?
You may fool the regulators but you can't fool your own mechanics.
What a classic terrible management move. I worked at a ski shop in college that also sold (really bad) used ski equipment on eBay. One time the lead manager accidentally let a lot of stock go live with a $1 minimum. They realized it and went back in under different accounts and bid them all up. When eBay caught them, they blamed the employees for it (myself and 4 others), who all had nothing to do with it.
We all quit the next day. Who wants to work for someone like that?
"This was a couple of software engineers who put this in for whatever reason"
This was a system. A complex one. No way it was done by a couple of guys at 2AM, or whatever. This has to have been inter-group collusion, and management is in CYA mode and shoving individuals under the bus.
VW has gone from being a cravenly stupid company that is going to be fined billions of dollars, to a place where I suggest it is in every engineer's best interest to resign, and the earliest opportunity they can manage.
Say the detection for the emissions testing is something like a flag in the software that says the car is on a dyno, or some other factors that the ECU has legitimate reasons to keep track of.
In that case, the defeat system is just a quick if else predicated on those state variables and swapping between a set of engine parameters that is emissions test safe and another set that provides better performance.
I don't mean to insist that this must be the case, I'm just trying to illustrate that the cheat could happen at a level where much of the complexity has already been abstracted away.
Okay, imagine you're working on the hardware of the engine. Pistons, bolts, lubrication. One Friday afternoon you leave work depressed because the emissions suck, they've sucked for months, and they won't get better without a major redesign.
On Monday morning, the emissions tests are suddenly passing.
"Ha, just one of those things," you say, "I thought we had emissions problems, but I guess we don't after all. Let's ship the engine and have a party!"
I'm not arguing that it was a few people working in secret, just that it would be possible for the changes to be done at a high level (in the engine system), without pulling in all the people that were involved in developing the vehicle (or even the powertrain).
Expect VW US to play dumb until something out of Germany is officially done. VW wont go out of business, but they are 10 years from growth in the US and probably most other markets. I think the real fallout will come if the prestige brands are made to look bad. Audi and Porsche have much more to lose, and those are the profit centers for VW.
In my country Audi says the cars using the engine are "technically safe to use and drive". I did report that to the consumer oversight board. I believe it's false marketing, the cars are a danger to pedestrians and cyclists outside the car.
I believe it's lies upon lies from the upper management seems like a corporate culture going wrong.
There's clear evidence that many cities throughout Europe have too high air pollution, and that it kills close to what smoking does. Cities have been ordered by the EU to clean up by the end of the year, or face harsh fines. Over 80,000 people die in the UK per year (smoking kills 100,000). Mostly from diesel engine pollution.
Now try to think how many people it kills in the rest of Europe, the USA and the world?
Cars (and buses) stink, are noisy, and worse -- they kill on a massive scale. There is alternative tech available for order now that doesn't kill as much, and provides better service.
Demand that the killing machines are replaced, and the makers, and the users are brought to justice if they don't!
That's evidence of pollution in general being a problem, which isn't even in the same ballpark as establishing culpability for any amount of deaths for an extra amount of one kind of pollutant by one carmaker over a fixed time period.
Correct me if I'm wrong, but that article talks about particulate emissions, something which even the cheating VWs don't have a problem with - the datapoint that led to this whole thing coming to light were high NOx emissions, not particulates.
This pass the buck attitude really pisses me off. You're the head of the company, department, whatever, it's your gawd damn responsibility if your subordinates do something wrong. If you hired bad people it's still your responsibility. If you hired people that hired people it's still your responsibility. I don't care if you knew or not. You set the tone, you set the impossible deadlines or requirements.
You fostered a culture where someone thought that would be okay to do. Own up!
He was very old or simply lying. Very first thing University of Washington Center for Information Assurance and Cybersecurity (accredited by U.S. Department of Homeland Security, whatever that means) teaches you about becoming a CIO is precisely delegating responsibility :)
258 comments
[ 3.0 ms ] story [ 257 ms ] threadI really expected VW to do better in handling this. Some business analysts are saying VW may be out of business within five years time. Based off this testimony I say good riddance.
The sort of shops where most every discussion is double top-secret: happens in-person, quietly, behind closed doors and the email trail is kept to strictly vague coordination details, at most, since it's subject to discovery. Oh and information is hoarded, because details create weaknesses.
http://www.ldra.com/en/software-quality-test-tools/group/by-...
The industry also has specific coding standard for every language that is used in embedded systems like MISRA-C https://en.wikipedia.org/wiki/MISRA_C
Today every thing you do with your car from using breaks to deploying air-bags is done through the computer, there's no way in hell that the motor industry would be releasing poor code "intentionally" because it would cost them billions in liabilities.
"Barr checked the [Toyota] source code against MISRA’s 2004 edition and found 81,514 violations."
http://www.safetyresearch.net/blog/articles/toyota-unintende...
It is inaccurate to claim Michael Barr determined the cause of Toyota's unintended acceleration: he proposed a possible failure mode that was persuasive to a lay jury. His proposed failure mode didn't leave a DTC, so there's no way to actually know if his proposed failure mode actually happened in this case (or in any other).
NASA found 7000 and change Barr found over 81,000, I'm not sure whats the size of the code that was reviewed but that's allot, even if you count the fact that half of them might be silly like naming convention violations that still leaves quite an odd number for actual violations especially when considering that the code for an embedded system can't be that huge.
And the 10,000 global variables thing, well 10,000 sounds like a huge number but there wasn't any mention if they were a) necessary, b) implemented correctly, and c) out of how many variables in total? if the code they've tested has say 1 million variables then well.... And how they described global variables is also weird, they claimed that every software within the system can access them to me this sounds that your stereo or the rain sensor for the wipers can override data for the breaks, but that might be true only if everything is running within a single application.
If the breaks run as an independent application well then only every function within the break software (depending on the language and how global variables are implemented) could potentially access those variables.
To me pretty much that entire explanation seem to be constructed to sound worse than what it is, which is why I think that expert witnesses should only be allowed to be appointed and called by the court it self (as in by the judge so they and the jury could understand the technical details better) and not by either the defense or the prosecution.
They were not. It is never necessary to have 10,000 global variables. If you have 10,000 global variables you are systematically Doing It Wrong. (That's not to say it's uncommon. From my experience I strongly suspect that it's a very common practice.) As one of the witnesses put it, quite correctly IMO:
"And in practice, five, ten, okay, fine. 10,000, no, we're done. It is not safe, and I don't need to see all 10,000 global variables to know that that is a problem,” Koopman testified.
Trying to justify this is like trying to justify not having a bug tracker. Which, BTW, they didn't.
> b) implemented correctly,
There's no way to check 10,000 global variables to see if they're implemented correctly, and that's the entire point.
> and c) out of how many variables in total?
That is completely irrelevant.
> they claimed that every software within the system can access them to me this sounds that your stereo or the rain sensor for the wipers can override data for the breaks,
This was the ECU, it only controls the engine, not the radio or the wipers. It has tasks for stuff like monitoring the engine and wheel speeds, the accelerator and brake pedal positions and controlling the fuel injection. And all of those tasks were only an extra header file and a typo away from stomping on one of the 10,000 global variables belonging to another task, but that isn't even the issue. The issue is that there is no way to trace the flow of data in the system because it's completely unstructured.
> but that might be true only if everything is running within a single application.
Welcome to the world of embedded RTOSs, where everything is running within a single application. It's not a PC. "Applications" aren't a thing. There's certainly no memory protection.
But in the end the important consequence is: Having everything in one address space encourages people to violate the concepts that memory protection between tasks on your PC typically enforces, e.g. tasks reading from, and writing to other tasks' data structures without proper synchronization; instead of using proper IPC mechanisms like queues. Which is probably what these 10'000 global variables were used for.
Dav3xor's Law -- Code Quality is Inversely Proportional to Risk.
Anecdotal evidence is anecdotal.
I read that as:
> This is a company whose products are high-velocity 1500kg chunks of steel zooming around in public, not a company that sells an intangible virtual good/service
---
Yes, I know software is historically poor with car companies, but we should still expect better. Let's not just lower our standards because of cynicism :-)
When an engineer builds a bridge, she has to personally sign off on the bridge, saying it's safe, and is risking not only her professional career, but I think she can also be jailed and held criminally liable if the bridge kills people due to negligence.
It blows my mind, at least, that no such thing exists for software.
I'm not asking if you think such a thing would be possible or not - I'm asking if you would accept an alternate means of getting what I think we both want.
- https://en.wikipedia.org/wiki/ISO_26262
- https://en.wikipedia.org/wiki/IEC_62304
- https://en.wikipedia.org/wiki/MISRA_C
There aren't any guarantees that they'll be useful, that they'll match the modern development processes in your language, that they'll fit your problem domain, etc.
Those standards are there primarily to make the publisher a buck--not to represent the codified wisdom of up-to-date practitioners in a field.
Until we've got a truly open-source standard for people to code against, we should stop wringing our hands about these things.
http://www.ldra.com/en/software-quality-test-tools/group/by-...
https://en.wikipedia.org/wiki/Motor_Industry_Software_Reliab...
https://en.wikipedia.org/wiki/MISRA_C
https://en.wikipedia.org/wiki/ISO_26262
OUR LORD AND SAVIOR UNCLE BOB MARTIN SEZ WE NEED STANDARDS BEFORE THE GUBBERMINT MAKES US DO IT.
FOR THE LOVE OF <deity> DO NOT CONTINUE POINTING OUT THAT STANDARDS HAVE NECESSARILY SPRUNG UP IN SEGMENTS OF THE INDUSTRY THAT INVOLVE RISK.
UNCLE BOB WILL FIND YOU... UNCLE BOB KNOWS ALL, SEES ALL... PUNISHES ALL.
/end sarcasm
ISO, IEC, etc.
> There's no licensure body for software engineers who build software running your car, and therefore no accountability on a personal level.
MISRA, SCSC, etc.
If you use software to build a bridge, bridge standards should apply. If you use software to build a car, car standards should apply.
If you use software to build a fart app, fart app standards should apply. (Which frankly, don't have to be very high.)
Then you don't know the motor industry very much as it has very strict coding standards like ISO 26262, the code has to be audited internally and by certified 3rd parties with every release to meet the development cycle requirements of the standards. http://www.ldra.com/en/software-quality-test-tools/group/by-.... The industry also has specific coding standard for every language that is used in embedded systems like MISRA-C https://en.wikipedia.org/wiki/MISRA_C
http://www.detroitnews.com/story/business/autos/foreign/2015...
Ignore whatever comes out of the VW USA guy. The investigations in Germany will most likely be more fruitful especially because police is involved.
It seems like the German government is more competent that the US at enforcing laws against corporations, even when those laws are USA laws...
*EDIT: This comment is poorly worded. What I mean to say is, Germany is doing a better job at holding PEOPLE AT THE TOP responsible, whereas the USA is predictably allowing CxOs to plead ignorance and hide behind the corporate veil. Meanwhile in Germany, http://www.wsj.com/articles/german-prosecutors-open-investig...
This all happened because of the work of the West Virginia lab, plus CARB and EPA's investigation, and the EPA's refusal to issue an emissions certification to 2016 diesel VW passenger cars which has led to likely tens of thousands of un-sellable cars sitting on US dealer lots.
The German government has been shown to have looked the other way repeatedly and to be complicit in fighting on behalf of the automakers instead of the environment.
The European emissions and mileage tests have been shown to be an utter and complete sham.
The story you're peddling is really the exact opposite of reality. Take a look around at the other stories that have been posted on HN on the matter.
The ICCT (an NGO) that appears to be based on Europe, commissioned the West Virginia study to try to figure out how more stringent US Diesel standards were being met, so they could encourage automakers to meet the same standards in Europe.
http://www.detroitnews.com/story/business/autos/foreign/2015...
"In 2013, his organization [the ICCT] commissioned a study of VW diesels by West Virginia University after questions were raised about European diesel emissions standards and whether European vehicles were emitting too much nitrogen oxides linked to smog. Testers looked at three diesel cars for the U.S. market: a 2012 VW Jetta, a 2013 VW Passat and a BMW X5. The group expected the cars they tested would perform better than those in Europe, because U.S. regulations are tougher. They were surprised the two Volkswagens had significantly higher-than-expected emissions, while the BMW performed well. They were so certain they had done something wrong that they tested the cars two more times with similar results."
I think it is cowardice to blame them.
The cars sold in the US have nothing in common with the European models but the name and engine.
The US models have different sizes, different engine selections, are fully assembled in the US, etc.
Some example images:
Interior: http://www.myturbodiesel.com/images/b7/tdi/2012-vw-passat-eu...
Exterior: http://www.myturbodiesel.com/images/b7/tdi/euro-passat-diffe...
Seats: http://www.myturbodiesel.com/images/b7/tdi/interior-differen...
And in this 2011 discussion there are already discussions about how VW is able to handle NOx without AdBlue in the US variation of the car: http://www.myturbodiesel.com/threads/differences-between-201...
But if they could just go out and get jobs, why would they fear for their jobs in the first place?
I find this plausible...
(Particularly since these hearings seem designed to score political points rather than to find out what actually happened.)
The company should have processes in place that can verify the claims it makes. Plausible deniability is really just a claim of neglect.
Edit: I'm being rate-limited, so I'll respond to the points below right here:
>What processes should be in place, exactly, to do what?
How about driving a vehicle in the manner most of their customer's do and examining the emissions?
>Quarterly polygraph tests performed by all engineering managers
How would that help exactly? Polygraph is demonstrably unreliable.
>to assure top management they haven't broken any laws?
It's mostly irrelevant how assured top management is about anything. What matters, in this case, is how far out of spec the vehicles Volkswagon produced are. That is what needed to be verified: specs vs. facts.
Let's assume the individial he is talking about is a manager of a group of hundreds of engineers working in this area. This means this guy's direct manager most likely would not be in a position to inspect the work of people 2-3 levels down for any irregularities.
What processes should be in place, exactly, to do what? Quarterly polygraph tests performed by all engineering managers to assure top management they haven't broken any laws?
All you need are strong whistleblowing protections and mandatory independent investigation of any claims made.
Do you really think of the dozens of people who had to be aware of it none of them would have thought "this is really dodgy" and wanted to push it up the chain?
His big thing was to be the largest car maker and he needed to juice US sales to get there. He had to have spent a lot of time on US strategy. The shitty part is that he succeeded--they became #1 (this year!) and he has a huge pension to retire on.
"Winterkorn’s professional career began in 1977 as a specialist assistant in the research division "Process Engineering" at Robert Bosch GmbH. From 1978 to 1981 he headed the refrigerant compressor development group "Substances and Processes" at Robert Bosch GmbH and Bosch-Siemens-Hausgeräte GmbH.
In 1981 Winterkorn joined AUDI AG as assistant to the Member of the Board for Quality Assurance. Two years later, he assumed responsibility for "Measuring Technology/Sampling and Test Laboratory" at Audi. At the beginning of 1988, he was made departmental head of "Central Quality Assurance", and in 1990 Head of Audi Quality Assurance.
In 1993 Winterkorn became Head of "Group Quality Assurance" at Volkswagen AG and was appointed General Representative of Volkswagen AG in March 1994."
These would be his formative years. It seems he focused on processes and quality rather than actual technology, IMO.
He was also known as a micromanger, I just don't see how he didn't somehow figure out that his entire clean diesel strategy was bullshit.
you can't get this running for 6 years without widespread cooperation, or at least willfully turning a blind eye.
those engines ran loads of miles on benches before being put into cars, and it's a large group running multiple cross tests on many parameters including fuel consumption and efficiency and operating temperature, which should give anyone within the sector a good rough idea of how an engine performs emission wise.
Now it may very well be true that VW is lying and it was a corporate decision. But it's not completely implausible that this really was the decision of a few individual employees.
Other commenters here are expressing surprise that engineers could "slip through arbitrary code" to do this. But isn't that kind of the job of software engineers, to write code to do things? I would hope they have code review going on at VW, but code review doesn't always catch everything, or maybe several engineers who were working together would review each other's code.
As someone who has worked in a comparably highly regulated industry (pharma) I can state with full confidence that Mr Horn is a lying bastard.
Bingo! A lot of HN is acting like "it could slip through the processes we have at work so I can see how it could happen here" without realising that the processes, documentation, and independent review requirements are 100x more involved and thorough for critical car firmware than they are for a basic software shop.
Oh, I fully believe it was the decision of a few individual employees. I just don't believe those individuals were the actual coders. Coders will code to the spec they are given; problems are far more likely to lie in the spec layer.
It is possible it will turn out the coders should have known better. But it isn't even that hard for me to create plausible circumstances in which the coders didn't have any reason to believe they were doing anything wrong. Remember, we're only seeing the end-results, but it's not hard to imagine a combination of specifications where the programmers are assured each little step is legal and OK, and a combination of other optimizations combining together to produce this result, with no one person quite understanding the interactions.
And just to be clear, I work a lot in security, on the defense side. I'm familiar with taking the possibility that coders will be actively hostile seriously. But I'm also very familiar with seeing five intelligent people each acting according to their own best info and with their own best interests at heart interacting to produce blithering stupidity. It's not always as simple as it looks when you're just looking at the outcome.
> The German automaker has suspended 10 senior managers, including three top engineers, as part of its internal investigation.
So it could be a few engineers and their managers that colluded together on this.
I really do feel this is analogous to some large pharmaceutical finally admitting that they cheated clinical testing on an important drug. You can't just pin that back to a couple of "rouge scientists".
Except when obviously true, companies that make public statements like this will have difficulty recruiting the best engineers.
"I'm sure there was no pressure on individual performers to make the results look favorable for VW."
Some people; he doesn't actually say it was the programmers. Both statements are perfectly consistent with the cheating being orchestrated from the top. This looks like a very careful attempt to mislead without actually risking a visit to Club Fed.
I'm guessing the reason was because their boss told them.
The same principle applies here, in a general sense - but it's sufficiently different in practice, I wonder how it would be treated too.
When a soldier is ordered to torture - its clearly wrong to do so. Its a question of human ethic.
Is it the responsibility of the software engineer to know the exact level of NO_2 0.001 or 0002 that he should be coded into their product ?
Also a software engineer doesn't exist in a vacuum. There is a whole testing regime. Why should the coder be solely responsible ?
Also, since we are CS and not Professional Engineers, we did not have any ethical ceremony like Order of the Engineer.
Some type of Professional Ethics course is part of the ACM model curriculum for computer science (https://www.acm.org/education/curricula-recommendations), but I think how that's fulfilled varies widely across universities.
http://www.nspe.org/resources/licensure/what-pe
In that case the organization would be responsible to a degree, but so would engineers that sign off on something like this. The coder shouldn't solely be responsible, but if you sniff something dirty in the water imo it's your moral responsibility to raise something, or resign. I hope that if I'm in a similar position (probably won't be because I don't work on anything that will kill a person if poorly engineered) that I would react appropriately.
That said, this guy is probably full of crap and is just throwing whoever he can under the bus to preserve whatever stock value is left.
To be fair, I think he might not have meant it to come out that way.
EDIT: Looking again at the article, it seems like the US CEO isn't even aware of this: "The German automaker has suspended 10 senior managers, including three top engineers, as part of its internal investigation." So VW doesn't necessarily have a poisonous culture. But they do have a US CEO who doesn't prepare when he testifies to Congress.
Behind the scenes in VW's HQ in Wolfsburg, it must be chaos.
I also assume that testing of emissions software like this would involve a lot of lab/road testing with the vehicles against the EPA test profile and therefore a lot of people would be involved. Suggesting this is the stealthy work of just a couple of software engineers is very hard to believe.
Can anyone who's worked on embedded software in the automotive environment comment?
I don't claim that VW had comparably crappy code, but "industry practice" likely wouldn't have caught it either way.
http://www.safetyresearch.net/blog/articles/toyota-unintende...
https://en.m.wikipedia.org/wiki/V-Model
It is as if a US diplomat stationed in New Zealand used his authority to declare that Bush had never authorized any torture, and it was rogue individual interrogators who did it. How would he know given he was on the other side of the world when the conversations would have taken place.
In the absence of such evidence, the responsibility for misdeed falls to the corporation's most senior executives -- for failing to create and maintain a process that should have captured and revealed such misdeeds, thereby PRODUCING THE MISSING AUDIT TRAIL.
Blame others without evidence ==> cover up.
The ones fired would have been the mechanical engineers that designed the engine, aren't they?
Germans are known for their ridiculously meticulous standards and testing. VW should have many in-house and external-independent labs performing tests.
The engineers who write the ECU code have a great distance from the mechanics who test every engine and car batch from their production line.
How did they never tested and engine entirely; power, emissions, consumption?
You may fool the regulators but you can't fool your own mechanics.
We all quit the next day. Who wants to work for someone like that?
This was a system. A complex one. No way it was done by a couple of guys at 2AM, or whatever. This has to have been inter-group collusion, and management is in CYA mode and shoving individuals under the bus.
VW has gone from being a cravenly stupid company that is going to be fined billions of dollars, to a place where I suggest it is in every engineer's best interest to resign, and the earliest opportunity they can manage.
In that case, the defeat system is just a quick if else predicated on those state variables and swapping between a set of engine parameters that is emissions test safe and another set that provides better performance.
I don't mean to insist that this must be the case, I'm just trying to illustrate that the cheat could happen at a level where much of the complexity has already been abstracted away.
On Monday morning, the emissions tests are suddenly passing.
"Ha, just one of those things," you say, "I thought we had emissions problems, but I guess we don't after all. Let's ship the engine and have a party!"
There is no way that this happened.
I believe it's lies upon lies from the upper management seems like a corporate culture going wrong.
Now try to think how many people it kills in the rest of Europe, the USA and the world?
Cars (and buses) stink, are noisy, and worse -- they kill on a massive scale. There is alternative tech available for order now that doesn't kill as much, and provides better service.
Demand that the killing machines are replaced, and the makers, and the users are brought to justice if they don't!
http://www.theguardian.com/environment/2015/jul/15/nearly-95...
http://www.independent.co.uk/news/world/europe/volkswagen-em...
#enron #pinto #oilspill