Of course people do mistakes and fail to follow procedures. That is why the company should have processes to catch dangerous mistakes and omissions in place. If one low level guy can cause such huge problems, then there is something very wrong with the way whole process works.
I am a SW engineer currently working in a regulated industry, and there is no procedure in place for me to fix something quickly behind everybody's back. The code should be at least reviewed, integrated and tested by 3 independent people. And the issue and the resolution action would have been reviewed/signed off at the system level as well. So it would be very unlikely that I could hide something for years without suspicion.
This is in a way cumbersome and annoying but it is understandable why it is put in place this way.
I used to work in consumer electronics, where I could commit whatever I wanted without any kind of supervision or checks, but this is generally not something you can do when it is a matter of life and death.
It is possible a single engineer was negligent or even malicious, but I would rather believe the whole organization was at the very least encouraging a culture of "do whatever to be ready at the end of the quarter and I don't want any detail" (orally of course)
You can't be reckless in a place where everyone is paying really close attention to safety and regulations. On the other hand if you are in environment where everyone is pretty lenient, it is much easier to slip between the checks.
But, from a journalistic point of view, even if the story is not fabricated, it is a better story to have a villain, than trying to convey to the readership how complex organizations (mis)behave.
In the situation you describe above, it would be nearly impossible for the defect to appear in the first place. You wouldn't need to hide anything for years, because it would never need to be fixed.
Furthermore, when trust is placed in the system and blame not given to the individual, then the momentum is to improve the final quality, not to hide a defect. The problem would have been fixed as soon as it was realized without a need to hide, fear, or cover up. No one would be blamed and no one held responsible, and the problem may never have escalated.
The system was poor, therefore the quality was poor. This was not an individual's fault. Quality is a systems problem, plain and simple.
As an engineer that works on safety critical stuff I see it as GM finding a fall guy, and/or having laughable-if-it-wasn't-so-tragic controls in place.
Edit: The bbc report says 15 were fired, half were senior legal and engineering executives. To me the issue stops being DeGiorgio's fuckup, but how so many other folks were incentivized to roll with the fuckup. The fuckup is bad, but the the environment that permitted it and later didn't correct it is the real danger. But don't worry, GM assures us that the Valukas report doesn't find a conspiracy or cover-up.
I think it can be both. It can be a single guy who is responsible for the switch being deliberately misleading through acts and omissions. Then it can be the chain of command above and around him as well. To have GM yesterday fire 15 people - including many senior execs - and then today accuse them of copping out and finding a fall guy and planting the media story doesn't make a lot of sense. It's possible one of the execs who was fired leaked this if it was in fact leaked/planted.
All of the above, always. I highly doubt the legal / regulatory staff and the HR and PR staff in large corporations are stupid, you can bet they've covered every angle. It's the scapegoat (the single engineer), it's the staff around him, it's the intentionally leaked email, it's the exec who's a plant in the fired group, who will gave a job next week with some other auto manufacturer. Is there a term 'Public Relations Theatre'? If not, I coin it now. Everyone's got everyone else's back, and hands in each others pockets. Maybe someone will appear to take the fall, but behind the scenes that person will be looked after. Call me cynical, or maybe realistic?
Does anyone with experience in the automotive industry have any thoughts on this? What sort of culture would lead to the selection of an inappropriate part in the first place? Would it be cost driven? Does this idea that an individual engineer's component selection decision would play such a big part ring true with other commentators experiences?
Well when I did my mech eng college course one lecturer mentioned that a UK car manufacturer used the wrong type of component to fix the steering wheel to the shaft.
Resulting in it being possible for the steering wheel to detach! whilst driving.
It's likely just an error in the design process that turns into a problem that is cost driven, but from the perspective that the problem was found after tooling had been acquired and production started and would require a lengthy/costly recall procedure.
For reference, I work in the space industry, where piles of paperwork are produced to justify any component selection decision. I can't imagine a situation where anything, even down to individual capacitors, would be selected without at least 2 people having some input to that selection. Our qualification processes are absolutely massive and take decades though - not particularly suitable for the automotive industry.
I don't think that this can be compared directly to the kind of adhoc software development that is routine in many places. Engineering processes have developed over decades to eliminate human error and very, very rarely allow single unjustified decisions made by a single person to propegate through to a finished product.
I'm surprised that in a world where individual car platforms cost billions of dollars to develop there was never a point at which the switch selection was challenged. I haven't read the full report though - perhaps it's not quite as the news article suggests. It would be interesting to hear the point of view of someone in the automotive industry (if any of them are HN readers?)
I've actually lived with a DRE for several months, and he's a drinking buddy of mine (works at GM too). The first thing you need to understand is the pressure DRE's are under.
It sounds odd, but General Motors is a large and complex entity. Their release system is old and buggy, not to mention never completely overhauled, just constantly having more checks ticked on when ever a recall happens. This is actually true at Ford too, the Big 3 hold on to technology a lot longer then other companies. Ford's time clock system is still ran on emulated System360's using MTS a circa 1967 OS.
Lastly DRE's are more of a projects internal sales arm (they sell the product to management. A DRE's job is to also get the test results from the test engineer, get it to the management, take the managements complaints to the designers/hardware engineers, and relay the tests plans back to the test engineers.
Basically high technical middle management. Their responsibilities at some companies may involve actually writing test plans, sometimes not. I don't believe GM does directly but somebody under the DRE does this.
Ultimately the DRE is just the guy who gets all the forms to all the right people so he can push the plans into the system. And takes all the responsibility when a product fails. On top of that your normally managing 5-6 products concurrently, each taking 18-72 months to clear the system. Your work load is basically endless meetings 500+ email a day, and 100+ phone calls.
It sounds like the automotive manufacturing industry is ripe for disruption. Enter the likes of Telsa Motors. Being able to focus on two or three products, getting them right, being 'agile'. Of course, for the agile new company to survive there has to be old cumbersome companies for it to carve a market from. I have a hunch there will be others, probably 'cheap Chinese knock-offs'[1] that will outperform the incumbents on price and quality. Hopefully the whole industry benefits.
1. I use the term 'Chinese' in this phrase to represent any state actor who can rapidly deploy, or incentivise, manufacturing capability at a significant discount to their Western counterparts.
The auto-industry isn't easy. The money involved in these industries is massive, far larger then tech as Ford arguably the middle of the big 3 has 50% more annual revenue then Microsoft (who beats Google), and owns more then 200 billion in assets.
>Enter the likes of Telsa Motors
I don't think we can call Telsa Motors a success yet strictly speaking. When its own CEO states its stock is overvalued we need to reconsider out position on the company. Telsa has only had 1 profitable quarter so far in its entire history.
>I have a hunch there will be others, probably 'cheap Chinese knock-offs
I'm not even gonna talk about problems with Chinese steel, Chinese quality control, Testing, etc. Lets just focus on sourcing. Developing a car's transmission costs roughly 5-10 billion dollars. Its so expensive to do that really no automakers do it themselves, same with engine designs. They share these collectively.
To disrupt the industry you need cash, and a lot of it. Metal dyes are outrageous, 250k to 500k+. The initial investments a company needs to break into the industry is stupid high.
On top of that, as an in-experienced new comer it'll be challenging to gain the public perception of having a higher quality product, as your first attempt may not be.
Products in the Auto-industry typically have decade long life spans, with half a decade spent testing them before release. So your trying get 1 billion in angel funding, for a decade you won't even be profitable?
> Ford's time clock system is still ran on emulated System360's using MTS
Fuck! MTS? The Michigan Terminal System? I haven't heard that being mentioned since I was, literally, a kid. Even IBM considers that obsolete, and IIRC even development stopped about 15 years ago!
zOS and SystemZ's are backwards compatible to System360 for a reason. IBM may consider a lot of old stuff obsolete but their customers don't.
A lot of the Detroit Metro UUG (Unix Users Group, the LUG is very small) people like to find some of the old tech in the basements of Ford and GM and use it for show and tell :)
zOS and SystemZ are indeed backwards compatible, and not without reason, but stuff like MTS really is obsolete. Or at least I think it should be (I'm not involved in maintaining such systems, I just enjoy collecting old computers)
I have a friend who worked for TI's automotive. They sold it off as the entire sector is bellow TI's minimum profitability level. So cost is issue #1..
But the subtext in the article makes it pretty clear. They are tracking cost of design as accurately as unit costs.
It is largely an "in for a dime in for a dollar" scenario for each internal engineer when they approve a design direction that appears to cut a cost. The larger oversight is unaware of some of the hidden issues and each engineer is motivated to paper over them to some extent to protect their relative standing. The engineers who are completely transparent are completely unemployed.
Well when you have too many conflicting responsibilities tied up in one person your bound to see something slip through.
Its not a culture problem, its a management problem. One person cannot be allowed to essentially design, development, qa, and support. If they are who knows what essential steps are not fully performed if at all? checks and balances fall by the wayside when your both.
One note I remember from some buddies who used to be in similar careers, part costs are calculated not by individual unit prices but the production run. Hence a change you say, that is only a few cents can be hundreds of thousands of dollars.
I hear you. But "Government Motors" is a clever marketing phrase right along with "Death Panels" that became a catch cry for a political movement high on rhetoric and low on intellectual rigor. Thus my disappointment that someone on HN might also inhabit that camp.
Thus my disappointment that someone on HN might also inhabit that camp.
Yeah, I hear your disappointment. But, HN is not (and doubt ever was) a homogenous community. It would be pretty boring and even more of an echo chamber if it were. Roses and Dandelions, one more than the other ;-), all form the HN crowd! Sure, dissing redditization is a part of being a responsible HN citizen, but let's not throw the baby out with the bath water!
(now get off my nice lawn all you pesky whippersnappers) :P
It doesn't matter what the engineer did. The fact that the car company had so few controls that a single engineer could approve something faulty then change the specs of it in future models without anyone noticing is worrying to say the least.
It is a bit like banking and these "rogue traders" who spend a billion dollars. They are at fault certainly BUT more so it is the banks fault for having such lax measures that they had access to this much money.
Right. They're called checks and balances for a reason. It's more than fair for GM to assume that it's paid employees follow procedures to ensure that this type of thing doesn't happen. So to say it doesn't matter what the engineer did is a poor choice of words. Given the stakes though it's also incumbent upon GM to ensure that there are other procedures and processes in place should an employee not do the right thing. It sounds like multiple people over the course of time (failure at a media launch?!) were suspicious about this switch but no one escalated the issue.
They had checks in place for errors, not for being deliberately misled by one of their own employees. If they had checks against that, we'd be reading a blog about procedures and corporate culture in GM suffocating engineering.
Being a pretty big process guy... I have to agree with SilkRoadie here. I should preface this by saying that my experience comes from developing FDA regulated medical diagnostic software and not automobiles. So I realize there may be a significantly lower quality threshold that GM is obliged to meet. Having mentioned that, I can tell you that, at least in that medical diagnostic software startup, the idea that an engineer could design and implement ANYTHING and get it into the software build without QA noticing is somewhat fanciful. Someone from QA would have had to be in on it.
The process and company structure pretty much made it impossible for the the type of situation outlined in the article to occur.
Just as a 30,000 ft overview of the process... imagine a place where every little change required a minimum of 4 signatures, engineer, QA engineer, Head of QA/Reg, and CEO. This allows not only QA, but also the FDA to determine precisely who was responsible for each change. Imagine further that the signatures have to be on paper, and that giant mass of paper documentation has to be submitted to the FDA, along with mountains of other documentation as well as the software itself, for approval. (But everyone understands that it's also submitted so that the FDA knows who to throw in prison if serious bugs crop up.)
The situation outlined in the article should be STRUCTURALLY impossible as well. At the startup I worked at for instance, build engineering was under QA. This meant that a development engineer could not give the build team a build directive. Only the head of QA/Reg could do that. Also, the head of QA/Reg didn't report to the COO or CFO or Chief Counsel or anything like that, she reported directly to the CEO. As well, she was a frequent contributor at board meetings. There was just no way to pull an end around on her.
That's just a rudimentary outline of the controls we had in place at that startup. Believe me, it actually got a WHOLE lot more restrictive than what I've outlined, but the other stuff is not important to the point I'm trying to make.
And that point is this...
While I realize that for cars the safety standard is probably lower than it is for medical imaging and RTP packages, you would think GM would have those rudimentary controls in place at a minimum ???
Unless you've got a series of physical safety measures (think GoldenEye style dual keys), your processes are just a series of rules. Enforcement of these rules is probably quite redundant, but I promise you that with sufficient motivation someone high enough in your company could knowingly insert (or cover up) flaws which could kill somebody.
Such motivation might come from the fear of losing your job.
Well... yes... we had "dual key" type things going on as well. (That was part of what I meant by "it got a WHOLE lot more restrictive".)
But, really, isn't an "engineer", someone pretty LOW in the hierarchy ?
I mean, that's kind of the point, the better the structure and processes... the higher up you will need to go to subvert them. In the case of GM... the processes were so poor that you didn't need to go beyond the engineer level to subvert them. (Assuming the information in the article is to be believed.)
I imagine that it may have been somewhat simple for GM to implement a system to prevent this, but from another perspective I also can conceive that such a system might effect GM's bottom line negatively. If having engineers cut corners on their own (likely with the encouragement of management) saves money, incompetent upper managers might pad their bonuses with these little cut corners(an out of spec switch here, a faulty connection there). Normally this just means small problems for consumers down the line, this time it just ended up killing people. So who gets blamed? The guy at the bottom? I'm not buying it.
Thanks for saying this. I've worked for companies who didn't trust their engineers. It's terrible.
To anyone who read this article and thought what the GP thought, consider that the value of any "thought worker" is inherently tied to the amount of trust you're able to place in them. This is why as an engineer the more you get paid, the more accountability you have.
If you ever work for a place which suffers from institutionalized distrust for its employees you should run. Run very very quickly in the opposite direction.
If the engineer was the same one making those checks, then they effectively did not have checks. I can't believe the quality control was so lax that nobody else checked the final product against specs, but apparently nobody did.
Yes! Systems are the problem here. He did not make that decision in isolation—there was some pressure from the supply chain, his manager, or cost that he would have been punished for had he not made this decision. A fear or motivation existed which was the result of poor management practices or a fragile quality process at odds with many other variables.
No matter how you slice it, this is a management problem, and a cultural problem. It is not an individual responsibility to ensure quality. If it is left to an individual, then things like this happen out of sheer statistical probability. The only thing you can depend on is human fallibility, so your systems should be designed for it; they should expect it, more than anything.
We should all look at quality as a systems, management, and fundamentally cultural pursuit.
Sadly, the individual getting the brunt of all the blame here will only worsen the cultural and management factors that caused it in the first place. Future employees will fear individual retribution, and will make rash, illogical, uninformed decisions in the future out of fear. This, too, will not be their "fault" but the fault of the company, the management, and the systems around them.
Deming. Look him up. There's a reason this does not happen in Japanese car companies. There's a reason that when it does, individual employees are not blamed. The reason is critically important to the success of American manufacturing in the future.
The fact remains though that if you are doing anything that could result in injury or death to third parties, you should never bow to management pressure.
I used to inspect amusement rides for safety when I was in college. Occasionally there would be a defect that would arise that would make me feel it was not as safe as I thought it could be. Managers and owners of course would not want their ride shut down on a busy summer day and would pressure me not to close it but I did not care. I knew if children were hurt or killed I would feel horrible, and be the guy taking the blame or criminal penalty for negligence.
So if you are an engineer for some safety critical system never bow in to management pressure as it will be viewed as 100% your fault when something goes wrong. It is not all this engineer's fault as people have pointed out, but when you put your name on the line approving something it is then your responsibility.
This news story talks about the proximate cause of the failure and how the cover-up started. But it doesn't say how the cover-up propagated. 14 other people were fired. But we don't know why or whether that was the extent of either covering up or incompetently failing to to uncover either the problem or the cover-up.
I think modern testing culture and source repositories can help avoid this kind of negligence in software projects today. I'm hardly Panglossian enough to suggest that it's common, but if you have an engineering culture where specifications and especially regressions are encoded in tests, it starts becoming clear when your software project is or is not satisfying the requirements. Decisions like removing a test because it fails intermittently are documented by source control, and can be reviewed by peers and management.
I have to wonder how much the focus on the switch is papering over other problems. The fact that a non-essential mechanical component can cause accidents like this seems fragile.
When the key pops out, the engine stalls, power steering goes out and the airbags do not inflate. Where is the "defense in depth"? A robust system should really require two things to go wrong.
Is it that "key in ignition" is a critical part of the vehicle's state machine?
Not being able to decisively turn off the engine has also killed people, remember the Toyota issue not long ago? We think that in general the drivers panicked and floored the accelerator thinking they were pressing the brakes, but the complexity/difference of turning off the engine with a button---as I recall it had to be held down for something more than a second---vs. turning a key all the way counterclockwise and ultimately pulling it out could have resolved the user errors short of fatal crashes.
Ditto the theory that some or many of these were caused by the accelerator getting stuck on a floor mat. Or an embedded computer and/or components attached to it failing in a bad way.
Compare to the big red buttons on some computers and hopefully in all big machine rooms: there are times, when lives are threatened, that you want an easy way to immediately turn everything off. Even if you add a molly-guard in front: https://en.wikipedia.org/wiki/Big_red_button#Molly-guard
If you're driving down the freeway, putting the gear selection lever to neutral would be a much better choice than killing the engine and taking out the key.
If you're so panicked you're flooring the accelerator thinking it's the brake, you might not think of that, and since you've not actually depressed the brake pedal, it wouldn't shift anyway, would it?
If you're truly standing on the brake, then it ought to work, but you might not think of it. I'll bet a lot of people don't usually think of any gears but park, normal forward and reverse.
On my car, you can always shift from Drive to Neutral just by pushing forward on the gear lever, without your foot on the brake, without pressing the button on the gear lever, etc.
Its interesting how much attention the general public has given to this yet for only a handful of deaths. For some perspective, 34,000 people died in the US during 2012 due to motor vehicles[1].
The big automakers are ultimately causing way more deaths by not adopting autonomous vehicle technology sooner if you ask me. And they will continue to do so because they realize that autonomous fleets make more sense than every consumer buying a car or two hence autonomous cars mean their business is in jeopardy/going to shrink.
We might also look into deaths associated with air quality/pollution and the number of deaths resulting from it. Its always going to be hard to pin down what number of deaths correlates to how many cars driving some number of miles, but I would put serious money on lower emission vehicles significantly reducing deaths resulting from air pollution. Detroit's reluctance to adopt cleaner technology and its obvious love affair with oil is killing many more than their incompetent switch DREs.
Ultimately, even the switch design is a similar phenomena. While many automakers started to get away from keyed ignition switches altogether, GM has continued to use a super cheap part that's obviously a design from the past. You can't tell me the radio on my grandmothers key ring cost versus their switch is worth 13 lives.
Certainly, don't mean to trivialize those that have been killed by this, just want to point out that Detroit is so slow to change that ultimately its costing us way more lives than just those killed by this switch problem.
I disagree. This is particularly interesting because there was an active coverup of the issue. This could have been resolved with almost NO deaths, had the actions of this one engineer not been taken.
I don't mean to troll. I agree its interesting because there is a cover up and an engineer who could have saved 13 lives and didn't.
My main point is that big automakers are doing this same think, cover up and all on a grand scale but because you have to look at it from 10,000 feet to see it, it doesn't get the same type of criticism.
you said you disagree, there were like six points I tried to make, was there a particular set you disagreed with?
I never considered your post a troll, I merely disagree with the general premise that I took from your post that this issue is less important than tardy development of autonomous vehicle and emissions controls technology.
Those are important, but not quite as immediate as this matter, is really the issue I had. But you are quite right, those are still important issues.
(shame on those who voted you down, IMO you made a good point but dialogue would be more appreciated than random clicking on a down vote).
I will challenge you on one point: Keyless ignitions are kind of cool (I own a vehicle with one), but the mechanisms of enabling the two stages of aux power before "engine on" are cumbersome. The behavior of the button changes with whether you're in N or D, in addition to whether your foot is on the pedal or not.
Knowing what state the vehicle will be in with a keyed ignition is much easier.
Source: mid-twenties engineer. I can't imagine how grandma must feel trying to fiddle with 'radio on, windows rolling, engine off'.
thats an interesting thought. My grandmother hasn't had issues lol but more relevantly, I think you are talking about the fact that without aux, power steering, power brakes, airbags, etc. won't work?
I can kinda see that, but seems like there are other solutions to that. Say I was driving down the road at 40, and my ignition indicated that the car was to be turned off, you would think some little state machine (or even stateless) would say, oh hey, we are actually moving....maybe don't turn of aux power?
65 comments
[ 4.3 ms ] story [ 82.2 ms ] threadThis is in a way cumbersome and annoying but it is understandable why it is put in place this way.
I used to work in consumer electronics, where I could commit whatever I wanted without any kind of supervision or checks, but this is generally not something you can do when it is a matter of life and death.
You can't be reckless in a place where everyone is paying really close attention to safety and regulations. On the other hand if you are in environment where everyone is pretty lenient, it is much easier to slip between the checks.
But, from a journalistic point of view, even if the story is not fabricated, it is a better story to have a villain, than trying to convey to the readership how complex organizations (mis)behave.
Furthermore, when trust is placed in the system and blame not given to the individual, then the momentum is to improve the final quality, not to hide a defect. The problem would have been fixed as soon as it was realized without a need to hide, fear, or cover up. No one would be blamed and no one held responsible, and the problem may never have escalated.
The system was poor, therefore the quality was poor. This was not an individual's fault. Quality is a systems problem, plain and simple.
Edit: The bbc report says 15 were fired, half were senior legal and engineering executives. To me the issue stops being DeGiorgio's fuckup, but how so many other folks were incentivized to roll with the fuckup. The fuckup is bad, but the the environment that permitted it and later didn't correct it is the real danger. But don't worry, GM assures us that the Valukas report doesn't find a conspiracy or cover-up.
Resulting in it being possible for the steering wheel to detach! whilst driving.
It's no different than an engineer selection of a single line of code, if it's in the wrong place, crashy crashy.
Here's a much better article: http://www.seattlepi.com/news/us/article/Engineer-s-switch-f...
Perhaps the engineer was distracted by an underground boxing ring.
I don't think that this can be compared directly to the kind of adhoc software development that is routine in many places. Engineering processes have developed over decades to eliminate human error and very, very rarely allow single unjustified decisions made by a single person to propegate through to a finished product.
I'm surprised that in a world where individual car platforms cost billions of dollars to develop there was never a point at which the switch selection was challenged. I haven't read the full report though - perhaps it's not quite as the news article suggests. It would be interesting to hear the point of view of someone in the automotive industry (if any of them are HN readers?)
It sounds odd, but General Motors is a large and complex entity. Their release system is old and buggy, not to mention never completely overhauled, just constantly having more checks ticked on when ever a recall happens. This is actually true at Ford too, the Big 3 hold on to technology a lot longer then other companies. Ford's time clock system is still ran on emulated System360's using MTS a circa 1967 OS.
Lastly DRE's are more of a projects internal sales arm (they sell the product to management. A DRE's job is to also get the test results from the test engineer, get it to the management, take the managements complaints to the designers/hardware engineers, and relay the tests plans back to the test engineers.
Basically high technical middle management. Their responsibilities at some companies may involve actually writing test plans, sometimes not. I don't believe GM does directly but somebody under the DRE does this.
Ultimately the DRE is just the guy who gets all the forms to all the right people so he can push the plans into the system. And takes all the responsibility when a product fails. On top of that your normally managing 5-6 products concurrently, each taking 18-72 months to clear the system. Your work load is basically endless meetings 500+ email a day, and 100+ phone calls.
1. I use the term 'Chinese' in this phrase to represent any state actor who can rapidly deploy, or incentivise, manufacturing capability at a significant discount to their Western counterparts.
>Enter the likes of Telsa Motors
I don't think we can call Telsa Motors a success yet strictly speaking. When its own CEO states its stock is overvalued we need to reconsider out position on the company. Telsa has only had 1 profitable quarter so far in its entire history.
>I have a hunch there will be others, probably 'cheap Chinese knock-offs
I'm not even gonna talk about problems with Chinese steel, Chinese quality control, Testing, etc. Lets just focus on sourcing. Developing a car's transmission costs roughly 5-10 billion dollars. Its so expensive to do that really no automakers do it themselves, same with engine designs. They share these collectively.
To disrupt the industry you need cash, and a lot of it. Metal dyes are outrageous, 250k to 500k+. The initial investments a company needs to break into the industry is stupid high.
On top of that, as an in-experienced new comer it'll be challenging to gain the public perception of having a higher quality product, as your first attempt may not be.
Products in the Auto-industry typically have decade long life spans, with half a decade spent testing them before release. So your trying get 1 billion in angel funding, for a decade you won't even be profitable?
Fuck! MTS? The Michigan Terminal System? I haven't heard that being mentioned since I was, literally, a kid. Even IBM considers that obsolete, and IIRC even development stopped about 15 years ago!
A lot of the Detroit Metro UUG (Unix Users Group, the LUG is very small) people like to find some of the old tech in the basements of Ford and GM and use it for show and tell :)
But the subtext in the article makes it pretty clear. They are tracking cost of design as accurately as unit costs.
It is largely an "in for a dime in for a dollar" scenario for each internal engineer when they approve a design direction that appears to cut a cost. The larger oversight is unaware of some of the hidden issues and each engineer is motivated to paper over them to some extent to protect their relative standing. The engineers who are completely transparent are completely unemployed.
Its not a culture problem, its a management problem. One person cannot be allowed to essentially design, development, qa, and support. If they are who knows what essential steps are not fully performed if at all? checks and balances fall by the wayside when your both.
One note I remember from some buddies who used to be in similar careers, part costs are calculated not by individual unit prices but the production run. Hence a change you say, that is only a few cents can be hundreds of thousands of dollars.
- Gerald Weinberg's 'First Principle of Financial Management' and 'Second Rule of Failure Prevention' [1]
[1] 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993
Yeah, I hear your disappointment. But, HN is not (and doubt ever was) a homogenous community. It would be pretty boring and even more of an echo chamber if it were. Roses and Dandelions, one more than the other ;-), all form the HN crowd! Sure, dissing redditization is a part of being a responsible HN citizen, but let's not throw the baby out with the bath water!
(now get off my nice lawn all you pesky whippersnappers) :P
It is a bit like banking and these "rogue traders" who spend a billion dollars. They are at fault certainly BUT more so it is the banks fault for having such lax measures that they had access to this much money.
according to the article, this blameless paragon of virtue deliberately hid his actions.
The process and company structure pretty much made it impossible for the the type of situation outlined in the article to occur.
Just as a 30,000 ft overview of the process... imagine a place where every little change required a minimum of 4 signatures, engineer, QA engineer, Head of QA/Reg, and CEO. This allows not only QA, but also the FDA to determine precisely who was responsible for each change. Imagine further that the signatures have to be on paper, and that giant mass of paper documentation has to be submitted to the FDA, along with mountains of other documentation as well as the software itself, for approval. (But everyone understands that it's also submitted so that the FDA knows who to throw in prison if serious bugs crop up.)
The situation outlined in the article should be STRUCTURALLY impossible as well. At the startup I worked at for instance, build engineering was under QA. This meant that a development engineer could not give the build team a build directive. Only the head of QA/Reg could do that. Also, the head of QA/Reg didn't report to the COO or CFO or Chief Counsel or anything like that, she reported directly to the CEO. As well, she was a frequent contributor at board meetings. There was just no way to pull an end around on her.
That's just a rudimentary outline of the controls we had in place at that startup. Believe me, it actually got a WHOLE lot more restrictive than what I've outlined, but the other stuff is not important to the point I'm trying to make.
And that point is this...
While I realize that for cars the safety standard is probably lower than it is for medical imaging and RTP packages, you would think GM would have those rudimentary controls in place at a minimum ???
Such motivation might come from the fear of losing your job.
But, really, isn't an "engineer", someone pretty LOW in the hierarchy ?
I mean, that's kind of the point, the better the structure and processes... the higher up you will need to go to subvert them. In the case of GM... the processes were so poor that you didn't need to go beyond the engineer level to subvert them. (Assuming the information in the article is to be believed.)
To anyone who read this article and thought what the GP thought, consider that the value of any "thought worker" is inherently tied to the amount of trust you're able to place in them. This is why as an engineer the more you get paid, the more accountability you have.
If you ever work for a place which suffers from institutionalized distrust for its employees you should run. Run very very quickly in the opposite direction.
No matter how you slice it, this is a management problem, and a cultural problem. It is not an individual responsibility to ensure quality. If it is left to an individual, then things like this happen out of sheer statistical probability. The only thing you can depend on is human fallibility, so your systems should be designed for it; they should expect it, more than anything.
We should all look at quality as a systems, management, and fundamentally cultural pursuit.
Sadly, the individual getting the brunt of all the blame here will only worsen the cultural and management factors that caused it in the first place. Future employees will fear individual retribution, and will make rash, illogical, uninformed decisions in the future out of fear. This, too, will not be their "fault" but the fault of the company, the management, and the systems around them.
Deming. Look him up. There's a reason this does not happen in Japanese car companies. There's a reason that when it does, individual employees are not blamed. The reason is critically important to the success of American manufacturing in the future.
I used to inspect amusement rides for safety when I was in college. Occasionally there would be a defect that would arise that would make me feel it was not as safe as I thought it could be. Managers and owners of course would not want their ride shut down on a busy summer day and would pressure me not to close it but I did not care. I knew if children were hurt or killed I would feel horrible, and be the guy taking the blame or criminal penalty for negligence.
So if you are an engineer for some safety critical system never bow in to management pressure as it will be viewed as 100% your fault when something goes wrong. It is not all this engineer's fault as people have pointed out, but when you put your name on the line approving something it is then your responsibility.
When the key pops out, the engine stalls, power steering goes out and the airbags do not inflate. Where is the "defense in depth"? A robust system should really require two things to go wrong.
Is it that "key in ignition" is a critical part of the vehicle's state machine?
Not being able to decisively turn off the engine has also killed people, remember the Toyota issue not long ago? We think that in general the drivers panicked and floored the accelerator thinking they were pressing the brakes, but the complexity/difference of turning off the engine with a button---as I recall it had to be held down for something more than a second---vs. turning a key all the way counterclockwise and ultimately pulling it out could have resolved the user errors short of fatal crashes.
Ditto the theory that some or many of these were caused by the accelerator getting stuck on a floor mat. Or an embedded computer and/or components attached to it failing in a bad way.
Compare to the big red buttons on some computers and hopefully in all big machine rooms: there are times, when lives are threatened, that you want an easy way to immediately turn everything off. Even if you add a molly-guard in front: https://en.wikipedia.org/wiki/Big_red_button#Molly-guard
If you're truly standing on the brake, then it ought to work, but you might not think of it. I'll bet a lot of people don't usually think of any gears but park, normal forward and reverse.
The big automakers are ultimately causing way more deaths by not adopting autonomous vehicle technology sooner if you ask me. And they will continue to do so because they realize that autonomous fleets make more sense than every consumer buying a car or two hence autonomous cars mean their business is in jeopardy/going to shrink.
We might also look into deaths associated with air quality/pollution and the number of deaths resulting from it. Its always going to be hard to pin down what number of deaths correlates to how many cars driving some number of miles, but I would put serious money on lower emission vehicles significantly reducing deaths resulting from air pollution. Detroit's reluctance to adopt cleaner technology and its obvious love affair with oil is killing many more than their incompetent switch DREs.
Ultimately, even the switch design is a similar phenomena. While many automakers started to get away from keyed ignition switches altogether, GM has continued to use a super cheap part that's obviously a design from the past. You can't tell me the radio on my grandmothers key ring cost versus their switch is worth 13 lives.
Certainly, don't mean to trivialize those that have been killed by this, just want to point out that Detroit is so slow to change that ultimately its costing us way more lives than just those killed by this switch problem.
1 http://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_in...
My main point is that big automakers are doing this same think, cover up and all on a grand scale but because you have to look at it from 10,000 feet to see it, it doesn't get the same type of criticism.
you said you disagree, there were like six points I tried to make, was there a particular set you disagreed with?
Those are important, but not quite as immediate as this matter, is really the issue I had. But you are quite right, those are still important issues.
(shame on those who voted you down, IMO you made a good point but dialogue would be more appreciated than random clicking on a down vote).
Knowing what state the vehicle will be in with a keyed ignition is much easier.
Source: mid-twenties engineer. I can't imagine how grandma must feel trying to fiddle with 'radio on, windows rolling, engine off'.
I can kinda see that, but seems like there are other solutions to that. Say I was driving down the road at 40, and my ignition indicated that the car was to be turned off, you would think some little state machine (or even stateless) would say, oh hey, we are actually moving....maybe don't turn of aux power?