Yea. I doubt we'll see it, but there should be criminal negligence changes in there somewhere.
There are so many people involved it's easy to distribute blame or blame the entire process. But if the right email can be tracked down, there's going to be someone(s) in that loop that fits the charges for 200+ counts of manslaughter.
physical sensors malfunction all the time, and you have a fleet of several hundred aircraft.
You would be shocked to learn how many faults an aircraft can have at a given time and still be allowed to take off. That's what redundancy is for, if everything had to be all-systems-go every time then you couldn't run an aircraft for a week straight. It doesn't matter as long as you have a spare, and it'll get fixed the next time you're in for maintenance.
the issue with the 737 MAX is that there are only two sensors, and so far I don't see that being addressed. One sensor failing should not kill the aircraft, and it should not disable a control regime that is necessary for the aircraft to receive type certification.
> You would be shocked to learn how many faults an aircraft can have at a given time and still be allowed to take off.
Yep. Next time you fly, look through the cockpit door as you board for maintenance tag-outs -- usually red flags with 'INOP' labeling, most likely on the circuit breaker panels above the pilots. The Gimli Glider incident occurred partially because of a circuit breaker tag-out in the 767 fuel quantity indicator system.
Aircraft manufacturers and regulators produce a Minimum Equipment List which stipulates what is required for flight. If you get your hands on one for the plane you're on, you might be surprised to discover what _isn't_ on it and what is permitted to fail. It's heavy, but here's an older American MEL for the 737 family:
In the Gimli glider case the aircraft wasn't legally allowed to take off. The pilots also deferred the decision on whether to take off to the maintenance crew who are not the ones authorized to make that decision. The pilots were initially demoted because of that but were restored due to public pressure.
There was mitigating circumstances though, the procedures were frequently being revised (the 767 was brand new) and previously it had been permissible to take off without a fuel indicator.
None of this takes away from the huge achievement of their landing. Simulator runs show the landing was near impossible. Had Captain Pearson not been an experienced glider pilot its likely they would have ditched in the river, and the historical survivability of water landings are not good.
This observation is incomplete. The problem is not the absence of more sensors. It's that, in order to quickly produce (and sell) the 737-MAX, it was changed so much that its behavior was fundamentally different than that of the 737.
The MCAS system, which was dependent on those two sensors, was developed to compensate for the degree to which the plane behaved differently. And Boeing underplayed how much the performance of the new plane had changed, and hid the existence of the MCAS system from commercial plane pilots, to the point where it wasn't even in their manuals.
Had Boeing behaved ethically, they would have told the FAA that the new 737 required pilot training. But that would have diminished sales and hurt Boeing's stock price. That's the managerial misbehavior that many of the commenters in this thread are remarking on.
MCAS was required for certification, in that it increases the linearity of control forces through to a stall. Without it, it becomes easier to enter a stall.
MCAS was not a fix to make training lighter, the aircraft would not be able to get a type certificate without it.
>Employees at the Renton, Wash., factory where the Max is produced were overworked, exhausted and making mistakes, Mr. Pierson said in an interview. A cascade of damaged parts, missing tools and incomplete instructions was preventing planes from being built on time. Executives were pressuring workers to complete planes despite staff shortages and a chaotic factory floor.
I wonder how much of this is directly the result of penny pinching by MBA types. I'm gradually coming to believe that non-technical management of technical endeavors is the root of most failure in the engineering domain.
Technical management of technical endeavors can be just as bad, although you could be right when you say it's "the root of most failure in the engineering domain," because when technical people manage, the product is often either not delivered, or is delivered but doesn't have any critical failures because people don't want to use it. It's very good when managers have technical expertise in addition, but managerial expertise is much more important in management. Just something I learned in my two-plus decades in the software industry.
You need both technical and non-technical management skills, and if you can't get both in the same person (likely!), you need the people from each side in a relationship that's based on trust and recognition of each other's expertise.
Too often these relationships break down and become adversarial, to the detriment of everyone involved, and of course the project.
Well, it is difficult to find an industry with worse track record of product quality and reliability than the software industry, I wouldn't use software development management as reference.
(also two decades is software development... and software usage!!)
In my experience managers lacking domain knowledge make more harm than good but are excellent putting emphasis on adjacent and auxiliary parts they are good at. Anyone put in charge of something they are not competent in would do the same.
Why does HN love to bash "MBA types" so much? The bit you quoted doesn't even have anything to do with any type of technical background. Do you think that only people with MBAs are taught that they need to meet deadlines and obligations? Do you seriously think that someone with an engineering degree would for some reason say "actually we don't need to meet deadlines, I won't pressure you to finish these planes on time"?
Because their decision making is political and short-sighted with no sense of ownership of their management decisions for the future of the company; rather they are looking for their next golden parachute.
Honestly, and I'm being really frank here, a lot of the management "expertise" at fortune 50+ firms that come from traditional (re: MBA) tracks has just two tools in their toolbelt:
1. Meetings to "align"
2. Cracking the whip.
That's it. That's all they have. They have poor insight into creating team chemistry, culture, focused work environments, mentorship, long-term planning, QA processes, sourcing efficiency feedback, etc. - all those things that are crucial to boosting productivity and creating supersonic teams. The people that really know their stuff and can work miracles are people that have been in the trenches for 20 years, regardless of profession. And they hands-down make the best managers.
That's my honest 2 cents, it's a bit of disgruntlement from working in large enterprise environments with engineering and technology.
That’s a hilariously narrow minded generalization. If that’s your honest opinion then I have to assume you’ve never actually sat down and had a real conversation with any of these “MBA types” and talked about what they do. My guess is that the reason you think all they do is crack the whip and hold pointless meetings because that’s the only interaction you have with them, but I can guarantee you that every organization you have ever worked in has had plenty of “MBA types” working in ways that are invisible to you building exactly the team chemistries, culture, planning, processes etc that you mentioned.
I would agree that this is narrow minded - and would say that this opinion is a result of exactly what you mentioned - my interaction with them being fairly limited and clinical, despite them having a direct leadership role in my org.
I would argue that this is the problem unto itself. The idea that a manager can work invisibly "behind the scenes" to create these conditions is the squarest negative stereotype of the MBA - that they are clerical, remote, number-crunchers who don't have an intuitive human grasp of the unique challenges their teams face, because they lack experience where the rubber meets of the road of their organization. I think this Boeing case conforms to this stereotype, as does my personal experience.
But I don't believe they're a scourge, or anything. I quantified it with "a lot" but not "most" ;)
When I said they work in ways invisible to you, I did not mean it as they are intentionally working “behind the scenes”. It is more likely that your narrowmindedness just blinds you to all of the things they do. The work is probably all done out in the open, and in almost all cases I’ve ever seen, the “MBA types” specifically seek out interaction with and input from “rubber meets the road” folks (side note: this is another hilarious notion that MBAs apparently aren’t involved in “rubber meets the road” tasks), but it’s generally the engineers that shy away from this interaction and then blame management for being out of touch.
In our organization we created a special "department" which is disconnected from the traditional corporate structure.
We're a compound of self-organized teams of Product Owners and Engineers.
Our traditional organization takes 10 weeks for feature X, we need at most 2 weeks, averaging at 1 week (10%).
We have the luxury of delivering the exact same product (as a green-field variant of our classical product as a SaaS solution), so it´s quite comparable in terms of scope.
The main difference is there is no hidden agenda of would-be managers, nothing between the customer than a PO who knows what he is doing (i.e. is doing regular A/B tests, customer interviews, involving the dev team as deep as possible to understand customer requirements).
We were awarded a nation-wide award for digital transformation.
No middle-managers, no HR.
You can reach me @ hackernews@disposable-email.ml for additional details if you like.
Summarizing: we got rid of all non-relevant management ballast and are able to deliver features at a pace of around 10x of a traditionally managed line.
Overtime: around 0% with a tendency to dip below 0%.
Addendum: the POs are 70% MBAs, but they are good (i.e. they learnt to deliver as opposed to manage)
Keeping in mind this whole thread is about the process of manufacturing aircraft, do you think this approach works equally well with traditional manufacturing?
My experience is that these lean approaches work well for something low on the severity scale, like an SaaS solution, but can more easily falter with complex safety-critical systems that blend multiple domains (e.g., mechanical, software, etc.) I think sometimes people interpret the process rigor that gets added to critical design to management bloat.
Do you have any insight into how BPS addresses software development? I've looked but so far most of the information is related to hardware manufacturing or using software as a tool (e.g., for training).
Sorry, you are going way to deep here, that can't be efficient can I just get the key figures and a time prognosis , thank you.
Modern mbas are not trained to manage, but to be con artists that fake it until they can get out of the scam.
They structure wise have the functionality of glue code. And glue code can be emergent if you two good component devs work to breach a gap. Fire them and give your interface architects a communication responsibility.
If you continue to discuss this, I will need you to explain to the group and my supervisor, why the HN post project is x weeks behind the schedule your numbers made it out to be.
Leaders with a technical background will have a better understanding of how increasing the pressure leads to poor workmanship and worsens the chances of finishing on time in the long run.
I don't speak for HN but personally I find the notion that business management can be distilled and generically applied across industries and disciplines to be nonsensical, when every technical project requires a firm grasp of at least the basic scientific/engineering principles to make sound planning decisions at all scales. How can you possibly estimate how long something will take to code for example, or evaluate the estimates of your subordinates, if you don't know how to code? Sure, this isn't the best example as code estimation is notoriously unreliable, but the point applies to technical work in general.
Much of the trouble at Boeing has been explained by various sources as having started with the replacement of managing engineers at a company known to be founded and run by engineers with executive types after the "reverse takeover"[1] by McDonnel. This seems to be as close as one could get to an A/B test and the results are exactly what I would have expected, as technical constraints are relaxed to boost margins and shrink/meet deadlines. Though I admit this is a small sample size and could be confirmation bias on my part.
>Do you seriously think that someone with an engineering degree would for some reason say actually we don't need to meet deadlines, I won't pressure you to finish these planes on time
No, but I would expect a manager with an engineering degree to be more likely to understand the absurdity and danger of building passenger aircraft with missing tools and incorrect instructions, red flag such issues, and push back deadlines to resolve them.
> I would expect a manager with an engineering degree to be more likely to understand the absurdity and danger of building passenger aircraft with missing tools and incorrect instructions, red flag such issues, and push back deadlines to resolve them.
You don't need to be an engineer to do any of these things. You really just need to ask questions to people who understand the details and listen to them. A good manager will know the right people to ask these questions to and follow their advice. Good management is not an easy skill however.
yes, someone with engineering background will be more inclined to say 'no' to unrealistic deadlines because they are aware of all the technical challenges and sometimes even overestimate the tech challenges.. someone with MBA would never know all things that could possibly go wrong, and might be focused more on the release features, deadlines and the budget cost.
Technical people think the "traveling salesman" is an interesting and difficult problem. Difficult even though the robot salesman will do everything exactly as the tech commands him to.
Now what if the salesman was human. This human has a feelings and a personality and other things going on in its life. Now what if you have to manage 100 of these salesmen, all with different culture, religion, education and age... and when you put these salesmen together in teams, they all have a different group dynamic. Sometimes you'll get a salesman who will even try to sabotage you. Plus you need to motivate, train, and lead all of them. For some reason technical people think this problem is so easy. (I'm not an MBA by the way)
This piece in The Atlantic, "The Long-Forgotten Flight That Sent Boeing Off Course" (https://news.ycombinator.com/item?id=21583001), blames management from afar by executives with no understanding of the engineering processes around aviation as the start of Boeing's current problems.
Did any of these production problems relate to the MCAS malfunction though?
Also, this is buried further down in the article than it should be.
> Mr. Pierson and his lawyers declined to answer whether he was seeking whistle-blower protection or filing a federal whistle-blower case. He has hired a prominent whistle-blower lawyer, Eric Havian, and could stand to gain monetarily if he pursued such a case.
If the AoA vanes were installed incorrectly, perhaps so?
"Mr. Pierson believes that the production problems may have played a role in the crashes. In both accidents, MCAS was triggered when a vane installed on the plane’s fuselage malfunctioned."
"Workers were completing jobs out of sequence, leading to additional mistakes."
Compensation for filing legitimate claims is an important carrot to encourage people with inside information to speak out. Whistleblowers are routinely subjected to intense retaliation and professional difficulties when they come forward, and the process of claiming and securing an award takes years.
Everything this whistleblower is saying seems consistent with dozens of other stories which have emerged of Boeing's corroded engineering practices.
That's nice. I'm not sure exactly what you think is happening here, but "make up a bunch of stuff and use it to pursue a whistleblower claim" doesn't strike me as a very appealing or sound strategy for ill-gotten gains. This isn't something that gets litigated in backroom discussions or tabloid news outlets.
Have you ever been in a courtroom? Not a lot of neutral statements of unvarnished truth pass back and forth there. Some do, but not many. I am extraordinarily worried about the precedent of paying witnesses for their testimony, which is what this amounts to. However good we think that our reasons may be.
Expert witnesses are paid by the hour, and still get paid whether the case is won or lost. The abuses of honesty in the expert witnesses system are still horrific enough.
The point of the whistleblower program should be to always have conclusive proof. It should point the light at the problem so resources can be used to get to the truth. The monetary reward is necessary for people to give up their livelihood forever. You only get the money when it leads to something.
> Did any of these production problems relate to the MCAS malfunction though?
I mean, the concern would be that MCAS is just the tip of the iceberg, and that similar fuckups exist within the planes due to this management trouble.
The thing that strikes me about the Boeing MAX story is that executives made decisions which they were told over and over again would put lives in danger, and so far the likely monetary damages are so small the stock is completely unfazed, and criminal prosecution is a distant, all-but-impossible notion.
I don't think the majority of people understand how deep this problem went. Back when they were first grounded, we saw stuff on HN nearly every week. I don't think it got much place elsewhere. Many of my friends are really unaware of how serious the Boeing situation is. Same thing for back when Uber had a new scandal (grey ball, harassment memo, hell map, hiring Eric Holder to run damage control, the other harassment memo). I'd tell friends I refused to use Uber and they were mostly unaware of half of that stuff.
What's worse is that the upcoming MCAS fix is not a fix. If one sensor is misbehaving (seemingly more common than desirable) then pitch-up will not be automatically corrected:
"Boeing specifically highlighted in presentation documents the new MCAS compares inputs from two angle of attack (AOA) sensors as opposed to a single sensor in the original, flawed, design. MCAS will only activate if data from both sensors agree the AOA is too high. MCAS will activate only once per high AOA event. Pilots will be able to override it and the system will not activate automatically a second time. In the original design, the system reactivated over and over. The presentation reiterates the crew can deactivate MCAS by turning off the automatic trim system and manually turning the trim wheel."
Hmm I wonder if that will be accepted. The whole point of MCAS was to avoid getting assigned a new type which would require airlines to pay to have pilots certified on the new type, with MCAS only working and only fixing the maneuvering differences when both AOA sensors agree (and Boeing not making AOA a critical system with redundancy) it will be interesting to see if this still gets the benefit of sharing type rating with the existing 737s. I imagine the FAA is going to be reluctant to just wave them through this time and I wonder if one activation is enough to actually correct the pitch up.
Yeah. I don’t get it. If MCAS changes the way the plane flies, and MCAS can be switched off, don’t the pilots need to know how to fly the plane without MCAS? Then why even have MCAS?
62 comments
[ 2.7 ms ] story [ 120 ms ] threadIt makes sense that sensors on an expensive plane shouldn't malfunction so early.
Two planes crashed and the higher-ups knew about the problems and were actively showing it under rug hoping somehow they can fix it without backlash.
There are so many people involved it's easy to distribute blame or blame the entire process. But if the right email can be tracked down, there's going to be someone(s) in that loop that fits the charges for 200+ counts of manslaughter.
You would be shocked to learn how many faults an aircraft can have at a given time and still be allowed to take off. That's what redundancy is for, if everything had to be all-systems-go every time then you couldn't run an aircraft for a week straight. It doesn't matter as long as you have a spare, and it'll get fixed the next time you're in for maintenance.
the issue with the 737 MAX is that there are only two sensors, and so far I don't see that being addressed. One sensor failing should not kill the aircraft, and it should not disable a control regime that is necessary for the aircraft to receive type certification.
Yep. Next time you fly, look through the cockpit door as you board for maintenance tag-outs -- usually red flags with 'INOP' labeling, most likely on the circuit breaker panels above the pilots. The Gimli Glider incident occurred partially because of a circuit breaker tag-out in the 767 fuel quantity indicator system.
Aircraft manufacturers and regulators produce a Minimum Equipment List which stipulates what is required for flight. If you get your hands on one for the plane you're on, you might be surprised to discover what _isn't_ on it and what is permitted to fail. It's heavy, but here's an older American MEL for the 737 family:
http://fsims.faa.gov/wdocs/mmel/b-737%20r54a.pdf
Pages 35-1 and 35-2 offer some interesting reading about the passenger oxygen system, for example.
There was mitigating circumstances though, the procedures were frequently being revised (the 767 was brand new) and previously it had been permissible to take off without a fuel indicator.
None of this takes away from the huge achievement of their landing. Simulator runs show the landing was near impossible. Had Captain Pearson not been an experienced glider pilot its likely they would have ditched in the river, and the historical survivability of water landings are not good.
The MCAS system, which was dependent on those two sensors, was developed to compensate for the degree to which the plane behaved differently. And Boeing underplayed how much the performance of the new plane had changed, and hid the existence of the MCAS system from commercial plane pilots, to the point where it wasn't even in their manuals.
Had Boeing behaved ethically, they would have told the FAA that the new 737 required pilot training. But that would have diminished sales and hurt Boeing's stock price. That's the managerial misbehavior that many of the commenters in this thread are remarking on.
https://www.nytimes.com/2019/09/18/magazine/boeing-737-max-c...
MCAS was not a fix to make training lighter, the aircraft would not be able to get a type certificate without it.
https://leehamnews.com/2019/02/08/bjorns-corner-pitch-stabil...
I wonder how much of this is directly the result of penny pinching by MBA types. I'm gradually coming to believe that non-technical management of technical endeavors is the root of most failure in the engineering domain.
Too often these relationships break down and become adversarial, to the detriment of everyone involved, and of course the project.
In my experience managers lacking domain knowledge make more harm than good but are excellent putting emphasis on adjacent and auxiliary parts they are good at. Anyone put in charge of something they are not competent in would do the same.
That could be why he led with "I wonder..."
1. Meetings to "align"
2. Cracking the whip.
That's it. That's all they have. They have poor insight into creating team chemistry, culture, focused work environments, mentorship, long-term planning, QA processes, sourcing efficiency feedback, etc. - all those things that are crucial to boosting productivity and creating supersonic teams. The people that really know their stuff and can work miracles are people that have been in the trenches for 20 years, regardless of profession. And they hands-down make the best managers.
That's my honest 2 cents, it's a bit of disgruntlement from working in large enterprise environments with engineering and technology.
I would argue that this is the problem unto itself. The idea that a manager can work invisibly "behind the scenes" to create these conditions is the squarest negative stereotype of the MBA - that they are clerical, remote, number-crunchers who don't have an intuitive human grasp of the unique challenges their teams face, because they lack experience where the rubber meets of the road of their organization. I think this Boeing case conforms to this stereotype, as does my personal experience.
But I don't believe they're a scourge, or anything. I quantified it with "a lot" but not "most" ;)
We're a compound of self-organized teams of Product Owners and Engineers.
Our traditional organization takes 10 weeks for feature X, we need at most 2 weeks, averaging at 1 week (10%).
We have the luxury of delivering the exact same product (as a green-field variant of our classical product as a SaaS solution), so it´s quite comparable in terms of scope.
The main difference is there is no hidden agenda of would-be managers, nothing between the customer than a PO who knows what he is doing (i.e. is doing regular A/B tests, customer interviews, involving the dev team as deep as possible to understand customer requirements).
We were awarded a nation-wide award for digital transformation.
No middle-managers, no HR.
You can reach me @ hackernews@disposable-email.ml for additional details if you like.
Summarizing: we got rid of all non-relevant management ballast and are able to deliver features at a pace of around 10x of a traditionally managed line.
Overtime: around 0% with a tendency to dip below 0%.
Addendum: the POs are 70% MBAs, but they are good (i.e. they learnt to deliver as opposed to manage)
My experience is that these lean approaches work well for something low on the severity scale, like an SaaS solution, but can more easily falter with complex safety-critical systems that blend multiple domains (e.g., mechanical, software, etc.) I think sometimes people interpret the process rigor that gets added to critical design to management bloat.
Modern mbas are not trained to manage, but to be con artists that fake it until they can get out of the scam.
They structure wise have the functionality of glue code. And glue code can be emergent if you two good component devs work to breach a gap. Fire them and give your interface architects a communication responsibility.
If you continue to discuss this, I will need you to explain to the group and my supervisor, why the HN post project is x weeks behind the schedule your numbers made it out to be.
Much of the trouble at Boeing has been explained by various sources as having started with the replacement of managing engineers at a company known to be founded and run by engineers with executive types after the "reverse takeover"[1] by McDonnel. This seems to be as close as one could get to an A/B test and the results are exactly what I would have expected, as technical constraints are relaxed to boost margins and shrink/meet deadlines. Though I admit this is a small sample size and could be confirmation bias on my part.
>Do you seriously think that someone with an engineering degree would for some reason say actually we don't need to meet deadlines, I won't pressure you to finish these planes on time
No, but I would expect a manager with an engineering degree to be more likely to understand the absurdity and danger of building passenger aircraft with missing tools and incorrect instructions, red flag such issues, and push back deadlines to resolve them.
1. theatlantic.com/amp/article/602188/
+That's not strictly true, people can have different personalities at work vs family for example.
You don't need to be an engineer to do any of these things. You really just need to ask questions to people who understand the details and listen to them. A good manager will know the right people to ask these questions to and follow their advice. Good management is not an easy skill however.
Now what if the salesman was human. This human has a feelings and a personality and other things going on in its life. Now what if you have to manage 100 of these salesmen, all with different culture, religion, education and age... and when you put these salesmen together in teams, they all have a different group dynamic. Sometimes you'll get a salesman who will even try to sabotage you. Plus you need to motivate, train, and lead all of them. For some reason technical people think this problem is so easy. (I'm not an MBA by the way)
Also, this is buried further down in the article than it should be.
> Mr. Pierson and his lawyers declined to answer whether he was seeking whistle-blower protection or filing a federal whistle-blower case. He has hired a prominent whistle-blower lawyer, Eric Havian, and could stand to gain monetarily if he pursued such a case.
"Mr. Pierson believes that the production problems may have played a role in the crashes. In both accidents, MCAS was triggered when a vane installed on the plane’s fuselage malfunctioned."
"Workers were completing jobs out of sequence, leading to additional mistakes."
Everything this whistleblower is saying seems consistent with dozens of other stories which have emerged of Boeing's corroded engineering practices.
I've found over the course of my life that money encourages all sorts of people to say all sorts of things.
I mean, the concern would be that MCAS is just the tip of the iceberg, and that similar fuckups exist within the planes due to this management trouble.
There is some speculation in the article that the vane malfunctioned too frequently which may be rooted in production problems.
I doubt that Boeing would have been so gracious to Pierson if they had not already been waist-deep in shit.
"Boeing specifically highlighted in presentation documents the new MCAS compares inputs from two angle of attack (AOA) sensors as opposed to a single sensor in the original, flawed, design. MCAS will only activate if data from both sensors agree the AOA is too high. MCAS will activate only once per high AOA event. Pilots will be able to override it and the system will not activate automatically a second time. In the original design, the system reactivated over and over. The presentation reiterates the crew can deactivate MCAS by turning off the automatic trim system and manually turning the trim wheel."
https://imgur.com/a/H5milYP