148 comments

[ 3.0 ms ] story [ 250 ms ] thread
That answers the "why didn't they remember the airworthiness notice" question - they did, and tried to disable it, and it re-enabled itself.

Glad I didn't pick up Boeing shares on the cheap just yet.

> it re-enabled itself

No, the possibility that it reenabled itself is being investigated, but there's no reason to conclude that yet.

Amazing how easily “the possibility of X is being investigated” turns into “X happened.”
When there's a stake at the ready the bar for being a witch becomes lower.

People form a version of events in their head and then cling to it for dear life. This whole thing should also serve as a reminder why we don't rely on internet mobs to investigate important things.

> This whole thing should also serve as a reminder why we don't rely on internet mobs to investigate important things.

I'm more inclined to take it as a warning on regulatory capture, especially after it took most of the rest of the world grounding the craft to get the FAA to do so.

> Glad I didn't pick up Boeing shares on the cheap just yet.

Unfortunately, it's a focus on profits over people that likely led to these disasters. I don't know how you can stop or dissuade from people making a buck off of a tragedy, but it might be worth putting our attention on the human element of this mess. Something about merging this safety issue with an opportunity to "buy low" and make some cash just doesn't sit right.

Traditionally this is done by having a separate safety, QA or compliance organization (depending on industry) that has the power to overrule the bean counters and guys with white teeth.
That's how it works in the FDA regulated medical device and medical software industry. We had a woman who could shut us down on a whim. (And she would, from time to time.)

That said, in the FDA regulated industry, every change was signed by not only the engineer, but by the QA person, by the engineering manager, and by her. So in fairness to her, SHE would have been one of the people going to jail if the software told a doctor to take out the wrong kidney because of a bug in the calculation of an ortho normal basis or something. So I didn't blame her one bit.

Glad I didn't pick up Boeing shares on the cheap just yet.

Look at a six month graph, Boeing is trending up and the stock price is back to where it was at the end of January 2019. Yeah, gross.

Boeing is the very definition of "too big to fail." They'll be fine.
That's not what the article says. The most likely situation from the article is the pilots:

* disabled the trim motor

* realized they were unable to trim back to level flight without it, probably due to aerodynamic load on the stabilizer

* turned it back on

* MCAS was still enabled -- it can't be disabled directly -- and used the trim motor itself to nose-down more, making the situation unrecoverable.

I strongly suggest people skip this re-hosted article, and instead read the Reuters one here:

Title: Boeing software under scrutiny as Ethiopia prepares crash report

https://www.reuters.com/article/us-ethiopia-airplane-softwar...

I have numerous problems with Arstechnica's re-write, but suffice to say that nuance is lost and questionable conclusory statements added.

I will say in the case of BOTH articles, it sounds like we need to wait and see. Neither article has an answer for how auto-trim was re-actived, be it the pilots or something else. The answer to that question may have wide reaching implications.

This other HN submission has some interesting hypotheticals on why the system might have been reactivated multiple times:

https://news.ycombinator.com/item?id=19565056

This analysis sounds very plausible (all emphasis marks mine):

https://leehamnews.com/2019/04/03/et302-used-the-cut-out-swi...

"the high speeds observed in ET302’s FlightRadar24 trace (Figure 1) were logical. It’s a consequence of following the Emergency checklist for “IAS disagree” (IAS is Indicated Airspeed, i.e. the dynamic air pressure experienced by the aircraft) after takeoff.

The combination of the preceding checklist followed by an MCAS Trim Runaway checklist could create a situation where manual trimming after a Trim Cut-Out would be difficult to impossible and would require non-checklist actions."

"At a miss-trimmed Stabilator, you either have to re-engage Electric trim or off-load the Stabilator jackscrew by stick forward, creating a nose-down bunt maneuver, followed by trim.

Stick forward to trim was not an option for ET302, they were at 1,000ft above ground. According to The Wall Street Journal, the ET302 crew re-engaged electrical trim to save the situation, to get the nose up. It was their only chance. But too late. The aggressive MCAS kicked in and worsened the situation before they could counter it."

The Airport elevation of Addis Ababa airport is 7625 feet AMSL. The highest point of the flight recorded by the flightradar24 is 8600 ft AMSL which gives the plane's altitude of around only 1000 ft at highest! The speeds relative to the ground recorded near the end were 380 kt, i.e. 640 ft/s. Obviously every second mattered, and at the above speeds, the plane was only a few seconds from crash away when the MCAS was pointing the nose down.

Weird, I guessed at something like that in a related article submitted earlier:

https://news.ycombinator.com/item?id=19560596

Not quite the same -- the issue wouldn't be that they didn't crank enough, but that cranking was physically impossible due to the aerodynamic load on the stabilizer created by being at high speed with the elevator and stabilizer airflow opposing each other. You'd have to release the elevator in order to regain control of the stabilizer, and you can't do that without losing altitude that you don't have to give.
Leeham worked with MentourPilot, a training captain for Ryanair, on this article. There was a video included, but it's since been taken down. If you've seen any of Mentour's videos they are fantastic and he is typically quite good about remaining objective and avoiding wild speculation.
> The combination of the preceding checklist followed by an MCAS Trim Runaway checklist could create a situation where manual trimming after a Trim Cut-Out would be difficult to impossible and would require non-checklist actions."

This is a talk by Professor Nancy Leveson that describes exactly this type of problem. Dangerous interactions between systems. I posted this before when a lot of people were engaging in low grade racism by blaming the Indonesian and Ethiopian pilots competence and training. Seems very topical at this point tho.

https://www.youtube.com/watch?v=WBktiCyPLo4

> The highest point of the flight recorded by the flightradar24 is 8600 ft AMSL which gives the plane's altitude of around only 1000 ft at highest!

Not to be pedantic, but altitude is the distance measured from MSL, height is measured from the ground. So their altitude was 8600ft, their height was ~1000ft.

Thank you very much for the link.

If true, and more and more seems to point into this direction, we can pretty much lay the arguments of the "badly educated and overworked third world pilots who are too dumb to work a checklist" to rest.

I always found that reasoning rather bigoted and jingoistic.

It seems more that Boeing was selling a death trap certified by their cronies at the FAA.

What galls me most is the fact that they didn't ground that atrocity after the Lion Air crash, but used their energies to throw customers and pilots under the bus and developed a checklist, which, as it turns out, is actually counterproductive and does nothing to fix the inherent design problems of the plane. If anything, it serves the exact opposite.

The second crash is mass murder for commercial reasons, period. If not legally then at the very minimum morally.

> Neither article has an answer for how auto-trim was re-actived, be it the pilots or something else. The answer to that question may have wide reaching implications.

I can imagine that being irrelevant in this case: Even if the pilots activated it, it doesn't mean they must have made any error, see my other comment here for the details.

In short, if the pilots turned the switch off, turning off much more than just the MCAS (as there's no switch "just turn MCAS off, keep the rest on"), but then the plane was already not possible to save (it was only 1000 ft above the ground!) and they turned the switch back on, hoping for obtaining more power/possibility to trim, and the faulty MCAS kicked in again (despite being the reason for the switch cutoff before) and even sped up the crash, then the pilots in fact did absolutely all that was possible to do.

And I can even imagine that the code which included the promised "fix" by Boeing until just a few days ago still had the built-in "obvious to implement by default" logic: "oh, I'm booting, now I can point the plane to the ground again because the sensors say so and I'm completely ignoring the fact that the pilots already turned me off shortly before" and that that was the reason the fix release is now delayed. But also note that that plane, the way it is constructed, certified and sold simply needs the MCAS and can't reliably fly without it, otherwise Boeing's fix would have been turning the feature off completely.

"...if the pilots turned the switch off, turning off much more than just the MCAS (as there's no switch "just turn MCAS off, keep the rest on"), but then the plane was already not possible to save (it was only 1000 ft above the ground!) and they turned the switch back on, hoping for obtaining more power/possibility to trim, and the faulty MCAS kicked in again (despite being the reason for the switch cutoff before) and even sped up the crash, then the pilots in fact did absolutely all that was possible to do..."

If this is indeed what is announced, I sincerely hope the responsible parties for making the decision to go through with MCAS in this way are paid well in jail time.

> If this is indeed what is announced, I sincerely hope the responsible parties for making the decision to go through with MCAS in this way are paid well in jail time.

I'm sorry, but your wording sounds like you're advocating taking revenge on engineers / managers for poor decisions made in good faith. That's barbaric, and I hope we as a society move beyond this sort of communal revenge. We'll all be psychologically better off if we encourage more healthy ways of seeking relief from tragedy.

Such a proposal would mean no more new airplane designs. Note that modern airliners are much safer than their predecessors.

An awful lot of mistakes are only "obvious" in hindsight; engineering is full of such.

> Such a proposal would mean

But who actually proposed what here?

I don't think there is a hindsight here. Boeing made false statements when certifying the plane.
> but your wording sounds like you're advocating taking revenge

No.

It's you, KMag, who is creating a straw man against SolaceQuantum.

SolaceQuantum's actual wording is using "paid well in jail time." Which means the expectancy of the involvement of the judicial system (courts, lawyers etc.) which could also eventually produce such a sentence (jail time).

And that is certainly not "advocating taking revenge."

SolaceQuantum also never mentions "engineers." It's again your straw man construction and a false claim.

It's already clear that in this case the "fox was guarding the hen house" (As reported in (1)): "The FAA, citing lack of funding and resources, has over the years delegated increasing authority to Boeing to take on more of the work of certifying the safety of its own airplanes." Note: FAA "is a governmental body of the United States with powers to regulate all aspects of civil aviation." Also note, just as an example: "The FAA has been cited as an example of regulatory capture, in which the airline industry openly dictates to its regulators its governing rules, arranging for not only beneficial regulation, but placing key people to head these regulators." (3)

1) https://www.seattletimes.com/business/boeing-aerospace/faile... )

Also (2):

2) https://www.seattletimes.com/business/boeing-aerospace/senat...

3) https://en.wikipedia.org/wiki/Federal_Aviation_Administratio...

All that is certainly not about the "engineers." And the Senate Committee already investigates.

> Which means the expectancy of the involvement of the judicial system

Mob vengeance intermediated by the state is still mob vengeance.

> SolaceQuantum also never mentions "engineers."

The decision makers certainly included management and engineers, and the OP was ambiguous about which decision makers should be punished. I think my statement is just as true if amended to just read "management".

> It's already clear that in this case the "fox was guarding the hen house"

A conflict of interest is not proof that one is making decisions in bad faith. We should certainly strive to eliminate conflicts of interest.

On the other hand, to jump to the conclusion that decision makers are likely deserving of prison time shows a lack of perspective and empathy. I'm sure a lot of people at Boeing will carry a lot of guilt with them for the rest of their lives over this.

Maybe there were decisions made in bad faith, but I think we have far far too little information (and too much hindsight bias in the balance) at this point to determine if those making poor decisions were making them in bad faith, even if there was a clear conflict of interest.

It's comforting to believe that most tragedies are the result of evil people making decisions in bad faith, and that most tragedies can be avoided by identifying and punishing these evil people. The real world is seldom so black-and-white, and I strongly oppose our society's common knee-jerk reaction to quickly reach for imprisonment as the solution to the latest tragedy. By all means, punish people who make conscious decisions to harm their fellow humans, but many tragedies don't fit this simple narrative.

How exactly would this classify as mob vengeance?

Negligence in critical safety infrastructure is bad faith.

The OP said that if it's true that disabling MCAS made it impossible to trim the plane without losing too much altitude, that they hope those responsible for the decision to "go through with" (deploy?) MCAS are jailed.

The poor design of the system is directly stated as the reason the OP hopes people are imprisoned.

I'm al for an investigation. Publicly calling for jail time so prematurely, in a case that would surely be highly publicized and politicized, makes it even harder to conduct an impartial investigation and trial.

When the public puts pressure on the courts to hand down a predetermined verdict and punishement and they are successful in their efforts to bias the process, that is mob vengeance.

> punish people who make conscious decisions to harm their fellow humans, but many tragedies don't fit this simple narrative.

Causing death of 350 people due to "intended" ("it's good for our business") negligence is enough to deserve an evaluation of the decisions made and the evaluation if the punishment is warranted (including the "jail time"). Especially the Boeing's top executives behavior after the first crash can't be defended, given everything we already know.

I'm all for an investigation. I'm very much against public calls for jail time before we've been able to properly investigate the actions and intentions of the parties involved. The OP's stated justification for jail time is "[purely technical description of system functioning] ... if this is true, I sincerely hope [people are jailed]".

My basic point is that when ther's a tragedy, there's a knee-jerk reaction to punish people regardless of mens rea. Putting pressure on courts for predetermined outcomes, particularly without regard to intent of the accused, is the slightly civilized version of getting out the torches and pitchforks.

Deterrence requires interrupting the ill-intentioned thought process. If there are no ill intentions, then jail time has no deterrent effect. We may feel better if someone is punished, but if the public's feelings are the only positive outcome of punishing someone, then punishing them is barbaric. (Note that restitution is separate from punishment. Restitution, if possible, is still called for if there's no ill intent.)

No, not barbaric, that's the liability attached to all normal activities.

"Knew or should have known" is the standard. Boeing management knew or should have known that selling an essential safety system relying on a single sensor is categorically unsafe.

All blame to management unless the programmers knowingly miscoded the software.

(comment deleted)
Boeing deserves to get nailed for this. You can't tell me they didn't know it was the MCAS at fault after the first crash 6 months ago, and decided to do...nothing.
From a legal standpoint, shipping a "fix" could be used as evidence of guilt, though.
Things like that is why we have such insane damages dollar amounts for punitive damage. People reward huge punitive damages since the Ford Pinto case to compel companies to "do the right thing" because if you don't and you're found guilty, the punitive damage can be many times what the original cost would have been.
I've been saying that software engineering as an industry is in the caveman state of development. The amount of scrutiny that production software gets at _most_ companies is hilariously lacking compared to say, a civil engineering blueprint and there are virtually zero repercussions for failure or negligence. Compare the education & certification required to be called a civil engineer vs. a software engineer and you'll notice that the industry is basically the wild west. You could also argue that computer science isn't yet a "hard science" like mechanical, civil, chemical, etc. engineering are.

I predict that there will be a split in the industry between "software engineer" and "software laborer" where there are specific certifications you need to have before being called an "engineer". "Software engineers" will be held responsible, like civil engineers are, for mistakes that result in accidents. Companies that write safety critical software like auto, air, and space industries where human lives or huge $$$ are on the line will hire engineers, e-commerce companies will continue to hire "code laborers".

Maybe they will even give you a fancy ring to wear?

It just depends on the cost of a mistake, you don't need people trained at that level for web dev. I can do data science coding, but I wouldn't trust myself to write Anything where failure could kill someone.
A person should never be trusted for this. Airplanes are more reliable because the processes (and tech) are improving, not the people.

The problem here was that the process got too expensive, so Boeing was pressured into skipping it.

I get your point, and I agree largely, but there is still some element of the people. I wouldn't want someone like myself, who is trained on python and self-taught, to be responsible for any of these components.
The software that runs safety-critical flight controls is actually developed nothing like typical ecommerce or social media. It is designed and developed to an engineering standard. You can review DO-178C requirements[1].

Not all software has the same consequences for failure. The highest level is when failure is catastrophic, i.e. "Failure may cause deaths, usually with loss of the airplane." This requires the highest "Level A" assurance level.

Presumably MCAS was not originally evaluated as needing to meet the highest level. If it were, the single AoA input would have immediately disqualified the design right off the bat.

There's some speculation that the delays on the "fix" are because if they have to redevelop the software at a "Level A" standard that means basically a complete reimplementation with full tracability of requirements from design to source code through compilation to executable, with independent verification. That won't happen in a few weeks.

[1] https://en.wikipedia.org/wiki/DO-178C

Do _most_ companies follow DO-178C requirements? Do _most_ software engineers even know what that is?
Companies that develop aircraft control software do. I read your comment as implying that they don't.
> Do _most_ companies follow DO-178C requirements?

No, because most companies aren't developing life-critical software.

I would imagine most companies that are developing life-critical software are familiar with those requirements.

> designed and developed to an engineering standard

Hmmmm. Interesting. Is it legally required to use this standard for these critical softwares?

And are developers legally required to be licensed, in the same way that civil engineers are?

Yes, the FAA must certify the plane as airworthy for it to be legally flown, and the FAA does use this standard's criteria.
It looks like Boeing submitted MCAS at DAL C, using the rationale that it only mutates the stabilizer angle by a few degrees, so it can't be catastrophic. This was a terrible argument for two reasons:

1) It mutates it a few degrees at a time in a non-terminating loop, so it effectively has control authority of the entire stabilizer.

2) Even DAL C would require using more than a single input!

(comment deleted)
> I've been saying that software engineering as an industry is in the caveman state of development.

I once bought a domain called "paleolithic-computing" (neolithic wasn't available, even though I used the stonehenge picture), to use as a blog.

We are indeed in the caveman stage. In fact, in some ways, we seem to have regressed. There is a pressure to release features faster and faster, and everyone knows you can achieve anything in a sprint, right? Then just get 'devops' to deploy.

It helps that most other (or proper) engineering disciplines have been around for far longer. We are not even comparable to egyptians building pyramids, we are still figuring out how to build huts (and rebuilding them every few years with new tech).

Give it time.

This speaks to me. You said this much better than I did. What bugs me about a lot of developers is that they can't see the forest from the trees.
A lot of these serious professions are independent firms that are appointed by a client. I think this can create a different set of incentives compared to working directly as an employee. The firm will be less dependent on the clients success than a direct employee. Your income depends on the quality of your work rather than one single project. And you are more able to see bias in decision making when you are outside of the clients corporate culture. Of course this probably falls down with very large contracts.
On the other hand, being a contractor also means you don't need to be as invested into the maintainability. If it gets too bad you can simply stop working on that project. You don't need to select for solutions for the long term.

Not saying it's strictly that way, just different trade-offs.

Do not conflate software development practices with anything to do with computer science.
This, to me, highlights a deeper issue. The fear of legal ramifications shouldn't get in the way of creating better, safer products. If Boeing is avoiding taking the heat by withholding a fix it may cost more people their lives.
They weren't doing nothing. They've been working on a software fix since the Lion Air crash, but it wasn't ready by the time of the Ethiopian crash. The plan was to update some checklists as a stopgap and get new software ASAP. Obviously in hindsight they should have grounded the fleet until the new software was ready. At the time that would have been a very difficult decision to make internally at Boeing, given that the 737MAX has $250 Billion worth of orders, and arguably at the time, the updated checklists made a crash unlikely. This is the situation where an independent regulatory body is most important. The commercial pressure to keep flying was almost irresistable, and you need someone without those commercial pressures to put the brakes on.
commercial pressure should have been the liability exposure they find themselves in now.
The problem is this: From the CEO's standpoint, grounding the fleet might have cost him his job given that there was a plausible (even though we know now ineffective) checklist based solution to the problem. Another crash might also cost the CEO his job, but that has a probability of less than one. The incentive in this situation is to roll the dice. The problem is that the decision maker does not bear the liability for his actions. He's not the one making a smoking crater in the ground. That's why you need a regulator with different motivations. Historically, the FAA has been exceptionally effective at improving air safety. The Republican/business argument is that they have overdone it and the costs of compliance exceed the benefit at the margin. These accidents are a strong argument otherwise.
Or, you know, someone who values human lives more than money.
Statistical value of human life. Learn it , breathe it.
Ok mr expert, please tell me exactly how much these two crashes have & will end up costing Boeing.
You risk your life and the lives of others for money every time you drive to work.
We actually do put a monetary value on human lives in a huge variety of situations. We just like to pretend that we don't and try not to talk about it.

For example, most health insurance systems around the world have a value they place on one year of "quality" human life, and won't cover medical treatments that cost more than that amount per year. In 2008, that number was about $50k [1]. I don't know what it is now.

When considering whether the monetary costs of a regulation are worth it in the US the EPA used about $9 million as the worth of a life saved in 2010, and the FDA used about $8 million. The DoT used almost $10 million in 2016. Cites for those are in this Wikipedia article [2].

[1] http://content.time.com/time/health/article/0,8599,1808049,0...

[2] https://en.wikipedia.org/wiki/Value_of_life

Oh I'm sure they weren't literally doing nothing. I imagine meetings took place and they decided to roll the dice in not grounding the fleet before the software update was ready. But they rolled snake eyes, time to pay the piper.
> The commercial pressure to keep flying was almost irresistable, and you need someone without those commercial pressures to put the brakes on.

I fully agree (even expanded to all fields/areas/industries/activities - but let's try not to over-regulate everything, if possible :) ).

I hear sentiments like this a lot as if everything in the universe is over regulated.

What do you believe are some examples of things that are over regulated?

What is worse, this 737-MAX issues appears to have been caused by under regulation, which has subsequently resulted in the needless deaths of many hundreds of passengers.
> What do you believe are some examples of things that are over regulated?

Ok, this is a very good question.

Thinking about it, I cannot think of any area I know about that is regulated, as a baseline, too much for my taste.

BUT:

E.g. 2 years ago I thought that the logs of my app (whiĉh is SOX-relevant - https://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act ) weren't nice, so after talking to the other team members I slightly changed them to e.g. align strings (using tabs instead of fixed chars) and posting more informative/detailed messages.

When the auditors performed their review they saw that the new logs did not match the old ones (aaargh - string mismatch!!!) therefore we had to go over each change, checking why-it-was-done-by-whom-based_on_which_request_and_did_it_get_the_approval_to_go_live_and_was_it_reviewed_and_signed_off_by_business_after_go_live etc..., so an infinite pain. This is what over-regulation means technically for me :)

I'm not trying to be a jerk, but you asked that we "not over-regulate everything" and you couldnt come up with a single example of over-regulated things and then brought up one issue in an app you made as why you brought up that "everything" is over-regulated?

This logic makes me really sad. You're complaining about overregulation when under-regulation just killed 300 people because you had to make a few changes in an app once. Please think about this more deeply than, "I know a few hundred people died, but lets not respond too harshly and have me spend a few more hours on my apps".

Please do that next time.

In "Regulation of Pharmaceutical Innovation" Sam Pelzman shows with statistics that the increased safety of drugs is more than counterbalanced by the commensurate delays in approval of life saving treatments, along with new treatments never developed because of the cost of the approval process.
Sorry, I think that I don't understand meanings of the sentence => could you pls split it into smaller/simpler sections?

Thx :)

Costly regulations slow down or prevent life-saving drugs from being developed.
In reality it's the financing model that works against new life-saving drugs. Goldman Sachs says it's the financing model. I'm inclined to believe them instead of a neocon think tank.
I've read that in the US you need a license to cut people's hair for money. That seems like over-regulation to me.
Just because I'm pretty stunned. The answers to the above question of "what are some things that are over-regulated?" in response to "everything is over-regulated" are:

From the OP: "I CANT THINK OF ANYTHING"

"Haircuts seem overregulated"

"The drug approval process might move too slowly"

So on one side we have "planes not crashing", "electrical codes being followed", "air quality not as bad as in Shanghai".

On the other side we have "haircuts are over-regulated".

Hmm, not sure which side I'm on here...

> At the time that would have been a very difficult decision to make internally at Boeing, given that the 737MAX has $250 Billion worth of orders

That is exactly right and exactly why they need to be nailed for this.

No hindsight is required to know Boeing was focused on making sure the welfare of Boeing and it's shareholders was taken care of, with little or no regard to the welfare of the flying public.

It's not that simple. There's no denying that there was tremendous commercial pressure, but ... from a single data point (prior to the second crash), it wasn't obvious that the risk of continuing to fly the aircraft was greater than grounding the planes, which results in cancelled flights and greater risk of road deaths due to some passengers taking road alternatives instead of re-booking flights. Also, the 737 MAX is a longer-range aircraft, and some of the slack would be taken up by capacity in shorter-range aircraft, converting direct flights to multi-leg flights, where most of the risk is in takeoff and landing. All else being equal, more flight legs means more risk. Some of the slack would be taken up by smaller regional airlines flying older planes, and I'm not sure how those safety records generally compare to larger airlines flying newer planes. In any case, I expect the majority of the offsetting risk would be from people deciding to drive when their flights got cancelled.

Now that we see that training and procedural changes appear to be a less effective mitigant than expected, we can say that with a high degree of certainty that the risk of increased road deaths does not offset the risks of continuing to fly the aircraft.

> from a single data point (prior to the second crash), it wasn't obvious that the risk of continuing to fly the aircraft was greater than grounding the planes

Call me a skeptic but I have no doubt Boeing was well aware of the issue with the MCAS well before that first data point.

Even before that first crash, there have been reports in the media that claim many pilots had complained about the dangers of the MCAS.

> Even before that first crash, there have been reports in the media that claim many pilots had complained about the dangers of the MCAS.

I don't think this is true - the newspaper that first made this claim overstated their findings. Pilots, in general, did not know about MCAS before the Lion Air crash.

This has very little to do with the existence of the MCAS but instead was the plane actually safe to fly and when did information come to hand to show that it was not.

Consider this link:

https://www.abc.net.au/news/2019-03-13/737-max-pilots-report...

From that link:

Airline pilots on at least two US flights have reported that an automated system seemed to cause their Boeing 737 MAX 8 planes to tilt down suddenly.

That reporting suggests problems were known well in advance of the first crash.

> and greater risk of road deaths due to some passengers taking road alternatives

Such a consideration (while valid) is not Boeing's to make.

Almost no-one decides they'll just drive instead of flying because their plane was delayed.

After the first crash, Boeing certainly knew that it had hooked up MCAS to a single AoA vane. I don't think they could sincerely argue that hooking up a single sensor to a primary control surface with no backup sensor is airworthy -- either practically or legally. There are regulations about it, and that's why Airbus has three AoA vanes and uses majority quorum. That's why I think the grounding should have happened earlier.

If these crashes are both from the same system then they effectively did nothing. The only I'd accept that they didn't do nothing is if they intervened and stopped the problem. Still waiting till the dust settles to take my opinion tho.
I read something about how they did have a fix, but the government shutdown prevented it from being deployed in time.
Focusing & expanding your "deserves":

I guess that I agree at least partially with you (even without the 2nd incident, for which no official facts have been presented - but therefore I cannot upvote you as there is no confirmation yet, sorry), thinking about the facts that e.g. MCAS is not intuitive to understand/use for the pilots and that it relies on only 1 sensor (at least the original version) (any other flaw?).

About the consequences:

I just hope that the whole investigation won't boil down to just some "crazy" engineers that implemented that system because they were having a bad day or had no clue about what they were doing etc... .

Top-management must have known (at least implicitly/indirectly) the risks associated to a tight deadline and/or a miracle-like re-engineering of an old airplane and/or not having airplane-hackers available during the re-engineering phase and/or etc... because one of the roles of top management is to ask questions when something out of the ordinary happens especially when A LOT of money is involved.

Of course if top-mgmt puts enough pressure to achieve something, no questions will arise from both ends (and here we can probably optimistically go into the area of whistleblowers :) ).

I think that the same thing can be applied to Volkswagen (e.g. impossible that some engineer came up with "and here we have a Diesel engine outputting half as much noxious gas as everybody else on the planet" and nobody else asked questions because the mgmt wasn't pressuring those departments to get exactly such results) and other previous scandals.

Note to self: eventual consistency good for web/mobile apps, bad for avionics.
Depends on the app. There are a lot of web / mobile app use cases where stronger consistency is needed, e.g. banking.
You'd think, but banking (e.g. the ATM network) is all about choosing availability over consistency. If the ATM network can't reach your bank to discover your balance, it'll just let you withdraw and figure your bank will sort things out later after it comes back online.
I just had a 737 MAX fly low almost over my house [1]. That descending left turn at around 2300 ft was centered on roughly my neighborhood. (The flight track shown on Flightware is displaced to the NW a bit).

[1] https://flightaware.com/live/flight/BOE1/history/20190403/17...

I thought they were grounded.

Edit: Checked flightaware, it was Boing itself flying it.

Only from commercial service
Sort of. All flights in U.S. territory, and by U.S. airlines anywhere in the world, are banned. Non-passenger flights can be authorized but they require either a Special Flight Permit (a specific process for authorizing ferry flights of any non-airworthy aircraft) or an Experimental Airworthiness Certificate (which you may know of from homebuilts, but in this case it's for testing design changes).[1]

1. https://www.faa.gov/news/updates/media/Emergency_Order.pdf

Boeing's flying test-flights to test software fixes etc out of their Seattle facility
Southwest Airlines was allowed to fly all of their grounded jets to a facility in Arizona a few weeks after the initial grounding. The FAA allows this as long as there are no passengers on board.
That one was a Boeing 737 MAX 7 "Boeing 737 MAX 7 (twin-jet)".

Isn't it only the 737 MAX 8 + 9s that are grounded?

According to this article it is just the 8s and 9s.

> The United States will follow much of the rest of the world in grounding Boeing 737 Max 8 and 9 jets after the models were involved in two deadly plane crashes in five months.

The Boeing 737 MAX 8s and 9s are the ones with the bigger engines and single sensor MCAS that is essentially a single point of failure.

The 737 MAX 8 and 9 are essentially new planes, they should have just made new versions but they wanted to squeeze this into existing certifications and training to cut costs for Boeing and airliners. No idea why the FAA allowed them to do this. It has some blowback for Boeing as now people think all 737s are bad.

[1] https://www.vox.com/policy-and-politics/2019/3/13/18264348/t...

Isn't it only the 737 MAX 8 + 9s that are grounded?

Only 8s and 9s have been produced so far. I'm not even sure that anyone other than Southwest has ordered the MAX 7 (it's less of an improvement over the -700 than the 8 and 9 are over the -800 and -900/-900ER). There are no big conspiracies here just no sense in grounding planes that don't exist (or haven't been delivered).

https://www.planespotters.net/production-list/Boeing/737/737...

https://www.planespotters.net/production-list/Boeing/737/737...

Indeed, only max 8 and 9 are grounded afaik
Because no customers have MAX 7s and thus no MAX 7s could possibly be in revenue service with passengers. If Southwest tried to take delivery of and operate one of their MAX 7s in revenue service you can bet the FAA would have a conniption fit.
That's a MAX 7. Only the MAX 8 and MAX 9 are grounded. See the FAA order here: https://www.faa.gov/news/updates/media/Emergency_Order.pdf

> Effective immediately, this Order prohibits the operation of Boeing Company Model 737-8 and Boeing Company Model 737-9 airplanes by U.S. certificated operators. This Order also prohibits the operation of Boeing Company Model 737-8 and Boeing Company Model 737-9 series airplanes in the territory of the United States.

So the physical cutoff switches are still just inputs to software? WTF.
If that's true, it's inexcusable. But we should wait to find out more information.
No, the working theory I've heard is that the hand-crank trim wheels may have been overwhelmed by the air load on the stabilizer given its "nose down" configuration, probable full "nose up" pull on the elevators, and the very high speed they were flying. Speculation is they may have turned the electric trim back on to use the manual trim switches to trim up. By that time possibly too late or even the electric trim was overwhelmed by the load, and of course this would have brought MCAS back into play also.
Again, instead of relying on speculation, we should wait to get more information.
Agreed. Hard not to speculate though. I personally find this stuff fascinating from a technical and human factors perspective -- the deaths are tragic of course.
This isn't rumor mongering. This is straight out of Boeing's own documentation on the 737: https://www.satcom.guru/2019/04/stabilizer-trim-loads-and-ra...

When the stabilizer is extended and is being counteracted by elevator inputs, the forces on the control column make manually trimming stabilizers difficult or impossible. Boeing instructs that elevator input on the stick must be eased back in order to manually adjust the stabilizers.

Boeing documents possibilities, not what actually happened.
Yes, and?

If you object to people discussing the technical details of a plane w.r.t. the observed behaviour of that plane in an accident prior to the final report on the accident being released might I suggest....not clicking on the comment section, rather than trolling the comments on news items complaining that people are using the website for its stated purpose: to discuss the news being linked to?

And the news article we're commenting on raises the possibility that this is exactly what happened. If that doesn't make the hypothetical germane to discuss here, what can we talk about?
Why? I tend to enjoy informed speculation, that's one of the reason I visit this website.

Isn't speculation one of the things that characterizes us as humans?

This makes more sense, thanks.

I still don't understand why the trim is "stronger" on these planes than a fully pulled back column. Is this necessary on large planes?

probably mostly due to needing to have a larger operation margin for different center of gravity as load changes depending on the current flight passenger/cargo loadout.
Mentour Pilot (as others have mentioned in various other comments) does a great job explaining this.

On the 737 the trim adjusts the entire rear stabilizer (the entire horizontal structure of the tail), while the elevator is a sub-component of that stabilizer.

So, when you adjust the trim, it's a much larger surface area that is moving. Ultimately, the stabilizer just has more aerodynamic control that the elevator, due to the larger surface area.

I have no idea why the design is that way, though.

I have no idea why the design is that way, though.

Cost, maintenance, weight, etc., probably. The only jet airliner I can think of that uses an "all-flying tail" is the Lockheed L-1011. Nearly every other plane you'd be riding on will have a big stabilizer and a smaller elevator.

Redundancy too.

If both redundant hydraulic systems to the elevator fail, or the elevator gets stuck for other reasons, the emergency procedure is to control pitch with just the stablizer trim alone.

Presumably you could have a redundant drive system for the combined stabilizer/elevator. The L-1011, for instance, had a safety record Boeing should be envious of.
So they'd have to let the plane nose down towards the ground for a bit to then be able to trim back up. This sounds like a cascading failure to me.
Exactly. And Ethiopian was less than 1000ft above ground level at all times. They didn't have the altitude to trade.
After watching Mentour Pilot's demo correcting for runaway trim in a 737 simulator, it seems like a switch that cuts out computer (autopilot, auto-speed, MCAS) control of the elevator trim motors, but keeps the column switches working would be nice. Consider how cumbersome & slow the manual wheels are: https://youtu.be/xixM_cwSLcQ?t=1129

It would be one more layer of complexity, but perhaps when the column trim switches are operated, all computer control could stop until it (autopilot, etc) is re-enabled by a human. The cutout switches in the center are still there if two column switches short out (used to be a single up-down switch, but it was single point of failure to runaway so now there are separate enable and direction switches).

There are switches for auto pilot and auto speed. MCAS is disabled with flaps extended. It's not procedure but had they set the flaps to a single degree it would have saved them. I think the Lion flight experimented with flaps briefly.
This is sensible, but recall that Boeing is still claiming that the MAX flies like any other 737 and requires no special training. Adding a new button would require training in the use of that button.

(In case it's unclear, I don't support Boeing's position here.)

The cutoff switches actually disconnect the power to the trim motor, so this is unlikely unless Boeing changed something else they've told no-one about.

More likely is that the pilots turned the electric trim back on because as the aircraft gained airspeed (a normal thing when dealing with unreliable airspeed indications), the manual trim was not effective enough. This has been tried in the simulator:

https://leehamnews.com/2019/04/03/et302-used-the-cut-out-swi...

Generally, the manual trim and elevator can both overcome a nose-down trimmed stabilizer at the speeds seen just after takeoff. But if you don't rapidly trim back, and you let the airspeed build, you're now in a tricky situation. The manual trim is now too hard to move, increasing back pressure on the yoke is needed to counter the trim, and you don't want to throttle back, because the pitch up from the engines is all that's keeping you from diving. At this point, really your only course of action might be to re-enable electric trim.

This does lead to the worrying question as to whether a runaway nose-down trim in an older 737 just after takeoff could result in the same outcome.

This does lead to the worrying question as to whether a runaway nose-down trim in an older 737 just after takeoff could result in the same outcome.

Sure could but there are parts of the MAX that make it worse. The uncommanded stab trim movement on the MAX came about because of faulty angle-of-attack data. The bad AoA data meant that the pilots were faced with a "indicated airspeed disagree" warning (and on at least one of the Lion Air flights an "elevator feel system non-op" warning). The supposition here by MentourPilot is that the IAS disagree checklist dictated that the pilots increase their speed to the point where trimming manually is difficult or impossible. On an NG or Classic I don't think there's anything that would cause both IAS disagree and uncommanded stabilizer trim.

I agree that on the NG, IAS disagree and an uncommanded stabilizer trim are much less likely to occur together. But if you did have an uncommanded nose-down trim close to the ground, you still wouldn't want to reduce throttle because that would cause a pitch-down, so it's possible you'd still end up in the same place. You'd have fewer distractions though, I agree. Anyway, as far as I know, it's never happened, and the NG has a generally good safety record.
More to the point IAS disagree and uncommanded trimming happening from the same, single failure is new to the MAX (I think). If the same thing happened on an NG it would be due to multiple failures (which is both more terrifying and less likely).
Prior to MCAS, most causes of trim runaway could be stopped by countermanding it with the control column, but, because MCAS is intended to avoid a pilot-commanded (though presumably accidental) stall, this is not the case for MCAS:

"[MCAS intervention] can be stopped by the pilot counter-trimming on the yoke or by him hitting the CUTOUT switches on the center pedestal. It’s not stopped by the pilot pulling the yoke, which for normal trim from the autopilot or runaway manual trim triggers trim hold sensors. This would negate why MCAS was implemented, the pilot pulling so hard on the yoke that the aircraft is flying close to stall." [1] [my emphasis.]

So it seems we may have several issues here:

1) Prior to MCAS, trim stab cutout was almost the last-ditch option against trim runaway (there was one more: grab the trim wheels and physically prevent it), but it became the only way to deal with an MCAS-commanded runaway (other than grabbing the trim wheel.)

2) Boeing did not reveal this prior to the inquiry into the Lion Air crash, despite the fact that it introduced a new experience of trim runaway that differed from what pilots had been trained to expect.

3) The Boeing / FAA response to the Lion Air inquiry was an Airworthiness Directive that basically said "pay attention to what we said before about handling trim runaway" (though, whether or not Boeing / FAA made a point of it, pilots may well have become aware of how MCAS runaway differed from their prior training -- that seems to be the case for the Ethiopian Airlines captain, who did use the cutout switches.)

4) No one seems to have connected the dots to realize that one cause of MCAS failure (AofA sensor failure) would also cause other problems, for which the prescribed mitigation procedure made dealing with the consequent trim runaway very difficult, especially when close to the ground.

I would be interested in learning, from one of the 737 pilots here, how frequently the 'grab the trim wheel' situation is presented in simulator training.

[1] https://leehamnews.com/2018/11/14/boeings-automatic-trim-for...

> The manual trim is now too hard to move, increasing back pressure on the yoke is needed to counter the trim, and you don't want to throttle back, because the pitch up from the engines is all that's keeping you from diving. At this point, really your only course of action might be to re-enable electric trim

That doesn't jive with the Indonesian Air crash or how MCAS works. MCAS makes repeated minor adjustments and the pilots can compensate for them. Which the Indonesian pilots did 20 some times before inexplicably stopping. You can see it pretty clearly in this image:

https://spectrum.ieee.org/image/MzE4Mjg2OQ.jpeg

So I don't see how the Ethiopian air gets into a situation where they can't manually trim. They should have been able to get back to a neutral stabilizer position, disabled electronic control, and manually trim from there.

So I don't see how the Ethiopian air gets into a situation where they can't manually trim.

The Ethiopian flight was flying dramatically faster than a typical flight at that altitude would. If they followed the IAS disagree checklist that would explain the increased air speed as the general idea is to give the airplane enough thrust avoid a stall until you figure out which side has the correct air speed. Higher airspeed = more difficult to manually trim.

They should have been able to get back to a neutral stabilizer position, disabled electronic control, and manually trim from there.

They were barely 1,000 ft above the ground (if that). You could easily lose 1,000 ft fiddling with the elevator to unload the stabilizer to manually trim it into a safe position.

The Lion Air crew had far more altitude to work with (they were at 6,000-7,000 ft at their highest IIRC).

FWIW I think whoever's downvoting some of these comments is really hindering the discourse here. I think what you've posted is factually wrong, but not worthy of a downvote because it mostly seems to be wrong due to missing pieces from the puzzle (and not out of malice).

>Higher airspeed = more difficult to manually trim.

The point, which I think you missed, is that you don't need to manually trim. Look at the image I linked to. It's MCAS down 2.5 degrees. Pilots up 2.5 degrees. MCAS down 2.5 degrees. Pilots up 2.5 degrees. It's a repeated cycle of the pilots getting trim to where they want it and the MCAS kicking in to nose them down.

I suppose if you wait until the MCAS has trimmed all the way you might find yourself in an unrecoverable situation. But I'm not sure how that would happen and doesn't reflect what we saw in the Indoneisa Air crash.

The point, which I think you missed, is that you don't need to manually trim. Look at the image I linked to. It's MCAS down 2.5 degrees. Pilots up 2.5 degrees. MCAS down 2.5 degrees. Pilots up 2.5 degrees. It's a repeated cycle of the pilots getting trim to where they want it and the MCAS kicking in to nose them down

Yeah that's what the Lion Air crew did, and it worked until it didn't (lost control in a turn?). But counter the trim with the push buttons and then hit the cutout is not in any Boeing QRH (yet?). As a non-pilot this seems like a reasonable option, but there may be something glaringly obvious as to why it's a bad idea. Even if this is the ultimate solution it's quite different to how you'd react in an NG and will come with extra training. Basically a bad situation all around.

If I've parsed everything appropriately the Ethiopian crew did try to turn the stab trim motors back on presumably to regain control over the stabilizer and we saw how that worked out. There's just not a lot of room for diagnostic work at that altitude.

>Yeah that's what the Lion Air crew did, and it worked until it didn't (lost control in a turn?)

IIRC the pilot handed control over to the copilot so that he could investigate the issue. The copilot stopped counteracting the trim.

Part of the reason that the MCAS got passed regulators is that it was not given enough authority to override the pilots. It can send you on a wild ride, but it can't crash the plane unless the pilot doesn't counter act the trim when the MCAS is paused.

IIRC the pilot handed control over to the copilot so that he could investigate the issue. The copilot stopped counteracting the trim.

From what I saw on that graph someone was trimming up as the plane nosedived. Take a look starting at the 23:30:53 mark. Someone was hitting the trim up button and MCAS was continuing to input trim down until the crash. The big problem with the preliminary report is the X axis, IMO. It's different on the graphs between the two flights, and there's not enough precision... but to me it looks like either hardware failure or the FO was simply not inputting enough trim up. They fought that plane all the way into the ground.

Part of the reason that the MCAS got passed regulators is that it was not given enough authority to override the pilots.

From what I've read MCAS as described to the FAA (relatively low limit on the down trim) was different than MCAS as implemented (about 4x larger movements allowed). Additionally it's not clear if the repeated down trim was by design or by bug.

I think that even MCAS in its current state is probably something most pilots could recover from as long as they know what to expect. The way MCAS works now (both in terms of the amount of authority as well as the misdirection from "unrelated" warnings) combined with zero documentation is a recipe for additional crashes.

> They should have been able to get back to a neutral stabilizer position, disabled electronic control, and manually trim from there.

Yes. But that's not what the checklist in Boeing's MCAS bulletin said.

It said, disable the electric trim motor then trim manually.

In hindsight, prehaps it should have said "return stabilizer to neutral then disable electric trim motor".

I wonder if Boeing did that deliberately, because a bulletin with a modified checklist would have undercut Boeing's implied narrative that a procedure already existed and it was the Lion Air pilots fault for not using the existing runaway stab trim checklist. Or did Boeing simply not test the procedure in a simulator?

I'm definitely interested in seeing the details. IIRC, the jack screw was found fully extended. I wonder if that's recoverable with only 1,000ft of height even using electric trim.
You might be onto something. I went and took a look at the the Lion air data:

https://spectrum.ieee.org/image/MzE4Mjg2OQ.jpeg

They were successfully doing trim up commands for at least 21 cycles to counter the MCAS trim down commands. Then after the last few activations, they try pressing trim up again, but the button presses are shorter and the stabiliser barely moves up.

Did MCAS push the stabiliser down past a point the aerodynamic forces were so strong that even the electric trim motor was having problems trimming up?

That could explain why the last 4 trim up commands were so short, if the pilots heard a large electric motor stuck noise they might have stopped holding the button.

On the 757, they bypass the software and cut off the electrical signal to the drive system.
You can't fix a hardware problem with software.
when's the press conference?
Pure handwaving speculation here, but this incident is feeling like a result of the limitations of fly-by-cable - specifically taking the tried and true design of fly-by-cable, but augmenting it with motors in parallel to human control.

System integration is a known common source of errors, and what is less formally specified than the effective limits of a human being? If the human(s) are fully in the loop, then the buck stops with them and we know how to socially characterize and educate about failure modes. A pure-computer portion of the loop is characterizable by straight engineering.

But rather, we've got human pilots physically fighting against electrical motors. Either the pilot is right and the computer is wrong (in which case any physical effort expended is a further detriment), or the computer is right and the pilot is wrong (in which case letting the pilot effectively sabotage the computer is wrong). There is no middle ground between these two - weighing the input of a human versus the input of a computer based on them physically fighting is not a sound methodology.

Either powered drive of the trim is necessary or it is not. As it seems to be necessary due to the forces involved, the sensible design is to declare the actuator a critical part of the plane, and make the human input go through the actuator rather than around it.

Avionics software that has direct impact on the handling of a passenger flight should be globally mandated to be guaranteed faultless. There should be direct liability to the software engineering team. I know Boeing's software wasn't written in the USA, and was outsourced to the lowest bidder that met some on-paper requirements.

If you think this is outlandish, read "They Write The Right Stuff" https://www.fastcompany.com/28121/they-write-right-stuff

It can be done, but it is simply expensive. Boeing should hang for this.