41 comments

[ 5.2 ms ] story [ 120 ms ] thread
The thinking of "we don't want the system to not work if it's needed" thinking demonstrates a failure to understand the prior probabilities involved.

Most of the time the flight conditions will not warrant activation of the MCAS, and apparently the system is sufficiently unreliable/easy to damage that the chances of the system inadvertently triggering is high. But Boeing wanted to be sure that MCAS would always activate if needed, even if that resulted in unintended activations normally.

Boeing essentially optimized around a low probability of a false negative, while ignoring the high probability of a false positive, because they underestimated the consequences and confusion that would result from a false positive - in many cases, multiple alarms/system failures occurring simultaneously as other systems that rely on that sensor begin alarming as well. In that environment, designing around a 3 second response time (on a system that pilots weren't trained to know exists) is probably unreasonable.

Well, Boeing is multiple people. Business culture celebrates a manager who can push their project ahead of others. Maybe MCAS had a project manager who was a bit of an overachiever in that regard.
Surely I'm not the only one reading "overachiever" here as a wry euphemism.
Not really: person A is tasked with remeditating risk X (dangerous pitch-up at high AoA), persons B, C are tasked with remediating some other risks Y, Z, or just generally keeping an eye on the big picture. Now if person A is better than the others at pushing the needs of their particular target metric (or just more stubborn, a particularly easy form of "strength"), that plane will surely never crash due pitch up at high AoA, but maybe more likely due to other dangers that became more likely for the sake of never crashing from that pitch up problem that A was sent out to solve. This cost will be seen as a consequence of the weakness of B, C, not of the "strength" of A.

The high-resolve powerplayer who pushes their assigned task ahead of big-picture concerns is the hero of ambitious middle management culture. We see that happen many times, just rarely in domains as deadly as aviation.

If it were designed correctly, it shouldn't matter if it were activated or not - it would always be safe. If it were designed correctly there would be at least 3 AoA sensors and the loss or disagreement of one would not affect the functioning of the system other than displaying a warning message to the pilots. If the two remaining disagreed then the system would be disabled.

This is assuming that the system is necessary for certification or safety, if it is not necessary for certification or safety then it shouldn't be there at all.

The fact that Boeing put a secondary autopilot in there with a single point of failure, can only be explained by organisational failures reminiscent of the Challenger disaster.

The airframe cannot be certified airworthy without MCAS according to existing regulations. It is quite literally the only thing enabling the airframe to pass basic design requirements that have been mandated for years.
Citation? Nothing in FAR 25 permits a computer system to make airworthy an unairworthy airplane design.
Equivalent safety exception.

It's the blanket exception that makes most fly-by-wire controlled aircraft airworthy. The problem here of course, is that you never had enough scrutiny paid to the fly-by-wire implementation to make sure that under no circumstances can the system fail non-operational.

This was the subject of much controversy back in the 60's when th 727 was being certified. It required additional workarounds (implementation of a stick pusher) to be certified airworthy in U.K. airspace.

That certification went through despite the protest of Test pilots at the time that certifying an airframe with a widget to "plaster over" non-compliant behavior would simply start a slippery slope of normalization of deviance whereby engineers would over time be able to certify less "passively" airworthy frames. Those pilots were absolutely correct; as by definition that is exactly what most fly-by-wire aircraft are. Airframes that are only rendered airworthy with the addition of computer logic.

See the Royal Aeronautics Society D.P. Davies Interview.

Should be in this one. The 727 section starts at 51:00

Equivalent level of Safety discussion starts at 1:01:00

[1] https://www.aerosociety.com/news/audio-the-d-p-davies-interv...

Highly recommend to listening to all four episodes just because. It's informative, and for me was rather illuminative in terms of the constant conflict between aerodynamicists to leverage every trick possible to deliver more efficient designs, and pilots to ensure that planes don't become so complex to fly that they can't be reasonably saved even when things go wrong.

While history forces us to accept the validity of the approach (no one reasonably wants to kill Airbus); it is obvious there are ways that poorly implemented/communicated fly-by-wire piloting aids render an aircraft less safe. The MAX has illustrated one of these corner cases.

Furthermore, just read the article. They lay it out plain as day. The test pilots knew that the MAX's compliance and certifiability couldn't go forward in it's current configuration without being brought back into compliance with MCAS.

Despite that, testing to ensure this lynchpin system would not hit any edge case behavior with catastrophic results wasn't done; despite the fact that any reasonable engineer should have realized it was necessary. Give me a bit to drill through and I'll try to find the exact line this "equivalent safety exception" maps to. I'm not sure I'll find it because from the way Davies talks about it it strikes me as one of those possibly "unwritten" corollaries that tends to live beside anything codified. My main point though, is that this cavalier behavior with regard to airworthyness isn't unprecedented, and has been called out before. I fully accept it was different people working there then, but time and again I find echoes of an organization's risk management culture tends to survive the changing of the guard without active attempts to remedy it.

I don't accept that fly-by-wire airplanes are not FAR 25 airworthy in direct law where zero safe guards are in place. Those planes still have all the aerodynamic stability requirements in all three axis. Control surfaces move consistently with pilot stick input. You can't say the airplane is only airworthy when these safe guards are in place, and as soon as you're in some alternate law that the airplane is flying illegally (unairworthy).

Otherwise, every time such an airplane departs normal law, it should be reported (by plane's computer and pilots) that the flight became unairworthy in-flight. It became an illegal and unsafe flight. How does the mandatory incident reporting requirements get ignored in this case? Or maybe they aren't ignored by the aviation community, and merely go unreported to the public just how often airplanes depart normal law and are illegal flights? That'd be incredible, if it were true.

However, I think what we have in the case of the 737 MAX is altogether different from fly-by-wire. It's not clear to me from available reporting if the airplane, minus MCAS, did conform to FAR 25.173, all of it. If MCAS is required for the plane to conform to this section, and it either fails or is disabled, the pilots need to know the natural behavior of the airplane. They need to know they're in some equivalent of "direct law" (term used in fly-by-wire which the 737 is not) and the different flight behavior in place. And yet they didn't know, we now know they were intentionally not informed or trained.

So we have a case where something, MCAS, is so important that it's required for airworthiness, but has no redundancy, and totally insufficient self testing to know if it's spewing bogus data; and yet failure is not likely enough and/or not likely a big enough of a deal that pilots don't need any difference training for? Yeah I don't buy that shit and some heads absolutely need to roll for that, but at this point I have zero confidence they will. At least, not the correct heads will roll.

>I don't accept that fly-by-wire airplanes are not FAR 25 airworthy in direct law where zero safe guards are in place. Those planes still have all the aerodynamic stability requirements in all three axis. Control surfaces move consistently with pilot stick input. You can't say the airplane is only airworthy when these safe guards are in place, and as soon as you're in some alternate law that the airplane is flying illegally (unairworthy).

I absolutely hope that all fly-by wire aircraft meet minimum airworthyness standards in their minimum automation state. And I will admit my use of fly-by-wire is specifically constrained to the more general concept of a computer stabilized airframe rather than a more common assumption of it being associated with Airbus' particular implementation of fly-by-wire (I.e. Airbus Control Laws). The point I hope we can both agree on is that if you have an uncertifiable behavior corrected by an automation routine in a flight computer; the criterion of certifiability to be met is that the airworthy aircraft is airframe plus automation; not airframe without automation.

In that sense, there absolutely must be either sufficient redundancy to ensure that a pilot can land before the automation airworthyness is dependent on fails, or there must be enough training that should the automation fail, pilots are prepared to safely recover and land the plane. Neither of those is the case with MAX.

>Otherwise, every time such an airplane departs normal law, it should be reported (by plane's computer and pilots) that the flight became unairworthy in-flight. It became an illegal and unsafe flight. How does the mandatory incident reporting requirements get ignored in this case? Or maybe they aren't ignored by the aviation community, and merely go unreported to the public just how often airplanes depart normal law and are illegal flights? That'd be incredible, if it were true.

Dealing with the issue you describe here is/was a big part of the difference in philosophy between Airbus/Boeing.

With Airbus, they build the plane from the ground up with some degree of minimal automation in mind. These systems are designed to fail-operational, and with gradual decay of automated functionality in mind. I.e; the automation does as much as it can with the information it has available. Once inputs start disappearing, increased reliance on a pilot to replace the envelope protection software starts. An Airbus, to my current level of awareness, will never suffer an automation casualty that renders the frame unairworthy; and any such casualty that happens in flight that turns out to be recoverable in the sense of not resulting in an impact crater should be handled as an in-flight emergency.

With the MAX though, MCAS being disabled means that the aircraft no longer complies in terms of consistent stick force response curves in all requisite flight regimes. No matter how loudly Boeing protests it should never happen in normal flight, the rules are the rules. If it can't wind up turn, and maintain consistent stick force response curves, it shouldn't be flying as a civil transport aircraft. I.e. an MCAS casualty should be considered an in flight emergency.

This means, if you propagate through the fault tree, that AoA sensor malfunction/disagreement means a MAX should not be in the air as a civil transport. This is especially problematic, because unless you were Airbus, AoA sensors have hitherto rarely if ever been considered safety critical devices. So it would not be immediately intuitive to a pilot that a malfunction of an AoA sensor would potentially lead to a reasonable risk of catastrophic loss of the aircraft; at the hands of an undocumented and hidden automation system doubly so.

>It's not clear to me from available reporting if the airplane, minus MCAS, did conform to FAR 25.173

It did not. It failed to comply in terms of stick force response curves. That was why MCAS was created in the first place. To patch that non-compliance.

&g...

>If the two remaining disagreed then the system would be disabled.

On Lufthansa 1829, two of the three AOA sensors on an A321 erroneously agreed, causing a system similar to MCAS to kick in[1]

The sensors can also be damaged by birds

It seems like two sensors is not enough; nor are three. I'm curious whether it would be reasonable and feasible to add five to an aircraft

[1] Third bullet point under the A321 heading: https://en.wikipedia.org/wiki/List_of_accidents_and_incident...

If the MAX was designed correctly then MCAS would not be needed in the first place.

Fixing MCAS is fixing the wrong problem.

I agree ideally it would not be needed at all, however I believe that if Boeing decided that a new airframe would be too costly then they could have designed MCAS to be safe, by including a greater degree of hardware redundancy and an appropriately thorough development and testing process.

It seems that they did neither of these things, and still haven't learned their lesson because they are trying to rush to patch the lack of necessary hardware using software.

The real fix is to never fly on a 737 MAX
The plane needs to be refused certification, forcing Boeing to take them all back and scrap them. They need to go back to the drawing board and design a new jet to take its place. If that forces them into bankruptcy, then this should serve as a good lesson about what happens when your company that makes safety-critical products cuts corners and kills people.
If there is any decent probability it would fail danger (trigger a nose down as a result of bogus high angle of attack information from a single sensor), that means there is a need for pilots to be aware of both the true unaugmented aerodynamic behavior this (MCAS) system is meant to moderate as well as the function of that augmentation system.

Put another way: Just put another (virtual) button in the airplane and don't tell the pilots about it or what it's masking. It's important we must have this feature to certify, but it's also really important no one knows about it!

If what you're doing is so right, why lie about it? This is a lie of omission, at the very least.

So my understanding following this:

* they designed mcas with a g sensor and aoa sensor and a max pitch, to fix a problem in a high speed test. They provide safety assessments based on having two sensors, limited pitch changes, and it only triggering in specific rare situations.

* they provided that to the FAA as justification no training was needed, and that there was no need to acknowledge the existence of the system. FAA agreed.

* After FAA approval they discovered that the aircraft didn’t work correctly at low speed either, so they made it activate at low speed as well. Because it activated at low speed a g sensor wasn’t possible (?) so it got removed. Now you only have a single (notoriously unreliable) AoA sensor controlling it. To compensate for poor aero they also needed to increase the amount of pitch needed. So the FAA approved MCAS with a 0.6 degree pitch limit, occurring at high speed, triggered by two independent sensors. Boeing then made the system trigger at low speed, off a single sensor, and increase pitch adjustment to 2.6 degrees.

* Boeing chose not to test what happened if the AoA sensor failed.

* Boeing continued to insist that pilots not be made to retrain, because of contract penalties with southwest. That meant they did not want to mention the existence of the system.

* there safety margins also included an assumption that pilots could respond to a problem in 3s. But when mcas triggered incorrectly they were given speed, stall, and altitude warnings. All of which distracted them from the non-functioning led that indicate activation of a system that Boeing did not tell the pilots existed.

* the standard way to recover from the flight control computer triggering to much pitch was to flip a switch on the stick. Boeing made MCAS override that command, so the standard recovery could not work.

* Boeing continued to claim the plane was airworthy and provided a work around for its hellbent desired to crash, which was to use the manual pitch wheel. Except MCAS was designed in such a way that as part of its crash into the ground logic it put so much stress on the system that it was not physically possible for a pilot to perform the workaround.

In all: more than 300 people are dead because Boeing rushed to make a plane to compete with another company, and then hid information about an overriding control system so that they wouldn’t have to retrain pilots, and that completely changed the parameters of that system after FAA approval such that none of their approved safety claims were remotely true.

All to make some money so executives could get their bonuses.

Every executive who pushed for those changes should be put in jail.

And, Boeing's justification for increasing MCAS capabilities while reducing sensor redundancy needed was that these were changes in degree, not in kind. Nothing about the FAA certification actually depended on the number of sensors required, or the amount of command the system had, the FAA allowed pretty much anything to fly as long as it was only designed to operate when outside of the standard flight envelope.

This is the gotcha in the whole liability for executives thing. The FAA is extremely deferential to what manufacturers think is the best practice. Understandably it would be difficult to get every single little change to the flight systems approved - in broad strokes this kind of regulatory inertia is what makes physical changes so difficult in the first place, which indirectly caused this whole problem in the first place.

This isn't just a Boeing problem - Boeing was callous and indifferent to public safety, but the FAA let them get away with it. The takeaway from this crisis is that the FAA regulatory system needs significant reform, because it's obvious that it's got some unintended consequences. The correct answer here is that it should have been viable to design the airplane in an inherently aerodynamically safe fashion while the band-aid patch should have received a huge degree of scrutiny, but that's essentially the opposite of the system that the FAA has created.

The FAA situation is a difficult one to resolve - having full understanding of these systems requires a great deal of expertise, and that kind of expertise is found most easily at, obviously, the companies under regulation. More funding / regulators would certainly help, but I'm not sure if that would be sufficient.

> but the FAA let them get away with it

Well, the FAA thought the MCAS had a command limit of 0.6 degrees when it actually became 2.5 degrees after the changes mentioned in this article. From the other Seattle Times article about this:

> If the final safety analysis document was updated in parts, it certainly still contained the 0.6 limit in some places and the update was not widely communicated within the FAA technical evaluation team.

> “None of the engineers were aware of a higher limit,” said a second current FAA engineer.

To me it's less of a they "let them get away with it" and more of a "they had limited resources and things slipped through". Unfortunately this difference doesn't really matter when 300+ people are dead

I feel that's also by design: Boeing went out of their way to minimize the opportunity that the FAA had to catch what they were doing. Providing the experts no doubt helped, but just omitting changes by lying and saying all the changes were minimal to a system that had already passed safety review was their main tools.

Does the FAA have issues? yes. Will this event reduce the influence of the FAA in the future? Probably - we're already seeing the European equivalent adopting a stance of "the FAA is working for the corporations they're meant to regulate". Honestly the various politicians who gutted the FAA are fairly directly responsible for causing significant long term harm to American influence (for good or bad, depending of who/what/where you are).

Is Boeing entirely responsible? Yes. The FAA is meant to be adversarial to prevent companies from choosing money over safety, but fundamentally it was Boeing executives that made the choice to favour hiding design flaws to preserve profit (and hence bonuses). The fact is that any repercussions are almost certainly going to be entirely financial, and only harm shareholders (dividend and stock price) and lower level employees (e.g non-managers: managers, and especially executives, don't get canned when cash is low).

Wouldn't it be interesting if FFA signoff included approval by a regulatory team from the competing manufacturer. I wonder what abuses would crop up there.
Almost as if Google were to spent time and effort looking for security bugs in Microsoft products..
More like if Google had to approve every new release of MS products and shoulder some blame if things go wrong. Of course, if Google blocked some release of MS product, MS could retaliate by blocking a Google product. Or either could just stall until their own product could match features.

So obviously, any block would have to clearly state the rationale. Then the regulator could choose to overrule the block, but it would be more clear then where the gamesmanship was.

> the standard way to recover from the flight control computer triggering to much pitch was to flip a switch on the stick. Boeing made MCAS override that command, so the standard recovery could not work.

My understanding of this part was different. The standard way to recover is a pair of switches in the central part of the panel (not in the stick), which cuts off all power to the trim motors; these switches still work (though they work in slightly a different way than previous models). What MCAS overrides is a pair of sensors in the stick, which detect when the stick is being moved in the direction opposite to the electric trim, and in that case temporarily disable the trim motors; since MCAS by design moves the electric trim in the opposite direction that the pilot is moving the stick, it has to override these sensors, otherwise it couldn't do its job.

Ah, that seems entirely plausible (my misunderstanding I mean)
>Because it activated at low speed a g sensor wasn’t possible (?) so it got removed. Now you only have a single (notoriously unreliable) AoA sensor controlling it. To compensate for poor aero

To clarify: The G load gate had to be removed to enable the low speed activation.

The increased authority wasn't due to poor aero per se; but the fact that aerodynamic forces are dependent upon velocity. The slower you go, the more control surface deflection you need to get the same deflection that same control surface can generate at a higher speed.

>So the FAA approved MCAS with a 0.6 degree pitch limit, occurring at high speed, triggered by two independent sensors

Incorrect. The FAA approved input from only a single sensor, and the G load gate. Double sensor would have required simulator time; so was a non-starter. Both AoA sensors could talk to the Flight Control Computer, and MCAS as a result; but only one sensor was feeding data to the controlling FCC at a time.

The pitch adjustment wasn't actually up to 2.5 degrees. It was 2.5 degrees each time MCAS was activated. After two cycles it can reach maximum nose down trim of 4.7 degrees. (Or after 8 cycles under the original 0.6 degrees pitch limit).

The Boeing safety analysis did not seem to have realized this was a possibility.

> there safety margins also included an assumption that pilots could respond to a problem in 3s. But when mcas triggered incorrectly they were given speed, stall, and altitude warnings. All of which distracted them from the non-functioning led that indicate activation of a system that Boeing did not tell the pilots existed.

There was never an indicator that MCAS activated, what Boeing has said was missing was an indicator (text on screen) for Angle of Attack disagree -- this would have been in addition to the airspeed disagree warning that was present, since airspeed for the 737 MAX is calculated based on pitot tube and angle of attack measurements.

This indicator was missing unless the aircraft was purchased with the option to show angle of attack. It's not clear if knowing that the sensors differ without being able to see the readings is very useful. Even being able to see the readings and determine one sensor is broken doesn't help a whole lot without training on MCAS to know what a broken sensor could trigger, and how to handle that.

All of this to help avoid stalls. Such a pity because avoiding a stall seems to be one of the first things a new pilot learns. Every pilot should know how to detect and correct a stall.

Can an actual pilot conform or deny this?

It was less about avoiding stalls and more about making sure the performance characteristics of the MAX were similar enough to the previous generations of 737s so that a new type rating wasn't required for the pilots which would have required expensive training.
It’s not about avoiding stalls per se, it’s about the uncorrected aerodynamics of the 737 MAX being such that at a certain point it becomes easier to pull into a stall than push away from it.

While pilots learn to avoid stalls, commercial planes are designed deliberately such that at any point short of a stall it always takes less force to avoid one than create one.

Uncorrected, the 737 MAX’s behaviour is a dangerous and surprising deviation from norms. It runs counter to pilot training, and it makes the stall recovery they’re trained to perform significantly more challenging than it would otherwise be. That’s why the FAA rule about column forces exists.

The uncorrected behavior also violates a design directive encoded in the CFR's. The plane could not be certified as airworthy without MCAS active due to the non-compliant stick force response curves.
Retrospective ass covering, Boeing installed MCAS to save money, and in the process took control of the airplane away from the pilots. They should be looking at jailtime for this.
Boeing still gives the pilots a lot of control over the airplane, compared to Airbus. Airbus has defaulted to fly-by-wire (i.e: the computer has control and the pilot makes suggestions) for planes made since the 1980's.
Airbus doesn't try to hide that fact from its pilots.
Oh, for sure. The person I replied to made it sound like it was always a bad thing (to have a computer in charge), though, and that's certainly not the case.
Having a computer in charge works fine when the computer is designed correctly, with lots of redundancy and the pilots are trained to understand how it works.

This isn't what Boeing did.

Props to the author or editor that came up with the title of this. To make it more embarrassing:

> MCAS: How 737 MAX system gained power to compete with Airbus and lost safeguards

It's really tragic. If only they'd seen how much of a blunder this would be before lives were lost. But it's hard to get a big picture view until after the fact, when it involves so many people. I think perhaps Southwest was the closest to having a big picture view, as they ordered additional equipment [1]. I wish they somehow blew the whistle on it.

1: https://theaircurrent.com/aviation-safety/southwest-airlines...

It's really tragic. If only they'd seen how much of a blunder this would be before lives were lost.

Or, at the very minimum grounded that damn death trap volountarily after the Lion Air crash.

The first crash can be attributed to greed, short cuts, engineering blunders and immense pressure to rush the plane into production.

The second crash is corporate mass murder.