* There were two unusual events in very short order
* Someone quickly noticed and gave the order to go to ground stop
* The problem was figured out quickly and a work-around was developed
* Flights Resumed after successfully deploying the work around to the 'production process'
* A patch was quickly developed and deployed once the underlying bug was uncovered.
I think everyone (but perhaps the developers) comes out looking like a champ.
Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable - particularly if I was held to account for undetected failures.
The other part that I'd note here, is process is just as important as software could process alone have caught this failure without the tail strikes? possibly, and its something worth looking into further - but only if it doesn't add an unacceptable workload burden to pilots workload that could otherwise compromise safety further.
The purpose of telling pilots thrust was to save money and fuel. It’s a “least viable” optimization.
>“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance.
>Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy.
> That morning, a software bug in an update to the DynamicSource tool caused it to provide seriously undervalued weights for the airplanes.
>Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system.
> Both planes headed down the runway with less power and at lower speed than they should have. And with the jets judged lighter than they actually were, the pilots rotated too early.
>The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins. Both aircraft got airborne well within safety limits despite the lower thrust.
It’s also probably lucky that two Hawai’i bound flights both didn’t notice the discrepancy, or two bumps wouldn’t have bee as suspicious. Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human.
Like all complex failures at least 5 things had to happen. Software update that morning, some kind of hard to simulate resource issue in processing, overweight flights, pilot not pricing plane data appeared light, pilot turning early despite low speed, two in a row.
>Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human.
Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.
It does, and one class of mitigation is to improve automation, which airlines (and their regulators) do all the time.
But another class of mitigations is to make the human less error-prone in the first place. This is especially important in preventing failures that were caused by improper management of the automation. Hence the emphasis on rest periods, training, and other mitigations on the human side.
In the "swiss cheese" model, a disaster needs to slip through both the human's defenses and the automation's defenses to actually happen. It makes sense to improve both of those layers of defense against disaster, while acknowledging that humans will never be perfect. But at least they can be awake and knowledgable.
Right. Literally every pilot in the US has been taught the IMSAFE mnemonic to assess fitness to fly, even before their first solo in a single-engine piston: Illness, Medication, Stress, Alcohol, Fatigue and Emotion.
I just don’t see how it fit the article. Unless they are surmising the pilots poorly sanityized the data going into the computer because they were having an off day.
I mean, yes. That is literally exactly what they’re saying.
The full quote:
> …if there’s a glitch, naturally some pilot somewhere is going to miss it.
“Not everyone gets eight hours sleep the night before. Someone is going through a divorce. Someone is not so sharp that morning,” he said. “The sanity check isn’t perfect every day of the week.”
> The purpose of telling pilots thrust was to save money and fuel.
No, it's more that using lower thrust during take off saves on engine wear and noise levels around the airport. It doesn't really have that big an impact on fuel use, at least that's not the primary purpose.
The article concentrates far too much on the thrust setting. The important bit is the speed where the plane should be rotated to take off, which is known as V_r. That depends on the weight of the plane. On the takeoff roll, the pilots watch the airspeed indicators and pull back to lift off when that speed has been reached. If they have been told a V_r that is lower than it should be, then pulling back will rotate the aircraft without it taking off, and that's when you have this danger of the tail hitting the ground. When that happens, the plane must be inspected, because a tail strike has the potential to weaken the pressurised container that is the fuselage, and this could lead to an explosive burst and depressurisation when flying at high altitude. See https://en.wikipedia.org/wiki/China_Airlines_Flight_611 for why this is a Bad Thing.
It seems like a slower rotation might be useful, to wait until you feel it leave the ground before aiming higher. I'm sure there are reasons for the procedure as is.
Trying to "force the airplane off the ground" by rotating prior to Vr has a few safety downsides. One of them is that it makes tailstrikes more likely, as you'll have to carefully moderate the angle before lift occurs. Another is that it removes loading from the wheels more quickly, which extends the "awkward zone" in which the airplane is still rolling on the ground but has so little weight on the wheels that it will skid very easily. This can lead to aircraft being blown sideways off the runway since you can't yet roll into the wind.
A somewhat less direct concern but a related one is ground effect lift. the aircraft can remain a short distance off the ground (roughly a wingspan of altitude) at lower speeds than it can actually "fly," due to ground effect. Smaller aircraft might routinely spend some time in ground effect gaining additional speed before they begin climbing, but airliners have so much thrust they usually rotate pretty directly to their climb speed. This makes it more of an issue though that if rotation occurs too far before Vy climb speed the ground effect period will be prolonged and increase the amount of time the aircraft spends at risk, flying but slow with poor control authority and too close to the ground to have much of a recovery opportunity. The ground is a pretty safe place to be, the sky is a pretty safe place to be, but that first couple thousand feet between the ground and the sky is rather hazardous and jetliners get out of it as quickly as possible.
>> The ground is a pretty safe place to be, the sky is a pretty safe place to be, but that first couple thousand feet between the ground and the sky is rather hazardous and jetliners get out of it as quickly as possible.
Then the problem here is trading safety for something - cost apparently.
BTW, in a small plane I found soft field takeoffs quite fun. Nose up early, wheels up early, fly in ground effect to gain speed. I suppose that is a safety tradeoff and might not want to do it in some conditions.
Rotating at a slower speed is a bad idea. Indicating a lower weight, resulting in a lower calculated rotating speed, is the root cause of the problem. There is no "slower rotation", one does not just pull the yoke slowly, raise the nose wheel and wait for the plane to take off by itself when it will be fast enough, that you can do on a light prop plane where rotate speed is so low some planes will take off by themselves, but not on an airliner where you have a high chance that raising the nose too early results in speed not increasing or even decreasing.
That means rotating at a higher speed than calculated Vr, this is also bad as you put more wear in the tires and use more fuel (rolling on the tires vs flying in the air) to take off. The Vr is an optimal speed, below is dangerous, above is wasteful.
I think the more important factor is that it eats runway and takes longer to reach safe speeds. If there is a problem at rotation you want as much runway available as possible. If you delay rotation not only are you further down the runway before "really rotating", but you're traveling faster. A 10-knot increase in speed means 20% more energy to disipate. And you're even further above the brake's certified performace. And you've dawdled in getting to safe climb speeds; you accelerate more slowly rolling than flying.
Edit: Actually, that probably doesn't even come into it. Not a pilot, but don't imagine "barely rotate so the nosewheel is in the air but still have negative angle of attack so no lift" is very easy, or very stable if possible. Even if you could do it, you'd risk slamming the nosewheel back into the runway.
V_r is almost always above V_1, which is the decision speed. If you're going to abort a takeoff, that decision needs to be made before V_1, otherwise you're likely to overrun the end of the runway.
So, once you get to the stage where you're ready to rotate, or even late in rotating, you're committed to getting the plane off the ground.
On a big airliner yes. V1 depends on the runway length so a smaller aircraft will easily reach Vr before V1. Because most runways are made for the biggest planes.
This is a failure in the development of the software that good process in the purchaser’s handbook saved them from.
That there were no tests for this system under high load indicates a poor engineering practice at DynamicSource. They don’t deserve any kudos for a post-hoc patch that a test in production discovered the bug for when the system is a safety critical system like this.
I don't think OP was making excuses. It is true that the software vendor's missing test case indicates a poor quality development environment and that humans are the last line of defense. No development process is perfectly resilient against errors, even good quality processes.
My completely uninformed opinion is the design is flawed if the calculations across planes can interact. Is this some single tenant cloud based system with poor multithreaded safety?
Like why would the increased system load change the results. The calculations should be independent and designed in a way where they can't interact.
The bug is in the weight of the payload of the plane. I don't believe pilots have any way to measure the actual weight of the payload of the plane, they rely on that sort of estimates to determine how much fuel they need, take-off speed, etc. It's not really a loss of authority.
Bad weight estimates have resulted in crashes in the past.
I wonder if it would be possible to put a weighing pad at airports for a double-check. Like on the access ramp to the runway, meaning you'd only need a handful per airport and planes pass them naturally.
> I wonder if it would be possible to put a weighing pad at airports for a double-check.
A few modern pickup trucks have a system for measuring how much weight is in the bed or being transferred through the tongue of a trailer because of how common it is for people to overload their trucks.
AFAIK these systems just use the same sorts of suspension height sensors that have been used for auto-levelling for years and are factory calibrated to know that if the rear end squats by X distance then there must be Y load in it.
Seems like something along those lines would be technically easy to retrofit to large aircraft, possibly even as a software update if they already have some relevant sensors.
> Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable
Pilot here :) the journalist got it a bit wrong, performance data isn't primarily about how much thrust to use. It is part of the calculations, but crews are free to deviate from reduced thrust calculations and will do so.
This incident is much more about the calculated speeds to use and aircraft trim settings. That's what caused the tail strikes, trying to lift off at a speed that is too low for the weight.
Those numbers are required by every airliner taking off. And getting them from software is standard everywhere. The alternative is doing the calculations manually using paper charts, every pilot is required to know how to do that, but nobody does it that way for commercial operations.
In aviation incidents there isn't really a "blame", it's about finding all contributing factors and working together to eliminate those. The intention of investigations is never to assign blame, because that stands in the way of finding all factors and improving them together. So the conclusion "pilots should check the automated data" is a correct one. It's not the only cause, but it is one of the factors and thus it needs to be discussed.
In commercial aviation there can never be a single point of failure that could lead to catastrophic outcome. The performance calculations are such a thing, so there must be validation steps.
In many airlines for example if the performance calculations are done manually (or by an app based on manual input) they need to be done by both pilots independently and the results cross-checked. Otherwise a single typo could have a big impact.
If it makes you feel any better, the people providing these calculations to the pilots are FAA-certified airline dispatchers. That means they've passed the same exam that airline pilots have to take, minus the flying. They may not have personal experience flying the airplane but they do have to know it inside and out. And if the dispatchers weren't running the numbers, the pilots would be running them using the same software and/or tables, so there really isn't much difference.
Both aircraft seat about 180 pax. To be 20,000 lbs off in your hypothesis, each passenger would have to be 110 lbs heavier than modeled, on average. For 30,000 lbs, 165 lbs.
> The new FAA standards will increase an average adult passenger and carry-on bag weight to 190 pounds in the summer and 195 pounds in the winter. Up 12% from 170 pounds and 175 pounds, respectively.
This is a lot smaller in magnitude than +110-165 lb.
That was an impressively quick call based on ability to reason about unexpected behavior of the system. It should give pause to anyone who wants to take the human out of the loop in safety critical software.
Agreed, although it takes an individual with a huge amount of integrity or a culture that values and promotes integrity to make a call like this:
> “At that point, two in a row like that, that’s when I said, ‘No, we’re done,'” said Peyton. “That’s when I stopped things.”
Hats off to Peyton, and possibly Alaska. I would also love to hear about times when something similar happened, and it was discovered that nothing was wrong.
Amazingly, this really is the approach that has characterized all aspects of aviation for decades. It’s incredibly admirable and worth emulating in many other situations.
It’s also fragile, and the fact that it strikes many as an uncommon level of integrity for a business decision is precisely why aviation needs to be shielded from having the prevailing business culture sneak in, like we have seen happen at Boeing in a few highly notable instances.
> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system.
That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.
There is certainly not enough information in that quote to say one way or another what the bug was. I've definitely seen concurrency bugs under load because data that wasn't supposed to be shared actually was, e.g. I posted this serious GitHub bug in a comment above, https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare....
Obviously sessions should be independent and not sharing data, but that's why it was a bug.
I believe any bug that only happens under load is a concurrency bug by definition. The shared resource is the thing under load. If it weren't shared, then the load from one computation would have no effect on another.
A proper transactional database would suffice. You could do this safely and performantly on '00s hardware in PHP3 for crying out loud, serving thousands of planes per second.
I refuse to believe that summing up weights for each of 300 planes is so hard that it can't be made correct by a team of moderately competent developers operating under sane management. There might be some complexity, but it's 2023 and we have enough knowledge and tools to solve that kind of stuff routinely and reliably. Somebody fucked up big time and should be fired never to work in software again.
I agree that it's hard to see from the outside how calculating weight figures for an airplane could be software load-dependent. But this all smacks of the point that Brooks makes in 'The Mythical Man Month' all those years ago. Two guys in a garage can make a program that does 'x', but to make a programming system product that does 'x' is more than an order of magnitude more work. The complexities introduced don't relate to the 'x', but the intricacies of large systems. Deciding that 'someone has fucked up and should never work again'? Sorry, competent and conscientious people regularly make such mistakes. Which is why we need tests, software review, engineering processes et al to catch those mistakes.
I can give you a simple example on how this can happen, from a real production system in a different regulated environment: all the SQL SELECT statements are WITH (NOLOCK) on MS SQL server, resulting in non-committed (dirty) reads. Under very light load it works fine, on higher loads it reads some dirty data. If you calculate the weight of the plane by adding the weight of the checked in luggage plus passengers read from a shared database, some differences will appear and they will increase with the load as more transactions will be in flight and possibly read or ignored.
Not according to the article, the values returned were within the realm of possibility, but wrong. Giving out zeroes might have actually been fine, it’s easy enough to realise there’s a problem if you see your plane weighs 0 pounds.
I mean it might be just one part. E.g. luggage gets counted as zero but passenger weight, fuel weight and aircraft weight correct - that might be pluausible but low.
> but being wrong under load sounds like a horrible internal architecture problem.
Tons of software can be "wrong under load" - things like race conditions and memory leaks are common problems, and they don't necessarily point to a huge architectural defect. E.g. Cloudflare had a bug years ago that caused highly sensitive data to leak, but only for a teeny (relatively) portion of their site. Similar issue happened to GitHub: https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare...
But calling something an "internal architectural problem" implies things are fundamentally designed wrong, and the code needs a rewrite. My point was the architecture can definitely be fine, but you can still have bugs (even catastrophic ones) because someone left out a `synchronized` keyword somewhere.
For example, one way to use SQL is to escape your parameters and then concat them straight into your SQL string. However,
if someone leaves out an ‘escape’ somewhere, you have a major security incident
The other way is to pass parameters separately as data to your SQL driver. It completely negates the problem
If you’ve chosen the first way in your project already, you’ve committed to a major internal architectural problem. In the same vein, maybe if your code requires sprinkling ‘synchronized’ everywhere, you did it wrong
But that said, we don’t know what actually is the problem since it’s just a marketing statement saying that high load was the cause and that doesn’t tell us anything
True, but software for domains where people can die is very different than Cloudflare. The scale is much higher in the latter, and it needs to be less "bulletproof".
If you're writing software for giant metal flying cylinder carrying hundreds of people, we can't just brush mistakes off the same way we might with web-based software (and yes, privacy is important, but it's not life or death).
I'm not "brushing mistakes off", at all. The process that led to the bugs needs a full review and root-cause corrective action, especially enhanced testing scenarios. I'm just pointing out that software "being wrong under load" is not some sort of crazy, unusual problem that points to poor software architecture. These bugs can easily be caused by a single line of code that wasn't synchronized correctly.
In most industries that are not IT, testing the IT systems very well is needed, but rarely achieved. In my area (FDA regulated) performance testing is almost never done correctly and (for sure) never completely, but as long as it works and the auditors never look at these details nobody from management is willing to know about it, because fixing it costs money. For many years the people signing off for all the software were intentionally not business people with zero IT knowledge, so they could not be accused of knowingly signing off low quality code. That saved tens or hundreds of millions of dollars in the past 20 years alone.
For what it's worth, I had the same reaction upon reading this paragraph. It seems there are HN commenters who understand software design and those who do not.
To be fair to the other commenters, it's conceivable that the practical difference between the current unsound architecture and a sound replacement might come down to to some small defect. But an attempt to fix the problem by fixing just the defect, even some kind of RCA process on it, will fail if the architecture itself is not fixed, and that requires a change in attitudes and understanding by the devs responsible for the system, not just a software change. That is where the real flaw lies. But some people don't believe that it's possible for the flaw to be in these places, outside the code. All the RCAs in the world can't help them.
I was just about to post the _Exact_ same thing. Each execution of the program should be completely independent from other calculations. Something is _Horribly_ wrong with their architecture.
I wonder how it works. Obviously you are correct, but the number of moving parts must be huge. The number of passengers, amount of cargo and the fuel load must be changing the whole time.
... Call to the baggage-tracking service returned 429 and an empty body, and that was treated as "there are no bags" because some dweeb [dev, but I'm leaving in what autocorrect said] had been reading to much design guidance from Netflix/Facebook/etc that assumes missing features on a fraction of page loads are NBD?
If you view software as a bunch of contracts between different services/libraries/etc. it is hard to ensure that each piece upholds its contract, so we trust them to a degree. I can see how a failure in one place due to load would be hard to catch or gracefully fail from.
That said they have a pretty explicit cap on maximum activity (number of planes) so it is weird they didn't test around this. It isn't like 400,000 aircraft suddenly DDOS their system.
The key here seems to be the word "many."
Alaska Airlines has 289 airplanes (per wikipedia).
All the other arguments seem to assume a large consumer type of load - tens of thousands of users, etc...
I just can't see an undue strain being placed on a well designed system from < 300 data points. And I haven't even accounted for the distribution of needing to compute takeoff data over the course of a day nor how many planes are NOT taking off at the same time, etc...
Also, to somewhat change the topic, didn't Alaska Airlines disband their QA org a few years ago as part of cost cutting? IIRC, they did this to model the software company models (that ship bugs regularly to consumers) and seem to be getting some data that they need to bring back that org...
So much I can't understand here. Why would the load matter- is this a web app? Why would that possibly be a good idea? This seems like software that should run locally, for security, assurance, and auditability reasons.
Follow up thought: When software results are this critical, I wonder if a totally separate program should be used as well and result compared. An independent implementation from another vendor.
> the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system
> the data was on the order of 20,000 to 30,000 pounds light. With the total weight of those jets at 150,000 to 170,000 pounds, the error was enough to skew the engine thrust and speed settings.
Multithreading/contention issue? But how would that alter the weights?
> The software code was permanently repaired about five hours later
Multithreading/contention issue? But how would that alter the weights?
Off the top of my head: failure to sum values correctly, misreporting the weight to the target plane (reading values from another plane’s weight data).
We had that error in an application at work. Essentially data was coming in on multiple threads, but there was a shared value between them that was being incremented and decremented. Without a lock and without using atomics, the tally could be incorrect. But under low load this didn't appear because the odds of interrupting +=/-=/++/-- was low. Under high load, however, the odds of a thread being interrupted mid-operation increased and so it appeared in production. At least in our case it was just an annoyance (no critical decisions were based on the value, it was merely informative).
If there are multiple data streams feeding in weight data per aircraft, under a low aircraft load (the scenarios they apparently tested) the odds of interruption during an operation like += can be low enough to not see the issue. Under high load, though, the odds of interruption and incorrect tallying increases substantially.
In what scenario would you anticipate data for the calculation of rotation speed and engine thrust for one particular plane to be based on a value that is not only shared and writable, but changing each time a plane's performance data is calculated?
"multiple data streams feeding in weight data per aircraft"? What does that word salad even mean?
Even if what any of what you said was true: these developers are writing critical safety software. If they can't manage to write code without generating race conditions and testing their system for such conditions, they are grossly unqualified to be writing this sort of software.
And that's not word salad, though maybe not as clear as I could have made it. But to clarify for this hypothetical (if this is the issue): Each luggage weigh station submits data to the application or database which then has to tally the weight. Those are multiple data streams, not a hard concept. An airport has many luggage weigh stations all potentially submitting data at the same time.
If those are being processed concurrently and the database or however the data is stored is not properly locked then the tally can be incorrect for the same reason non-atomic increments/decrements can become incorrect in a multithreaded application. But just like the problem with non-atomic (unlocked) increments/decrements the problem may not manifest without high enough load on the system. Low-load testing (what the article seems to describe) means that updates can happen fast enough (server or application is under loaded, less likely to interrupt any update) or submitted slow enough that the issue never appears. So you need to test with a higher load to detect the problem (if you're going to be able detect it in testing at all).
Concurrency is one source, yes, but integration is another possible source. If the vendor software interfaced with Alaska's software then the interfacing layer could imperfectly handle pagination, assume ordering where there is none, and other such data source problems.
A trivial error could be "list all active planes", "list current luggage weight for all these plane IDs", and have the latter give responses in a different order than the input list. Or perhaps "fuel load" returns in a different order. Or something.
Concurrency seems like the obvious one, but it's not the only source I can imagine here. And ultimately, the max is under 300 planes. You can do that serially unless the integration layer is slow.
I'm surprised there wasn't any human or general guardrail around the calculation that would have flagged such a significant weight difference. No one looked at the passenger count and the weight calculation and thought "whoa, that doesn't look right?"
> The bug was identified quickly in part because some flight crews noticed the weights didn’t seem right and asked for manual validation of the figures.
Actually there were such guardrails, so the system had multiple manual checks built in.
How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable?
This story is about the system working well. Detected immediately via well-trained and alert humans, someone had and used their individual authority to ground the entire airline for safety reasons, mitigated with a workaround 20 minutes later, permanently fixed within five hours, and new tests implemented to account for the root cause.
I'm with you that a couple layers of the "safety in depth" worked impressively well here. But a bug of this types points to egregiously bad process and standards with the software vendor. And the airlines or FAA should have audits to catch that well upstream of this incident.
The SpaceX rockets launch routinely with automation at takeoff and landing. There is no need for human heroic drama in the loop when the software is AAA grade.
"SpaceX acknowledges that it failed to communicate due to the bug and missed the emails about a higher probability of collision. Finally, on Monday morning in Europe, ESA made the call and used the thrusters on Aeolus to raise the satellite’s orbit by about 984 feet (300 meters) without waiting for SpaceX to take corrective action."
I see this as a near-miss. At my work, there'd be a post-mortem written explaining what went well and what went poorly. Just because not every line of defense was breached doesn't mean everything's peachy.
The first failure was the bug itself - without knowing more about the details it's hard to say more. But one should ask what kind of practices exist at DynamicSource - e.g., compile time thread safety checks, code review practices, testing requirements, rollout procedures - that would prevent bugs like this from getting out into production.
The second failure was that the software failed to validate its own output and fail noisily or generate a warning of some kind. It's safety-critical software - bugs may be excusable, but not giving at least a warning for an unusual output is poor engineering.
The third failure was that the flight staff failed to notice the error until tail strikes happened. Human errors do occur, but we need better systems to assist human operators. E.g., perhaps the flight software itself could act as another line of defense to prompt the pilot when values appear out of normal range.
Yes, in this case the planes weren't anywhere close to crashing. But if the assumption is that bugs are unavoidable, then there needs to be better systems in place to catch those bugs before they cause a major accident. Because the next time we might not get so lucky with detecting the issue and having someone with integrity calling the shots.
Also, no need to start off your comment with an ad hominem - a simple "software errors are unavoidable" would've said just as much.
> How does a "SaaS Founder, Senior DevOps Engineer" get the idea that software errors are entirely avoidable?
It's literally a safety critical system, not move-fast-and-break-things front end web dev code.
Remember the extraordinarily in depth review of their software while their entire fleet was grounded, where each and every individual bug found was reported in mainstream media?
Where they weren't allowed back in the sky until we were all assured everything was fixed?
The bug they're reporting here sounds like they didn't do reasonable testing. And that's the kind of thing that shouldn't be happening with Boeing 737* aircraft especially, after their recent problems. :/
"This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats : the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors."
> Where they weren't allowed back in the sky until we were all assured everything was fixed?
Because no aircraft would fly again with that standard.
> And that's the kind of thing that shouldn't be happening with Boeing 737* aircraft especially, after their recent problems.
The software in question isn't made by Boeing, and it's not just for Boeing aircraft. It's a third-party thing, picked by individual airlines. https://www.dynamicsource.se/ lists support for 13 aircraft across 7 manufacturers.
“the bug was missed because it only presented when many aircraft at the same time were using the system.”
The system reports data on number of passengers, weight of cargo, plane balance, etc., to the pilots. The calculation is done by the plane’s flight computer. How can it be off by 20,000 pounds, but only under heavy server load?
The explanation that comes to mind is that DynamicSource has a subservice for each source of weight and one of those subservices crashed under heavy usage. So the top-level aggregate-and-report service got an error from one subservice and said “well, guess it’s zero, lol”?
Makes me think this could have been a race condition where these aircraft somehow received the load calculations from other aircraft being calculated at the same time window. Some kind of poorly managed event stream / consumer that assumed no parallel jobs would be run? Or maybe the jobs are deduped by departure / arrival instead of flight number? Seems possible
>The calculation is done by the plane’s flight computer.
Not exactly true.
The dispatch release has a flight & performance plan based on the load manifest. It's calculated out of band (here, by the software vendor) and punched into the airplane configuration for departure.
If the plan is wrong, the plane flies wrong. GIGO at its finest.
Lots of blame to pass on the software, but the real culprit is spelled out clearly in the middle of the article:
> the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance.
This is not an accident, but rather a feature of capitalism, this time it is human lives that are commodified and have costs externalized unto.
In a competitive industry saving money can mean lower fares for customers. It can also mean fewer total planes or engines produced reducing environmental
side effects of that activity.
It can mean those things, yes, but only as side-effects. The primary effect being to increase shareholder value.
There’s no need to have market based competition to keep costs down. Look at USPS for example. That operation is run without a profit motive, serves literally every zip code in the US (by definition), and runs healthily year after year (except for the part where Congress imposed arbitrary pension funding requirements[1] and private capital interests have been chomping at the bit to buy up all the real-estate holdings).
They also don’t have UPS’s problem of not putting AC in their delivery vehicles.
Calculating the appropriate amount of liftoff thrust based on the weight of the plane is universal for flight ops.
This feature of capitalism caused the money to be invested to achieve this efficiency, and also the systems and processes in place to ensure it was implemented properly; with sanity checks, authoritative oversight to halt ops when something was determined to be off, manual backup systems in place and well trained on, a fix rapidly deployed, and new testing put in place to detect any similar bugs in the future.
Yes indeed the stench of capitalism is strong here! </s>
I’m not sure that thrust calculation is actually incredibly common. I’d be curious what the handbook on these aircraft say, but common practice for many aircraft is full thrust on takeoff - rarely is there a negative safety penalty for too much thrust. Just an earlier rotate.
Comment OP isn’t wrong entirely - this is the result of an economic decision. Not without it’s good effects - both can be true, but this is not a safety optimization by any means.
Many aviation accidents can be traced back to capitalistic pressure - cost saving on inspections, time saving for flow, etc. It’s appropriate to ask whether this is true here as well.
“Common practice for many aircraft is full thrust on takeoff” - not for the class of aircraft we are discussing. What hobbyists do on their Turbo Props is irrelevant to the operation of a 737.
There are many considerations to setting takeoff thrust like climb speed limits, tire speed limits, elevation, temperature, engine-out safety margins, and many more I’m sure I don’t know about with my limited flight knowledge.
This is before we get into discussing weight load, which is primarily a factor of fuel load, which is also not just filled to 100% for every flight, but carefully calculated based on distance, weight, wind speed, etc. to not carry unnecessary fuel, which is also a huge efficiency boost.
In short there are safety reasons to use the appropriate (non-maximum) thrust on these types of planes just as much as there are efficiency reasons.
Capitalistic market forces are a big reason we have such an amazing airline industry with planes that are technological marvels of comfort, speed, and efficiency. Capitalism drove 120 years of relentless R&D into commercial flight systems, from the very first flight in 1903.
Not paying for necessary safety inspections is not “capitalistic” in any way. It seems like an ideological battle to use the term that way.
I mean yes, would you rather flights are $10,000 or $100,000 each so the engines are taken apart and rebuilt each time after a flight? Reducing the maintenance interval requirements 'safely' is good for everyone.
Less fuel is better for the environment. That should be counterbalanced with safety, which they did - even with a bug and a heavy plane the plane still took off safely.
This means they are also leaving a safety margin of fuel, even though that costs extra money.
I don't see how this is an example of that? Saving fuel and reducing wear on the parts is a good thing. In this case, nobody's lives were in danger either.
That’s a feature of engineering. Figuring out what performance and safety requirements are and then optimizing the resources required to meet those requirements is what engineering is for.
The influence of capitalism would be on the requirements. But assuming the requirements are well-considered trying to minimize fuel and engine usage to meet those requirements is a good thing.
I think this is pretty subjective. You could also say that engineering should be used to increase the margin of safety (what is often labeled over-engineering).
If the engineering requirement is “do something that brings you close to the margin of safety in order to save resources”, is that not an influence of capitalism?
No, again the decision on where to draw the line is a political decision. It’s produced by humans making values decisions. There’s no absence of risk situations so there are always trade offs.
How to hit that line precisely and with confidence is where engineering takes over.
If efficiency is the goal, packing people into a standing-only flight would also save fuel and reduce wear, I suppose you don't see any problem with that either?
Seems like something as critical as thrust in this phase of flight should have some kind of cross-check validation. Sounds like a bad single point of failure to me, but maybe there is a cross check I don’t know about that isn’t covered in the story.
What I do know is that with something like this, a little could go a long way. I wonder what the inspection and repair for a tail strike is, and whether that cancels out the money saved by minimum viable thrust across the fleet. I’m sure someone is punching the calculator on this to determine that.
I was under the impression that SOP for most aircraft was to take off at full power, and if you happen to be lightly loaded you just get into the air faster. Was I wrong, or is this a recent change? If so, it seems like a concerning way for airlines to cut costs. Having excess power available at take off means there's a greater margin for error to handle emergencies (engine failure, bird strike, etc.).
Yes. Chelsea Sullenberger ("Sully") made the same point in an interview. He said in the early days of the airlines, the CEO was always a pilot, and error margins were set much higher than legally required. Now that CEOs tend NOT to be retired pilots, new regulations are needed to raise the safety margins beyond current regulatory points to what a safety-conscious pilot would choose.
Note that these are manufacturer-sanctioned procedures, not applicable to small piston aircraft, which should be using full power/RPM if the AFM/POH calls for that.
Note also that the full rated power (and sometimes more) is available in the case of an abnormal or emergency.
I'm surprised too. I would have thought the take-off system would incorporate some sort of feedback about how close the tail was to touching the ground and either tell the pilot or correct for it and tell the pilot.
Seems like any modern jet would have a load sensing system built into the landing gear. Something so critical such as takeoff weight would be both estimated by a software system and compared to an onboard system for redundancy. Any discrepancy beyond a set limit would turn on a warning light.
> The Alaska captain said that, as for many things in aviation, pilots routinely use an acronym when they do the pre-takeoff “sanity check”: TLAR, which means “That Looks About Right.”
> Bret Peyton, Alaska’s on-duty director of operations, immediately ordered no more planes were to take off across the airline’s network. All Alaska flights not already airborne were stopped nationwide.
An absolute pro. There's a hundred variations of this story, to varying degrees of criticality and impact; seeing a pattern out of two data points, connecting the dots, making the tremendous call to immediately pull the plug, to stop the world and give engineering time, then diagnosing and triaging the problem in less than a half hour; that's world class reliability engineering.
I think you misunderstand my comment. I'm saying, for instance, that during a calm period when a group of people can take their time and review each other's reasoning, a rule might be put in effect like, "Two tail strikes within X hours of each other means an automatic, immediate halt." Humans execute that procedure (or override it at the risk of their career) and then decide what to do next, having been in the loop the entire time.
The natural tendency for military air crews is to complete the mission if humanly possible. To counter this inclination, the Wing Commander had designated certain emergencies sufficiently critical to require immediate landing. This was one of those emergencies.
737-900s are notoriously tail-heavy. Some airlines require that they use tail stands when parked, and some airlines (like United) have taken measures to redistribute weight forward of the rear landing gear (by moving one of the economy lavatories so that it sits directly behind first class/in front of most economy seating).
Do 737s in the US still even have first class?. I'm surprised. I've never seen any in Europe in the last 15 years on short haul which is of course what a 737 is used for.
Yes, they do, but the domestic US version of first class. Also, the 737s in the article were departing on a 6+ hour flight to Hawaii, so they're not only used for short haul flights. Lots of coast to coast flights of 5ish hours are also on single aisle planes.
My fucking washing machine can sense how big a load is in it. Surely the plane could measure how fast it is actually accelerating given its thrust and thus derive its actual mass. If it doesn’t match what it’s been told it should alarm.
Also shocking that the system gives an incorrect answer when it is under load. Lack of load testing is a problem, sure, but an architecture that allows the answer to just be wrong is fundamentally flawed.
If weight calculation is so crucial for take off, it seems like redundancy of the weight calculation is needed - a different weight calculation from a different company using a different method to verify against the main one.
Would it be possible to actually weigh the loaded plane to check the calculations, say with something like a bigger version of truck scales that the plane could taxi over?
I don't understand why an airplane being heavier than what was estimated by the software would cause the tail to drag. I would assume it would cause the plane to not have sufficient thrust to take off, or to use more runway. But having the tail drag makes it seem like it had too much thrust, which translated to a steeper take off than anticipated... Can somebody explain?
In order to achieve the required lift force for takeoff, a higher angle of attack was needed to compensate for the incorrect low rotation airspeed. In effect, pilots raised to a higher nose-high attitude to achieve lift off by the expected point, and this resulted in a tail strike.
Not an expert, but here's my best guess. When I learned to fly a small plane, you pulled back the yoke at a specific speed. This is called 'rotation' and lifts the nose of the plane off the ground, (wheels are still on the ground) and changes the attitude of the wings, creating substantially more lift for the speed. The plane then lifts quickly off the runway. Presumably different possible weights in a larger plane would cause you to choose different speeds at which to rotate. I also presume rotation is also a more gradual process in a larger plane (a few seconds long???), so the 'early' part of the rotation occurs, gets the plane off the ground, and allows the space for the angle to become steeper in the later part of the rotation. Would be happy for an expert to review and comment on this.
Let's say your target rotation angle is 10 degrees, so at the specified speed (called VR) the pilots 'rotate' the aircraft nose up by 10 degrees.
When they start the 'rotate' maneuver, the nose of the aircraft doesn't immediately go to (for instance) 10 degrees. It's a progression.
As soon as the nose starts pitching up (by 1 degree for instance), the aircraft is already generating lift to pull the aircraft up.
If the speed is set right (so you're fast), when you get to that 10 degrees angle of attack, your tail has already cleared the runway by a very safe margin, because you've been producing X lift from 0 to 10 degrees of rotation.
However, if your speed is set too slow, you're not generating as much lift, so the aircraft will be pulled up slower (you have been producing Y% less lift from 0 to 10 degrees of rotation). This means that by the time you reach your designated 10 degrees of angle of attack, you haven't cleared the tail yet, and it will hit the runway.
I believe it's a question of angle of attack (orientation of the wings vs the direction of airflow). The more you angle the wings, the more upward lift they generate. So when the pilots commanded the plane to "rotate" (take off), they had to pull it back farther than expected to generate sufficient lift for the heavier-than-calcuated and slower-than-calculated plane to leave the runway. That means the nose of the plane was pointing higher than normal, which means the tail was lower than normal.
(Note that you can't keep increasing AoA indefinitely if you need more lift; at a critical speed, the airflow will separate from the wing and stop generating any lift. That's an aerodynamic stall.)
The article states the following about the software:
>"It delivers a message to the cockpit with crucial weight and balance data, including how many people are on board, the jet’s empty and gross weight and the position of its center of gravity.
>In a cockpit check before takeoff, this data is entered into the flight computer to determine how much thrust the engines will provide and at what speed the jet will be ready to lift off."
Given that overhead bins are regularly maxed out with carry-on luggage now since airlines began charging for checked bags, how are they able to accurately account for the weight and balance? Airlines seem to almost never weigh customers carry-on at check in.
"Regularly maxed-out" translates to "fairly predictable". If the 20 people in first class all have 500 pounds of uranium each they've snuck onboard, you have a problem, but otherwise, there's plenty of wiggle room to account for minor variability of this kind.
They'll have a decent idea of the average weight of the average passenger (https://airinsight.com/the-pending-new-faa-weight-balance-ru...). If you're chartered by an anorexia treatment facility to move 300 patients, you may put a little extra attention to the exact values.
> The new FAA standards will increase an average adult passenger and carry-on bag weight to 190 pounds in the summer and 195 pounds in the winter. Up 12% from 170 pounds and 175 pounds, respectively. This includes an extra ten pounds for winter and five pounds for summer. This also includes 16 pounds for personal items, up from ten. Airlines must increase the average weight of female passengers and their carry-ons from 145 pounds to 179 pounds in summer, and from 150 pounds to 184 pounds in winter. The average weight for males with carry-ons is increased from 185 pounds in summer to 200 pounds, and from 190 pounds to 205 pounds in winter.
This would be an effective side channel attack to use against one's enemies. Even if the true source was never known, the chaos and distrust in infrastructure after the crash would be nice second-order effects.
"Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system."
Now that's strange. Anyone have more details? Why should there be any connection between the calculations for different aircraft?
188 comments
[ 3.0 ms ] story [ 248 ms ] thread* There were two unusual events in very short order
* Someone quickly noticed and gave the order to go to ground stop
* The problem was figured out quickly and a work-around was developed
* Flights Resumed after successfully deploying the work around to the 'production process'
* A patch was quickly developed and deployed once the underlying bug was uncovered.
I think everyone (but perhaps the developers) comes out looking like a champ.
Now, do I think telling pilots what thrust they should use to take off is wise? I dont know - I'm not a pilot, and I'd want pilots feelings on the matter before commenting further, but it feels to me like a loss of authority that would make me uncomfortable - particularly if I was held to account for undetected failures.
The other part that I'd note here, is process is just as important as software could process alone have caught this failure without the tail strikes? possibly, and its something worth looking into further - but only if it doesn't add an unacceptable workload burden to pilots workload that could otherwise compromise safety further.
>“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance.
>Flights to Hawaii are typically full, with lots of baggage and a full load of fuel for the trip across the ocean. The planes are heavy.
> That morning, a software bug in an update to the DynamicSource tool caused it to provide seriously undervalued weights for the airplanes.
>Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system.
> Both planes headed down the runway with less power and at lower speed than they should have. And with the jets judged lighter than they actually were, the pilots rotated too early.
>The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins. Both aircraft got airborne well within safety limits despite the lower thrust.
It’s also probably lucky that two Hawai’i bound flights both didn’t notice the discrepancy, or two bumps wouldn’t have bee as suspicious. Interesting that someone cited divorce and sleep examples as emotional pleas, reassuring us they are only human.
Like all complex failures at least 5 things had to happen. Software update that morning, some kind of hard to simulate resource issue in processing, overweight flights, pilot not pricing plane data appeared light, pilot turning early despite low speed, two in a row.
Huh? FAA spends a huge amount of time and energy focusing on human factors, tasks saturation rates and crew resource management. Nothing to do with with argumentum ad passiones, just prudent risk mitigation.
But another class of mitigations is to make the human less error-prone in the first place. This is especially important in preventing failures that were caused by improper management of the automation. Hence the emphasis on rest periods, training, and other mitigations on the human side.
In the "swiss cheese" model, a disaster needs to slip through both the human's defenses and the automation's defenses to actually happen. It makes sense to improve both of those layers of defense against disaster, while acknowledging that humans will never be perfect. But at least they can be awake and knowledgable.
The full quote:
> …if there’s a glitch, naturally some pilot somewhere is going to miss it.
“Not everyone gets eight hours sleep the night before. Someone is going through a divorce. Someone is not so sharp that morning,” he said. “The sanity check isn’t perfect every day of the week.”
No, it's more that using lower thrust during take off saves on engine wear and noise levels around the airport. It doesn't really have that big an impact on fuel use, at least that's not the primary purpose.
The article concentrates far too much on the thrust setting. The important bit is the speed where the plane should be rotated to take off, which is known as V_r. That depends on the weight of the plane. On the takeoff roll, the pilots watch the airspeed indicators and pull back to lift off when that speed has been reached. If they have been told a V_r that is lower than it should be, then pulling back will rotate the aircraft without it taking off, and that's when you have this danger of the tail hitting the ground. When that happens, the plane must be inspected, because a tail strike has the potential to weaken the pressurised container that is the fuselage, and this could lead to an explosive burst and depressurisation when flying at high altitude. See https://en.wikipedia.org/wiki/China_Airlines_Flight_611 for why this is a Bad Thing.
Which is saving on money
There's a difference between dumping toxic waste into a river to save money and keeping a plane engine in safe operating conditions to save money.
A somewhat less direct concern but a related one is ground effect lift. the aircraft can remain a short distance off the ground (roughly a wingspan of altitude) at lower speeds than it can actually "fly," due to ground effect. Smaller aircraft might routinely spend some time in ground effect gaining additional speed before they begin climbing, but airliners have so much thrust they usually rotate pretty directly to their climb speed. This makes it more of an issue though that if rotation occurs too far before Vy climb speed the ground effect period will be prolonged and increase the amount of time the aircraft spends at risk, flying but slow with poor control authority and too close to the ground to have much of a recovery opportunity. The ground is a pretty safe place to be, the sky is a pretty safe place to be, but that first couple thousand feet between the ground and the sky is rather hazardous and jetliners get out of it as quickly as possible.
Then the problem here is trading safety for something - cost apparently.
BTW, in a small plane I found soft field takeoffs quite fun. Nose up early, wheels up early, fly in ground effect to gain speed. I suppose that is a safety tradeoff and might not want to do it in some conditions.
Edit: Actually, that probably doesn't even come into it. Not a pilot, but don't imagine "barely rotate so the nosewheel is in the air but still have negative angle of attack so no lift" is very easy, or very stable if possible. Even if you could do it, you'd risk slamming the nosewheel back into the runway.
So, once you get to the stage where you're ready to rotate, or even late in rotating, you're committed to getting the plane off the ground.
But it could also just be statistically true. Being in a position of power, surrounded by beautiful coworkers, lots of overnight hotel stays..
Without further investigation a conclusion can't be drawn though.
That there were no tests for this system under high load indicates a poor engineering practice at DynamicSource. They don’t deserve any kudos for a post-hoc patch that a test in production discovered the bug for when the system is a safety critical system like this.
Also process, procedure and more importantly people are always the last line of defense against failure.
Like why would the increased system load change the results. The calculations should be independent and designed in a way where they can't interact.
Bad weight estimates have resulted in crashes in the past.
A few modern pickup trucks have a system for measuring how much weight is in the bed or being transferred through the tongue of a trailer because of how common it is for people to overload their trucks.
AFAIK these systems just use the same sorts of suspension height sensors that have been used for auto-levelling for years and are factory calibrated to know that if the rear end squats by X distance then there must be Y load in it.
Seems like something along those lines would be technically easy to retrofit to large aircraft, possibly even as a software update if they already have some relevant sensors.
Pilot here :) the journalist got it a bit wrong, performance data isn't primarily about how much thrust to use. It is part of the calculations, but crews are free to deviate from reduced thrust calculations and will do so.
This incident is much more about the calculated speeds to use and aircraft trim settings. That's what caused the tail strikes, trying to lift off at a speed that is too low for the weight.
Those numbers are required by every airliner taking off. And getting them from software is standard everywhere. The alternative is doing the calculations manually using paper charts, every pilot is required to know how to do that, but nobody does it that way for commercial operations.
I thought I was missing something in the way the story read, I'm glad to know I was.
> “Still, the mishaps point to the need for more vigilance by pilots in checking automated data.”
In commercial aviation there can never be a single point of failure that could lead to catastrophic outcome. The performance calculations are such a thing, so there must be validation steps.
In many airlines for example if the performance calculations are done manually (or by an app based on manual input) they need to be done by both pilots independently and the results cross-checked. Otherwise a single typo could have a big impact.
There were recent changes to the weight and balance rules because of increased weight. It's certainly possible this was a factor.
This is a lot smaller in magnitude than +110-165 lb.
> “At that point, two in a row like that, that’s when I said, ‘No, we’re done,'” said Peyton. “That’s when I stopped things.”
Hats off to Peyton, and possibly Alaska. I would also love to hear about times when something similar happened, and it was discovered that nothing was wrong.
It’s also fragile, and the fact that it strikes many as an uncommon level of integrity for a business decision is precisely why aviation needs to be shielded from having the prevailing business culture sneak in, like we have seen happen at Boeing in a few highly notable instances.
[0] https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...
[1] https://en.wikipedia.org/wiki/Able_Archer_83
That seems horribly wrong to me. I can understand software being slow under load, but being wrong under load sounds like a horrible internal architecture problem.
"the bug was missed because it only presented when many aircraft at the same time were using the system"
Obviously sessions should be independent and not sharing data, but that's why it was a bug.
https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-... (paywalled)
https://archive.is/vdg9S (paywall workaround)
Note that there were several articles about it around the time, the above one is just the first reasonable seeming one from a quick search.
Such as weight=plane+fuel+people+luggage. If one of the variables became 0 you’d still have the other 3.
It was a low enough value to have made multiple pilots question its value that day, like fuel or passenger load was missing completely.
Tons of software can be "wrong under load" - things like race conditions and memory leaks are common problems, and they don't necessarily point to a huge architectural defect. E.g. Cloudflare had a bug years ago that caused highly sensitive data to leak, but only for a teeny (relatively) portion of their site. Similar issue happened to GitHub: https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare...
For example, one way to use SQL is to escape your parameters and then concat them straight into your SQL string. However, if someone leaves out an ‘escape’ somewhere, you have a major security incident
The other way is to pass parameters separately as data to your SQL driver. It completely negates the problem
If you’ve chosen the first way in your project already, you’ve committed to a major internal architectural problem. In the same vein, maybe if your code requires sprinkling ‘synchronized’ everywhere, you did it wrong
But that said, we don’t know what actually is the problem since it’s just a marketing statement saying that high load was the cause and that doesn’t tell us anything
If you're writing software for giant metal flying cylinder carrying hundreds of people, we can't just brush mistakes off the same way we might with web-based software (and yes, privacy is important, but it's not life or death).
FTFY. IT shops on the whole aren’t any better here.
Software's accuracy being affected by load is completely unacceptable.
There's no such thing as perfect software, but there is definitely software that doesn't make up a false value if it can't work.
To be fair to the other commenters, it's conceivable that the practical difference between the current unsound architecture and a sound replacement might come down to to some small defect. But an attempt to fix the problem by fixing just the defect, even some kind of RCA process on it, will fail if the architecture itself is not fixed, and that requires a change in attitudes and understanding by the devs responsible for the system, not just a software change. That is where the real flaw lies. But some people don't believe that it's possible for the flaw to be in these places, outside the code. All the RCAs in the world can't help them.
That said they have a pretty explicit cap on maximum activity (number of planes) so it is weird they didn't test around this. It isn't like 400,000 aircraft suddenly DDOS their system.
All the other arguments seem to assume a large consumer type of load - tens of thousands of users, etc...
I just can't see an undue strain being placed on a well designed system from < 300 data points. And I haven't even accounted for the distribution of needing to compute takeoff data over the course of a day nor how many planes are NOT taking off at the same time, etc...
Also, to somewhat change the topic, didn't Alaska Airlines disband their QA org a few years ago as part of cost cutting? IIRC, they did this to model the software company models (that ship bugs regularly to consumers) and seem to be getting some data that they need to bring back that org...
Follow up thought: When software results are this critical, I wonder if a totally separate program should be used as well and result compared. An independent implementation from another vendor.
> the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system
> the data was on the order of 20,000 to 30,000 pounds light. With the total weight of those jets at 150,000 to 170,000 pounds, the error was enough to skew the engine thrust and speed settings.
Multithreading/contention issue? But how would that alter the weights?
> The software code was permanently repaired about five hours later
That's surprisingly fast, isn't it?
Off the top of my head: failure to sum values correctly, misreporting the weight to the target plane (reading values from another plane’s weight data).
If there are multiple data streams feeding in weight data per aircraft, under a low aircraft load (the scenarios they apparently tested) the odds of interruption during an operation like += can be low enough to not see the issue. Under high load, though, the odds of interruption and incorrect tallying increases substantially.
"multiple data streams feeding in weight data per aircraft"? What does that word salad even mean?
Even if what any of what you said was true: these developers are writing critical safety software. If they can't manage to write code without generating race conditions and testing their system for such conditions, they are grossly unqualified to be writing this sort of software.
As others have said: this is a huge fuck-up.
And that's not word salad, though maybe not as clear as I could have made it. But to clarify for this hypothetical (if this is the issue): Each luggage weigh station submits data to the application or database which then has to tally the weight. Those are multiple data streams, not a hard concept. An airport has many luggage weigh stations all potentially submitting data at the same time.
If those are being processed concurrently and the database or however the data is stored is not properly locked then the tally can be incorrect for the same reason non-atomic increments/decrements can become incorrect in a multithreaded application. But just like the problem with non-atomic (unlocked) increments/decrements the problem may not manifest without high enough load on the system. Low-load testing (what the article seems to describe) means that updates can happen fast enough (server or application is under loaded, less likely to interrupt any update) or submitted slow enough that the issue never appears. So you need to test with a higher load to detect the problem (if you're going to be able detect it in testing at all).
"multiple data streams feeding in weight data per aircraft"
it's literally some integers (floats at the worst?).
Maybe they stored something in a global variable, and with two planes querying at the same time values for the wrong plane could be retrieved.
A trivial error could be "list all active planes", "list current luggage weight for all these plane IDs", and have the latter give responses in a different order than the input list. Or perhaps "fuel load" returns in a different order. Or something.
Concurrency seems like the obvious one, but it's not the only source I can imagine here. And ultimately, the max is under 300 planes. You can do that serially unless the integration layer is slow.
Actually there were such guardrails, so the system had multiple manual checks built in.
This story is about the system working well. Detected immediately via well-trained and alert humans, someone had and used their individual authority to ground the entire airline for safety reasons, mitigated with a workaround 20 minutes later, permanently fixed within five hours, and new tests implemented to account for the root cause.
Sure. And there are bugs in their code, too.
https://www.theverge.com/2019/9/3/20847243/spacex-starlink-s...
"SpaceX acknowledges that it failed to communicate due to the bug and missed the emails about a higher probability of collision. Finally, on Monday morning in Europe, ESA made the call and used the thrusters on Aeolus to raise the satellite’s orbit by about 984 feet (300 meters) without waiting for SpaceX to take corrective action."
The first failure was the bug itself - without knowing more about the details it's hard to say more. But one should ask what kind of practices exist at DynamicSource - e.g., compile time thread safety checks, code review practices, testing requirements, rollout procedures - that would prevent bugs like this from getting out into production.
The second failure was that the software failed to validate its own output and fail noisily or generate a warning of some kind. It's safety-critical software - bugs may be excusable, but not giving at least a warning for an unusual output is poor engineering.
The third failure was that the flight staff failed to notice the error until tail strikes happened. Human errors do occur, but we need better systems to assist human operators. E.g., perhaps the flight software itself could act as another line of defense to prompt the pilot when values appear out of normal range.
Yes, in this case the planes weren't anywhere close to crashing. But if the assumption is that bugs are unavoidable, then there needs to be better systems in place to catch those bugs before they cause a major accident. Because the next time we might not get so lucky with detecting the issue and having someone with integrity calling the shots.
Also, no need to start off your comment with an ad hominem - a simple "software errors are unavoidable" would've said just as much.
It's literally a safety critical system, not move-fast-and-break-things front end web dev code.
Remember the extraordinarily in depth review of their software while their entire fleet was grounded, where each and every individual bug found was reported in mainstream media?
Where they weren't allowed back in the sky until we were all assured everything was fixed?
The bug they're reporting here sounds like they didn't do reasonable testing. And that's the kind of thing that shouldn't be happening with Boeing 737* aircraft especially, after their recent problems. :/
So was the Space Shuttle, whose software team is widely regarded as the absolute gold standard in this regard.
https://www.fastcompany.com/28121/they-write-right-stuff
Still, bugs:
"This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats : the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors."
> Where they weren't allowed back in the sky until we were all assured everything was fixed?
Because no aircraft would fly again with that standard.
> And that's the kind of thing that shouldn't be happening with Boeing 737* aircraft especially, after their recent problems.
The software in question isn't made by Boeing, and it's not just for Boeing aircraft. It's a third-party thing, picked by individual airlines. https://www.dynamicsource.se/ lists support for 13 aircraft across 7 manufacturers.
The system reports data on number of passengers, weight of cargo, plane balance, etc., to the pilots. The calculation is done by the plane’s flight computer. How can it be off by 20,000 pounds, but only under heavy server load?
The explanation that comes to mind is that DynamicSource has a subservice for each source of weight and one of those subservices crashed under heavy usage. So the top-level aggregate-and-report service got an error from one subservice and said “well, guess it’s zero, lol”?
Not exactly true.
The dispatch release has a flight & performance plan based on the load manifest. It's calculated out of band (here, by the software vendor) and punched into the airplane configuration for departure.
If the plan is wrong, the plane flies wrong. GIGO at its finest.
> the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary. “The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance.
This is not an accident, but rather a feature of capitalism, this time it is human lives that are commodified and have costs externalized unto.
There’s no need to have market based competition to keep costs down. Look at USPS for example. That operation is run without a profit motive, serves literally every zip code in the US (by definition), and runs healthily year after year (except for the part where Congress imposed arbitrary pension funding requirements[1] and private capital interests have been chomping at the bit to buy up all the real-estate holdings).
They also don’t have UPS’s problem of not putting AC in their delivery vehicles.
[1] https://ips-dc.org/how-congress-manufactured-a-postal-crisis...
This feature of capitalism caused the money to be invested to achieve this efficiency, and also the systems and processes in place to ensure it was implemented properly; with sanity checks, authoritative oversight to halt ops when something was determined to be off, manual backup systems in place and well trained on, a fix rapidly deployed, and new testing put in place to detect any similar bugs in the future.
Yes indeed the stench of capitalism is strong here! </s>
Comment OP isn’t wrong entirely - this is the result of an economic decision. Not without it’s good effects - both can be true, but this is not a safety optimization by any means.
Many aviation accidents can be traced back to capitalistic pressure - cost saving on inspections, time saving for flow, etc. It’s appropriate to ask whether this is true here as well.
There are many considerations to setting takeoff thrust like climb speed limits, tire speed limits, elevation, temperature, engine-out safety margins, and many more I’m sure I don’t know about with my limited flight knowledge.
This is before we get into discussing weight load, which is primarily a factor of fuel load, which is also not just filled to 100% for every flight, but carefully calculated based on distance, weight, wind speed, etc. to not carry unnecessary fuel, which is also a huge efficiency boost.
In short there are safety reasons to use the appropriate (non-maximum) thrust on these types of planes just as much as there are efficiency reasons.
Capitalistic market forces are a big reason we have such an amazing airline industry with planes that are technological marvels of comfort, speed, and efficiency. Capitalism drove 120 years of relentless R&D into commercial flight systems, from the very first flight in 1903.
Not paying for necessary safety inspections is not “capitalistic” in any way. It seems like an ideological battle to use the term that way.
This means they are also leaving a safety margin of fuel, even though that costs extra money.
The influence of capitalism would be on the requirements. But assuming the requirements are well-considered trying to minimize fuel and engine usage to meet those requirements is a good thing.
If the engineering requirement is “do something that brings you close to the margin of safety in order to save resources”, is that not an influence of capitalism?
How to hit that line precisely and with confidence is where engineering takes over.
If efficiency is the curse capitalism brings us, I’m feeling a little better about it.
https://www.theguardian.com/business/2012/feb/28/ryanair-sta...
You’re trolling, right? That seems like a really implausible world view.
Software to optimize resource usage while providing a service => blame capitalism is che-guevara-tshirt-edgelord level of inane.
It basically boils down to “this flight started nosediving, almost hit the ocean, and we don’t know why”.
https://www.nytimes.com/2023/02/13/us/united-maui-flight.htm...
What I do know is that with something like this, a little could go a long way. I wonder what the inspection and repair for a tail strike is, and whether that cancels out the money saved by minimum viable thrust across the fleet. I’m sure someone is punching the calculator on this to determine that.
Make no mistake. The Exec/finance class will cut into any margin that you aren't willing to make them directly legally culpable for.
Note that these are manufacturer-sanctioned procedures, not applicable to small piston aircraft, which should be using full power/RPM if the AFM/POH calls for that.
Note also that the full rated power (and sometimes more) is available in the case of an abnormal or emergency.
As long as you save more than $1.5 billion ($5 million x 300 passengers) before a plane crashes, no problem.
Until we completely bankrupt a company for killing people under its jurisdiction, this will all continue.
Aviation's LGTM
An absolute pro. There's a hundred variations of this story, to varying degrees of criticality and impact; seeing a pattern out of two data points, connecting the dots, making the tremendous call to immediately pull the plug, to stop the world and give engineering time, then diagnosing and triaging the problem in less than a half hour; that's world class reliability engineering.
Such a rule likely saved an SR-71 once: https://theaviationgeekclub.com/the-story-of-belmont-86-the-...
The natural tendency for military air crews is to complete the mission if humanly possible. To counter this inclination, the Wing Commander had designated certain emergencies sufficiently critical to require immediate landing. This was one of those emergencies.
Also shocking that the system gives an incorrect answer when it is under load. Lack of load testing is a problem, sure, but an architecture that allows the answer to just be wrong is fundamentally flawed.
When they start the 'rotate' maneuver, the nose of the aircraft doesn't immediately go to (for instance) 10 degrees. It's a progression.
As soon as the nose starts pitching up (by 1 degree for instance), the aircraft is already generating lift to pull the aircraft up.
If the speed is set right (so you're fast), when you get to that 10 degrees angle of attack, your tail has already cleared the runway by a very safe margin, because you've been producing X lift from 0 to 10 degrees of rotation.
However, if your speed is set too slow, you're not generating as much lift, so the aircraft will be pulled up slower (you have been producing Y% less lift from 0 to 10 degrees of rotation). This means that by the time you reach your designated 10 degrees of angle of attack, you haven't cleared the tail yet, and it will hit the runway.
(Note that you can't keep increasing AoA indefinitely if you need more lift; at a critical speed, the airflow will separate from the wing and stop generating any lift. That's an aerodynamic stall.)
>"It delivers a message to the cockpit with crucial weight and balance data, including how many people are on board, the jet’s empty and gross weight and the position of its center of gravity.
>In a cockpit check before takeoff, this data is entered into the flight computer to determine how much thrust the engines will provide and at what speed the jet will be ready to lift off."
Given that overhead bins are regularly maxed out with carry-on luggage now since airlines began charging for checked bags, how are they able to accurately account for the weight and balance? Airlines seem to almost never weigh customers carry-on at check in.
They'll have a decent idea of the average weight of the average passenger (https://airinsight.com/the-pending-new-faa-weight-balance-ru...). If you're chartered by an anorexia treatment facility to move 300 patients, you may put a little extra attention to the exact values.
> The new FAA standards will increase an average adult passenger and carry-on bag weight to 190 pounds in the summer and 195 pounds in the winter. Up 12% from 170 pounds and 175 pounds, respectively. This includes an extra ten pounds for winter and five pounds for summer. This also includes 16 pounds for personal items, up from ten. Airlines must increase the average weight of female passengers and their carry-ons from 145 pounds to 179 pounds in summer, and from 150 pounds to 184 pounds in winter. The average weight for males with carry-ons is increased from 185 pounds in summer to 200 pounds, and from 190 pounds to 205 pounds in winter.
Now that's strange. Anyone have more details? Why should there be any connection between the calculations for different aircraft?