That seems like a process that could have eaisly been reverted to a paper based worksheet of some type. from my understanding thats how they did it before the software.
The contingency in this case is just to shut down apparently. They’ve probably figured out that training on this issue is more expensive than shutting down for a few hours.
Or maybe their risk management teams have never thought of this happening, who knows.
On short notice and without pilots having any experience in doing it? That’s another hole in the Swiss cheese.
You either do something manually all the time, or you automate it and declare it a critical system. Half measures of automation kill people in Aviation.
You can't simply roll back because the old ways required the use of phone lines and fax systems and record types and workflows that literally no longer exist.
That said, the critical plane flying and ground handling stuff (like loading cargo for proper balance) is well practiced and understood enough to run fine with ad-hoc paper records albeit it would reduce operational tempo.
It's putting the right luggage in the right place, making sure the right plane gets its poo pumped, making sure the cleaning crew knows what to do and where to go, recording maintenance needs and activities on anything that doesn't fly, etc, etc that doesn't have redundancy or good fallback plans or people/managers who understand how their jobs fit into the context well enough to improvise effectively and even if they could the B rate people you had design all your corporate process didn't account for that sort of stuff so people would be risking their jobs if they couldn't do sign offs on the company ipads and whatnot.
Weight and balance isn't simply a matter of add up the weight of all cargo on paper. You need to load the plane in a reasonably laid out way and the whole result dictates how you take off. Being wrong or winging it could easily mean a catastrophic failure when trying to take off.
Weight and balance calculations are something that anyone in aviation will wind up memorizing early in their career. You don't "need" bespoke software for it. You don't need software at all. It just makes it way easier when you have hundreds of individual items.
What's most likely to happen is that they will punch it all into an excel sheet and get their primary objective of weight distribution correct. What will suffer is cargo organization so the crew at the layover is likely to find that they need to unload a bunch of stuff in order to get at some cargo that needs to come off at that stop.
A plane "could" fall out of the sky but that would require multiple people not doing their job but there's no point in pearl clutching over that because even if the software works the same people not doing their job can still cause that outcome.
This vastly understates the seriousness of problems you can run into. It was just last year that all of Alaska Airlines was grounded [0] because of issues with weight and balance. While those takeoffs resulted in tail scrapes, they could've just as easily lead to running out of runway if the bug had been slightly different [1].
Comments like yours really drive me up the wall. Nobody doubts that a plane needs to be at least somewhat balanced to fly. It also needs fuel. It needs some semblance of a maintenance schedule. It needs many things. Everyone with a brain knows all this. So then what means beyond in-group signaling is served by hand wringing over any one of the dozens of critical links in the chain?
The bug you cite, which you would not likely even know about if not for other commenters, was only caught because people asked for an old school manual calculation. Had the software process been more transparent it perhaps could have been caught earlier.
The fact that someone failed to properly implement the calculation in software does not change the efficacy of doing the calculation manually. Furthermore, the fact that there were people capable of eyeballing the accuracy of the software outputs seems to back up my point that there are tons of people capable of doing these calculations kicking around.
The bug was caught because airplanes had tail strikes. It might help to actually read the article you're commenting on, especially with that high horse.
Like I said, hand wringing about someone's botched software implementation does not change the fact that this is simple math that tons of people at an airline are qualified to perform and check.
This is not rocket surgery. This is something that literally every private pilot with a Cessna does when they encounter a novel loading. Stop making it out to be some super scary thing requiring serious expertise to not screw up.
Around 25 years ago I worked the PoS at a retail establishment. We ran all credit card payments using a machine that would dial-up our payment processor for each transaction. It used the same line as the fax, so if we were receiving or sending a many-page fax this operation would fail. Sometimes it failed for other reasons.
As a backup we had the old carbon slips that went into a plastic rig and made an impression of the customer's card. We also had a printed out sales tax table and some more carbon slips we could use to hand-write a receipt. I had to resort to the paper based system on many occasions.
A few years back I was in a store shopping when the power went out. The staff didn't know what to do and closed the store early. They had no way to sell me the items in my basket. I asked if they could take an impression of my card and got met with blank stares. They couldn't even take cash because they didn't know how to calculate the total amount, had nothing to write a receipt on, and their cash drawers wouldn't open anyway.
Nowadays credit cards don't even come with raised numbers...
> Nowadays credit cards don't even come with raised numbers...
So that's what the raised numbers were for. I just got my first card without raised numbers last January. It still feels fake and "insecure" to me, even though I know better. Funny how things work.
My latest card doesn't even have the card number, expiry or CVC printed on it at all. And besides that, the CVC code is dynamically generated every time I need it, it doesn't even have a static CVC.
Seems relatively new, as some people look at funnily when I hand it over when they need it for "deposits" or whatever.
I've always wondered how that works for businesses that still prefer to compare the last four digits of a physical card with those of a booking made online, like some hotels or airlines still do.
Nope, a physical card from BBVA. I tend to use banks for my financial needs ;)
So far, they've looked at me strangely, fetched a superior who either says "whatever" or they run through the card through their reader to get the last four. Seems to work out well enough at least.
Don't worry, the Apple card is issued by a bank too, not Apple.
Running it through a reader would also not work – the number on the chip and magnetic stripe is different from the one you see in the app and use for online payments. That's why I'd rather not use it for a hotel or international airline booking, for example (some still have "you must have the physical card with you to be allowed to board" in their terms and conditions, although nobody has ever requested to see mine in practice).
Do you know if your last four digits (that are printed on receipts too) match those you can see in the app?
> Don't worry, the Apple card is issued by a bank too, not Apple.
Ah, first time I heard about it, seems to not be available outside the US so that explains it :)
> Do you know if your last four digits (that are printed on receipts too) match those you can see in the app?
Yes, it does. It also matches with whatever they see when they run the card, however they do that. The card number is identical for online payments, "offline" payments (with terminals) and what I see in my banking application.
The only thing that changes is the CVC, as it rotates every ~6 minutes or so.
> The staff didn't know what to do and closed the store early.
This was literally a bit in the Y2K episode of King of the Hill. Hank goes to buy a new "Y2K compliant" computer. The cashier couldn't ring him up and freaks out.
Interesting, would you mind sharing what country that is?
I have only a single card with raised/embossed numbers, and that's somewhat ironically a pre-tax transit benefit card, i.e. the least likely of any I own to ever be used in a carbon imprinter.
In France, all my credit cards except the very last one have raised numbers.
The last one with raised numbers was delivered to me in 2024. The one without came a few months later, from the same bank. It also has the number on the back, right next to the static CVC. I guess security is indeed their top priority.
Anecdotally, all the credit cards I've had started to have the paint rub off the embossed number fairly quickly, rendering it very difficult to read. The other text on them didn't have this issue.
Arguably it's a wash for security: Yes, you can take a single photo of all the card details if the CVC is on the same side as the number and expiry, but on the other hand, there's nothing of value on the front now, which helps with people taking photos of their cards and posting them on social media.
And I agree, the paint rubbing off is extremely annoying, and even worse when it's metallic paint, flakes of which have a habit of deeply embedding themselves in all kinds of leather and fabric materials. That, and the fact that you can't fit as many of them in a "card-only wallet", makes me not want to carry any of them anymore.
Because credit card fraud has had a few decades to evolve too, and is now orders of magnitudes more easy to coordinate.
If word gets out that a given store has no online processing and is accepting carbon slips again, you can bet that quite a few people with stashes of stolen/closed credit cards will show up and start buying the most expensive items in the store.
The modern equivalent would be offline chip processing. That is actually capable of at least ensuring that a given card is not heavily overdrawn or outright forged. Still, even that is unfortunately being scaled back significantly, given that "everything is always online anyway, so why do we need it?"
There should be a business continuity plan in case of “glitches”. But these airlines are cheap and greedy to develop playbooks, train airline staff, and airport personnel on the ground.
Much of the reason that commercial aviation is as safe as it is is the prioritization of safety over mission completion in the face of uncertainty. If you have uncertainty in any of the dispatch software or other dispatch items, as a passenger, I'm perfectly happy to wait until they get things sorted out rather than having this hypothetically trained staff manually executing alternative procedures that are unusual to them in order to get me to my destination a couple hours earlier...
I guarantee you that they have numerous continuity plans that probably get activated often, and because they have them, you don’t see any issues. It’s only when something happens far outside of those plans that it becomes visible. It’s confirmation bias.
I don’t think this is confirmation bias. Confirmation bias is putting more weight on information that support preexisting beliefs. If a continuity plan is activated and the passengers are unaware then that is a totally different thing. If the passengers are made aware of a continuity plan being implemented then Negativity Bias or the Von Restorff Effect would be more accurate descriptions, in which negative stimuli or unusual stimuli are more frequently recalled.
You’re right. I meant survivorship bias, where the survivors are the events that are bad enough for them to be visible to the public. Other events that are addressed by a continuity plan are “killed” and therefore aren’t visible.
can you imagine bieng in the room when they saw a problem with the weight/balance software?, planes in the air!
realy realy hope this was a false positive or a reporting glitch, rather than ANY flaw in what was shown to pilots
i assume it would be within bounds during “normal” atmospheric conditions. In the air with the wrong data and extreme conditions would be where it gets dicey?
Incorrect w&b can affect a lot of things. From longer ground roll during landing, to wrong brake settings, to worse handling of the aircraft and increased stall speeds. Under normal circumstances it should not be a problem due to huge safety margins, but if other things go wrong it starts adding up. One more hole in the swiss cheese model.
While far from ideal, a 90-minute delay isn't really comparable to what happened to Southwest[0] or Delta[1], that resulted dozens of cancelled flights and almost a week of disruption.
I happened to be flying right in the midst and worst of Delta's incident, and honestly: Kudos to them for trying to make the best of a shitty situation. I still got to where I wanted to go, and the worst I experienced was my checked bag missed my flight and arrived a day late on the return leg.
Looking over the flights that were cancelled while I was at the airport (Hartsfield-Jackson Atlanta International, oddly enough), Delta was clearly prioritizing cancelling flights that had alternatives so as many travellers had the best shot at getting to their destination.
I've liked them ever since the Northwest days, and I don't expect that to change any time soon.
I also like Delta and have forgiven them for that episode, but many of us flying that day didn't make it to our destinations until several days later (in many cases too late for the purpose of the trip). I was flying home from a work conference and was lucky that I woke up early, saw my flight was cancelled within an hour of the incident starting, and immediately booked on another airline. I got home 6 hours later than intended, but at least I got home the same day.
The alternatives that Delta booked for me were already going to be the next morning (Saturday), and I followed them after getting home: while the first leg worked fine, the second leg was cancelled late Saturday morning. It's likely that I'd have received yet another rebooking, but if that one also involved two legs it could easily have pushed my return date to Sunday, and by that point in the weekend I doubt I could have found openings on other airlines if I'd changed my mind.
I had coworkers who didn't make it back home from that conference until Monday night.
So yes, I think they tried hard to avoid inconveniencing people, but if you made it the same day you were lucky. Many didn't.
As being part of a software development team working on RPS (runway performance) I can tell you, it is stories like this which keep us up at night.
The amount of business logic involved is staggering, but miraculously there are experts who truly seem to understand almost all of it.
Understanding “almost all of it” is not enough, so there are several different experts who in their entirety hopefully capture 99.99…% But you don’t know what you don’t know…
One of the “calming” facts is, that in the tens of thousands of test cases we compared our results to existing solutions, we did discover errors in their calculations. The “calming” aspect is, due to the many safe-guards these errors seemingly never surfaced.
Coming from a machine-learning background I quickly understood that there is little appetite for AI solutions, as any optimization which would reduce fuel consumption, is easily offset by all the safety margins.
In my mind the goal of the software is to make everything explainable and proof that you truly did think of every eventuality.
I am very curious to learn what major f*-up lead to planes being grounded.
Serious question: Dont software changes with companies dealing with human lives have to go through a rigorous lab testing environment? And then maybe a beta group?
I’ve been to a talk from the chief test engineer at Saab talking about all the Hardware in the Loop labs they use to test at unit, integration, and full system levels. Then there’s additional testing after that phase… It’s not a CD world!
55 comments
[ 4.6 ms ] story [ 127 ms ] threadPreviously: https://news.ycombinator.com/item?id=34895061
Or maybe their risk management teams have never thought of this happening, who knows.
On short notice and without pilots having any experience in doing it? That’s another hole in the Swiss cheese.
You either do something manually all the time, or you automate it and declare it a critical system. Half measures of automation kill people in Aviation.
That said, the critical plane flying and ground handling stuff (like loading cargo for proper balance) is well practiced and understood enough to run fine with ad-hoc paper records albeit it would reduce operational tempo.
It's putting the right luggage in the right place, making sure the right plane gets its poo pumped, making sure the cleaning crew knows what to do and where to go, recording maintenance needs and activities on anything that doesn't fly, etc, etc that doesn't have redundancy or good fallback plans or people/managers who understand how their jobs fit into the context well enough to improvise effectively and even if they could the B rate people you had design all your corporate process didn't account for that sort of stuff so people would be risking their jobs if they couldn't do sign offs on the company ipads and whatnot.
What's most likely to happen is that they will punch it all into an excel sheet and get their primary objective of weight distribution correct. What will suffer is cargo organization so the crew at the layover is likely to find that they need to unload a bunch of stuff in order to get at some cargo that needs to come off at that stop.
A plane "could" fall out of the sky but that would require multiple people not doing their job but there's no point in pearl clutching over that because even if the software works the same people not doing their job can still cause that outcome.
[0] https://www.adn.com/alaska-news/aviation/2023/02/20/after-al...
[1] https://news.ycombinator.com/item?id=34898361
The bug you cite, which you would not likely even know about if not for other commenters, was only caught because people asked for an old school manual calculation. Had the software process been more transparent it perhaps could have been caught earlier.
The fact that someone failed to properly implement the calculation in software does not change the efficacy of doing the calculation manually. Furthermore, the fact that there were people capable of eyeballing the accuracy of the software outputs seems to back up my point that there are tons of people capable of doing these calculations kicking around.
This is not rocket surgery. This is something that literally every private pilot with a Cessna does when they encounter a novel loading. Stop making it out to be some super scary thing requiring serious expertise to not screw up.
As a backup we had the old carbon slips that went into a plastic rig and made an impression of the customer's card. We also had a printed out sales tax table and some more carbon slips we could use to hand-write a receipt. I had to resort to the paper based system on many occasions.
A few years back I was in a store shopping when the power went out. The staff didn't know what to do and closed the store early. They had no way to sell me the items in my basket. I asked if they could take an impression of my card and got met with blank stares. They couldn't even take cash because they didn't know how to calculate the total amount, had nothing to write a receipt on, and their cash drawers wouldn't open anyway.
Nowadays credit cards don't even come with raised numbers...
So that's what the raised numbers were for. I just got my first card without raised numbers last January. It still feels fake and "insecure" to me, even though I know better. Funny how things work.
Seems relatively new, as some people look at funnily when I hand it over when they need it for "deposits" or whatever.
I've always wondered how that works for businesses that still prefer to compare the last four digits of a physical card with those of a booking made online, like some hotels or airlines still do.
So far, they've looked at me strangely, fetched a superior who either says "whatever" or they run through the card through their reader to get the last four. Seems to work out well enough at least.
Running it through a reader would also not work – the number on the chip and magnetic stripe is different from the one you see in the app and use for online payments. That's why I'd rather not use it for a hotel or international airline booking, for example (some still have "you must have the physical card with you to be allowed to board" in their terms and conditions, although nobody has ever requested to see mine in practice).
Do you know if your last four digits (that are printed on receipts too) match those you can see in the app?
Ah, first time I heard about it, seems to not be available outside the US so that explains it :)
> Do you know if your last four digits (that are printed on receipts too) match those you can see in the app?
Yes, it does. It also matches with whatever they see when they run the card, however they do that. The card number is identical for online payments, "offline" payments (with terminals) and what I see in my banking application.
The only thing that changes is the CVC, as it rotates every ~6 minutes or so.
This was literally a bit in the Y2K episode of King of the Hill. Hank goes to buy a new "Y2K compliant" computer. The cashier couldn't ring him up and freaks out.
And I saw an old card imprint thingo at an op shop (thrift store), slight rusty but seemed to still seemed to work.
I have only a single card with raised/embossed numbers, and that's somewhat ironically a pre-tax transit benefit card, i.e. the least likely of any I own to ever be used in a carbon imprinter.
The last one with raised numbers was delivered to me in 2024. The one without came a few months later, from the same bank. It also has the number on the back, right next to the static CVC. I guess security is indeed their top priority.
Anecdotally, all the credit cards I've had started to have the paint rub off the embossed number fairly quickly, rendering it very difficult to read. The other text on them didn't have this issue.
Arguably it's a wash for security: Yes, you can take a single photo of all the card details if the CVC is on the same side as the number and expiry, but on the other hand, there's nothing of value on the front now, which helps with people taking photos of their cards and posting them on social media.
And I agree, the paint rubbing off is extremely annoying, and even worse when it's metallic paint, flakes of which have a habit of deeply embedding themselves in all kinds of leather and fabric materials. That, and the fact that you can't fit as many of them in a "card-only wallet", makes me not want to carry any of them anymore.
If word gets out that a given store has no online processing and is accepting carbon slips again, you can bet that quite a few people with stashes of stolen/closed credit cards will show up and start buying the most expensive items in the store.
The modern equivalent would be offline chip processing. That is actually capable of at least ensuring that a given card is not heavily overdrawn or outright forged. Still, even that is unfortunately being scaled back significantly, given that "everything is always online anyway, so why do we need it?"
[0] https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu...
[1] https://en.wikipedia.org/wiki/2024_Delta_Air_Lines_disruptio...
Looking over the flights that were cancelled while I was at the airport (Hartsfield-Jackson Atlanta International, oddly enough), Delta was clearly prioritizing cancelling flights that had alternatives so as many travellers had the best shot at getting to their destination.
I've liked them ever since the Northwest days, and I don't expect that to change any time soon.
The alternatives that Delta booked for me were already going to be the next morning (Saturday), and I followed them after getting home: while the first leg worked fine, the second leg was cancelled late Saturday morning. It's likely that I'd have received yet another rebooking, but if that one also involved two legs it could easily have pushed my return date to Sunday, and by that point in the weekend I doubt I could have found openings on other airlines if I'd changed my mind.
I had coworkers who didn't make it back home from that conference until Monday night.
So yes, I think they tried hard to avoid inconveniencing people, but if you made it the same day you were lucky. Many didn't.
Curious if AA and other airlines have as well.
The amount of business logic involved is staggering, but miraculously there are experts who truly seem to understand almost all of it. Understanding “almost all of it” is not enough, so there are several different experts who in their entirety hopefully capture 99.99…% But you don’t know what you don’t know…
One of the “calming” facts is, that in the tens of thousands of test cases we compared our results to existing solutions, we did discover errors in their calculations. The “calming” aspect is, due to the many safe-guards these errors seemingly never surfaced.
Coming from a machine-learning background I quickly understood that there is little appetite for AI solutions, as any optimization which would reduce fuel consumption, is easily offset by all the safety margins. In my mind the goal of the software is to make everything explainable and proof that you truly did think of every eventuality.
I am very curious to learn what major f*-up lead to planes being grounded.
software dev on field they haven't worked in yet: they actually take quality seriously, right?