The outages cost Delta an estimated $350 million to $500 million.
Delta is dealing with over 176,000 refund or reimbursement requests after almost 7,000 flights were canceled.
The lower estimate divided by the number of refunds is (suspiciously?) close to $2000. Do they sell tickets for this much on average or is there some other major cost?
P.S. On the other hand, 176000/7000 ≈ 25 refunds per flight is a bit low. Maybe the average ticket is two-way for a family. Then $2000 is not that much.
One thing that's not immediately obvious for passenger planes is that the cost of passenger tickets rarely justify the expense of the flight itself. The movement of cargo on these planes is really what keeps the airline profitable. Losses associated with cargo delay can run the number up very quickly.
Cargo on passenger planes is so profitable that I was on an international flight from Atlanta to Seoul and Delta was literally bumping passengers off the plane so they could carry more cargo. They bumped a ridiculous number of people off the flight for weight savings because the cargo is more profitable than the people.
Not sure Crowdstrike as a company survives the onslaught of legal issues and reputational damage. At least, as a viable growth business. Maybe, at all.
That's just one class-action judgment from being $0.00.
I don't think it will happen, but we're not talking about a "just restore from backup" situation. When you look at the fact that, statistically, at least one person likely died (or will die) from this due to the problems hospitals had, and that some customers are still dealing with the fallout... I think they're significantly financially compromised after all of the dust settles.
Why didn’t Delta use Linux to begin with instead of unreliable Windows systems bundled with lots of auto-updating addons where you have no idea what’s going on?
If it ran in user mode that meant any other program - including viruses - could easily tamper with it. Besides, antivirus software that ran in user mode couldn’t prevent boot time viruses.
Of course your operating system vendor will always have access to kernel and increased permissions to do things that other software won’t have.
"to begin with" is expecting miracles we don't have today. Delta has been around forever and presumably runs a lot of critical stuff was written in the 90s, or (hopefully not by now) even the 60s. And, as cool as Linux is as an OS, many smart people program in C#/MSFT-stack, and you don't want to remove them from a talent search if you're Delta and need 1000s of developers. And this CrowdStrike thing wouldn't have been mitigated by running a Windows App in a container, it would have killed the Windows OS supporting the container (I'm pretty sure...).
"Premature optimization is the root of all evil" is a common architecture guideline. Well, in a way Delta (and any company 30+ years old) optimized all their business processes using old coding standards. Now they have to effectively change the oil while the car is driving down the highway. Whoever figures out how to get AI to assist in this will make the first AI fortune.
Ok. But I think it's reasonable to ask why flight arrival information displays have to run Windows. And even if they have to run Windows why not some embedded ROM-boot library kiosk version? Why does a sign need to have a fully featured desktop OS?
(I have a picture of said sign displaying its blue screen lest anyone think I'm making this up).
Maybe because airline traffic is a hard business (airlines make cents per passenger per flight).
And thus airports are pressured by airlines to minimize costs. In the end, the answer is probably, that it's just cheaper to use windows than to invest any more effort into the task.
Best answer I can guess: because it's easy and cheap to set up, and the dev team involved was moved to another project years ago.
Windows is dominant for UIs because of a long history of having drivers for all the various peripherals which eases hw procurement issues. Sure that advantage went away long ago but the momentum takes a long time to die out. The brains with the expertise know how to do it on Windows. Combine that with the fact that Windows UI is not something you have to train new hires on and it makes business sense that all the user-facing interfaces ended up Windows applications. Sure, over time they should be migrated to containerized web UIs, but there's a lot of work to be done. Updating the flight arrival display isn't mission critical; it's at the end of the backlog.
I believe I heard one thing that got screwed up was a crew-scheduler product. Like: they couldn't check in crews or get their schedules written in so a system (presumably something on a mainframe) could work out where all the people needed to be at what times. Much more mission critical! The central big brain was (probably) not Windows, but those user inputs at the end were enough to screw up the whole business process.
Crowdstrike has been crashing Linux too, so what do you expect that to accomplish? Both are running kernel drivers, if they illegally access unallocated memory the kernel is meant to fail-safe.
Microsoft has deep pockets so why not throw lawsuits at the wall and see if they stick, but from a technical perspective Microsoft didn't do anything wrong here.
Now, do Microsoft need to redesign Windows to allow a larger percentage of security software to run in usermode? Yes. In the same way they redesigned it to run graphics and sound drivers in username because of the poor quality of third party vendors.
That's partially why it's hard. People are automatically unhappy if their flight is delayed, their bags take time to arrive, they have to gate check a bag, sit near a baby during the flight, etc. It would be very hard to run perfectly with the weather being such a large input factor. Even food tastes worse up 35k feet due to the lack of moisture and air pressure. Think about how unhappy people would be with Apple if every time it rained or snowed, the phone couldn't function due to physics?
I'm no lawyer, but I have to imagine that if Delta can successfully sue CrowdStrike for this, enough other lawsuits will follow that CrowdStrike would be forced into bankruptcy.
Valued at and have enough cash for are two different things. Their value relies on them not selling billions in stocks to cover a lawsuit. As they sell their stock will go down as will their valuation. Also losing multiple lawsuits can cause investors to get spooked further lowering the stock price.
36 comments
[ 2.4 ms ] story [ 74.0 ms ] threadP.S. On the other hand, 176000/7000 ≈ 25 refunds per flight is a bit low. Maybe the average ticket is two-way for a family. Then $2000 is not that much.
Cargo on passenger planes is so profitable that I was on an international flight from Atlanta to Seoul and Delta was literally bumping passengers off the plane so they could carry more cargo. They bumped a ridiculous number of people off the flight for weight savings because the cargo is more profitable than the people.
[1] https://www.macrotrends.net/stocks/charts/CRWD/crowdstrike/c...
I don't think it will happen, but we're not talking about a "just restore from backup" situation. When you look at the fact that, statistically, at least one person likely died (or will die) from this due to the problems hospitals had, and that some customers are still dealing with the fallout... I think they're significantly financially compromised after all of the dust settles.
https://www.theregister.com/AMP/2024/07/21/crowdstrike_linux...
Microsoft was forced to allow kernel access into Windows by third parties by the EU.
Over a decade ago, MS wanted to lock down the kernel. Anti virus vendors complained and the EU and MS came to an “understanding”
The regulation was about that they - MS - cannot use something others - Anti-Virus Vendors - could not.
Of course your operating system vendor will always have access to kernel and increased permissions to do things that other software won’t have.
"Premature optimization is the root of all evil" is a common architecture guideline. Well, in a way Delta (and any company 30+ years old) optimized all their business processes using old coding standards. Now they have to effectively change the oil while the car is driving down the highway. Whoever figures out how to get AI to assist in this will make the first AI fortune.
(I have a picture of said sign displaying its blue screen lest anyone think I'm making this up).
And thus airports are pressured by airlines to minimize costs. In the end, the answer is probably, that it's just cheaper to use windows than to invest any more effort into the task.
Windows is dominant for UIs because of a long history of having drivers for all the various peripherals which eases hw procurement issues. Sure that advantage went away long ago but the momentum takes a long time to die out. The brains with the expertise know how to do it on Windows. Combine that with the fact that Windows UI is not something you have to train new hires on and it makes business sense that all the user-facing interfaces ended up Windows applications. Sure, over time they should be migrated to containerized web UIs, but there's a lot of work to be done. Updating the flight arrival display isn't mission critical; it's at the end of the backlog.
I believe I heard one thing that got screwed up was a crew-scheduler product. Like: they couldn't check in crews or get their schedules written in so a system (presumably something on a mainframe) could work out where all the people needed to be at what times. Much more mission critical! The central big brain was (probably) not Windows, but those user inputs at the end were enough to screw up the whole business process.
Now, do Microsoft need to redesign Windows to allow a larger percentage of security software to run in usermode? Yes. In the same way they redesigned it to run graphics and sound drivers in username because of the poor quality of third party vendors.
The damages are estimated at $5.4B.
I think they can survive this. Though their reputation took a much bigger hit.
Its not impossible