I am looking forward to see how this goes in court, as it can be yet another step forcing companies into proper quality development workflows, and liability.
I'm hoping it results in ClownStrike being sued into bankruptcy and the whole company being dissolved, and the CEO never getting another job again except maybe as a janitor. They need to stand as an example of what not to do.
CrowdStrike caused millions of computers to be stuck in an infinite reboot-to-BSOD cycle.
The CrowdStrike CEO admitted fault and apologized publicly. CrowdStrike then sent out 'fuck you' $10 uber eats as a sorry, further admitting fault to places.
The CrowdStrike CEO did this at McCaffee? Aswell as their CTO.
I get the desire to assign a simple “blame” to a single entity, but reality has a habit of being more complex.
Even if we accept that CrowdStrike’s fuckup was the root cause of all the issues that Delta customers experienced, the level of impact CS’s fuckup was able to have is entirely on Delta. Their choice of platform, their level of resilience, the processes they have in place for rapid recovery, etc, are all Delta’s responsibility.
No, you cannot be resilient against a company directly injecting stuff into their software without user input.
There is zero complexity here. CrowdStrike accepted money for security services and offered a client. Then they didn't bother to test their stuff and literally caused the biggest IT outage in history to save a few bucks. They literally have a CEO with a history of not caring.
If you offer your stuff for use in critical infrastructure you cannot do stuff like that. This company needs to be sued into bankruptcy. This was plain bad engineering.
If the same happened with a bridge, and it collapsed no one would even hesitate to lay blame on the engineering company/architects. You have a choice if you want to drive over a bridge or not, but when it collapses and you are on there I doubt you'll think I have to accept the consequences of my choices on your way down
> Delta had a 6 day recovery time, why were other airlines able to recovery faster?
> Perhaps Delta had a shittier recovery plan.
Or Delta has shittier software. News articles kept mentioning Delta's crew scheduling system couldn't keep up with its backlog after it was brought back online, and I take that to be the reason they had to cancel flights for so long.
If true, that's not Crowdstrike's fault, it's Delta's for not investing enough in their own mission-critical systems to make them resilient after an outage.
> You have a choice if you want to drive over a bridge or not, but when it collapses and you are on there I doubt you'll think I have to accept the consequences of my choices on your way down
I don't think it's unreasonable to hold a company with $60 billion annual revenue, and millions of customers to be affected by any faults, more responsible for inspecting the infrastructure they choose and having contingency plans in place than an individual with little means to realistically do that. Particuarly when, as you note, there was a history of not caring to save a few bucks.
If not CrowdStrike, it seems as though something like a ransomware attack could've caused the same situation.
CrowdStrike are responsible for causing the initial damage.
Delta are responsible for not having good disaster recovery systems, and for purchasing CrowdStrike's software.
It's worth checking for any warranty disclaimers, particularly disclaimers of the implied warranty of fitness for a particular purpose. If they exist, the software shouldn't be purchased. While the licenses to enterprises are private, their website's license explicitly disclaims such things as them installing viruses on your computer:
> AS BETWEEN YOU AND CROWDSTRIKE, YOUR USE OF THE WEBSITE IS AT YOUR OWN RISK. THE WEBSITE IS PROVIDED TO YOU BY CROWDSTRIKE ON AN "AS IS" AND "AS AVAILABLE" BASIS, WITHOUT ANY WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED. NEITHER CROWDSTRIKE NOR ANY PERSON ASSOCIATED WITH CROWDSTRIKE MAKES ANY WARRANTY OR REPRESENTATION WITH RESPECT TO THE COMPLETENESS, SECURITY, RELIABILITY, QUALITY, SUITABILITY, ACCURACY OR AVAILABILITY OF THE WEBSITE. WITHOUT LIMITING THE FOREGOING, NEITHER CROWDSTRIKE NOR ANYONE ASSOCIATED WITH CROWDSTRIKE REPRESENTS OR WARRANTS THAT THE WEBSITE, INCLUDING THE WEBSITE CONTENT OR ANY PRODUCTS, SERVICES OR MATERIALS OBTAINED THROUGH THE WEBSITE, WILL BE ACCURATE, RELIABLE, ERROR‑FREE OR UNINTERRUPTED, THAT DEFECTS WILL BE CORRECTED, THAT OUR SITE OR THE SERVER THAT MAKES IT AVAILABLE ARE FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS OR THAT THE WEBSITE OR ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE WILL OTHERWISE MEET YOUR NEEDS OR EXPECTATIONS.
If your AV software's publicly facing website needs to disclaim liability for "viruses or other harmful components" that their site may serve up, you really ought to think twice about giving them money. After all, if it were a useful anti-virus then they wouldn't need to worry about people getting viruses from their own site, they'd just use their own product.
> No, you cannot be resilient against a company directly injecting stuff into their software without user input.
Well, if you as a customer decide to install said trojan software on all your IT equipments, you should be ready to have to recover all those IT equipment at any point in time. That is that simple. If you can't do that in a reasonnable time, that is on you as well.
You definitely can be resistant to a company directly injecting stuff into their software: don't make it company policy to install malware on all machines. Crowdstrike is a shit company. But the companies affected really have no else to blame but themselves for such idiotic IT.
In some industries, like vehicles, if the product has a malfunction, the government requires the mfg to issue a recall and repair the issue, even long after the warranty ends. And same thing with children's toys via the CPSC.
Yet in software this expectation is always absent.
There is no such thing as a root cause of the failure. There's over 40 years of academic research on the subject providing evidence, but still, here we are. Trying to reduce an incident into a single cause.
I think the Uber Eats voucher was intended to be an internal thing, like “sorry everyone, we’ve had a rough Friday, probably things won’t be smooth again for a while, here, order yourself a pizza while we work this through together” rather than something to be offered to customers, and it leaked.
To a certain extent I get it, although the root cause was the CrowdStrike issue, the knock-on effects were exacerbated by Delta’s duck-tape-and-prayers existing IT services.
Makes it harder to accept responsibility for that when you’re being sued.
There is a pie chart of accountability, including some with Delta themselves. Microsoft carry some blame, but yes, the lion share is carried by CS.
Delta took 6 days to recover from something as impactful as this, which is much worse than any other airline. So much so, the US Transportation Department is currently reviewing why it took so long.
It is vital that any critical infrastructure such as this has a low MTTR, which Delta clearly failed to plan and test for, therefore the IMPACT that Delta felt cannot be solely attributed to CS, albeit they did trigger it, but Delta failed to mitigate the impact.
By not providing a sane interface for security tools to exist. It should not have been possible for CS to fubar the whole system like this.
Imagine if your bios vendor made it trivial for an application to brick your hardware. Yes, you'd blame the software, but you'd also be asking why the bios didn't protect the system.
In Linux eBPF is a sane interface which CS now use. MS are now developing their own eBFP interface, but it's only a early concept.
According to former Windows developer David Plummer, Microsoft does, in fact, offer a number of APIs for third-party antivirus security. “CrowdStrike defaults to kernel mode, presumably because it needs to do things that can’t be done from user mode,” Plummer said in a YouTube video.
“And to me, that’s where Microsoft could be responsible, because on the Windows platform, to the best of my knowledge, some of the CrowdStrike security functionality requires deep integration with the operating system that can only be currently achieved on the kernel side.”
Microsoft has a number of APIs including Windows Defender Application Control API and the Windows Defender Device Guard, which Plummer said provide mechanisms for controlling application execution and ensuring that only trusted code runs on the operating system.
He said that the Windows Filtering Platform (WFP) allows applications to interact with the network stack without requiring kernel level code. However, quoting sources within Microsoft, Plummer claimed that the company had actually “tried to do the right thing” by developing an advanced API designed specifically for security applications such as that from CrowdStrike.
“This API promised deeper integration with the Windows operating system, offering enhanced stability, performance and security,” he added.
But the EU 2009 ruling effectively prevented such integration as it could potentially have given Microsoft an unfair advantage.
However, Ian Brown, an independent consultant on internet regulation, argued that Microsoft should have better security controls, rather than attempting to put the blame of the CrowdStrike crash on the EU anti-competition commission.
In a blog, he wrote: “For technology-dependent societies’ resilience, OS kernel-level software and equivalents on socially critical infrastructure systems (like travel, healthcare and banking) need to be very carefully tested (and ideally run on top of a formally verified microkernel) and controlled. But OS monopolists shouldn’t be making the final decisions about precisely what those controls look like, where they have implications for competition.”
People who gave a third party firm remote code execution control of millions of computers without any code review, reproducible builds, or accountability of any kind.
If you give third parties access to your systems with proprietary code without any checks or balances because you are too lazy to fight for the budget to deploy an in-house alternative, then whatever bad results happen are partly on you.
Yes, sue CrowdStrike, but also fire every sysadmin and IT manager that recommended or approved installing it on systems in the first place.
So when you go to a restaurant, you also go into the kitchen, review the origin seal of every ingredient, the tools using to prepare the dish, perform a background check of each kitchen employee, before commiting to anything on the menu...
The info was basically on the crowdstrike whitepapers. You don't have to know the origin seal of every ingredient, but if you are allergic to peanuts, you ask if said dish has peanuts in it.
Asking what the falcon agent does, how it is updated, what it has access to, and if it can be remotely controlled is part of the job of any IT engineer / manager who plans to purchase a license for crowdstrike. If you don't, you aren't doing your job.
Other airlines who were also similarly affected by the crowd strike outage did not display the same prolonged issues as Delta.
CrowdStrike was clearly responsible for some of it. It’s pretty clear that Delta had other major issues that were also triggered by the CrowdStrike outage which it wasnt responsible for since other airlines didn’t face those issues.
> Other airlines who were also similarly affected by the crowd strike outage did not display the same prolonged issues as Delta.
You raise a valid point, but imo the key question (to which I don't claim to have an answer) is whether Delta's extended problems are due more to their disorganization or simply to bad luck.
Here's a somewhat silly analogy: Imagine Kellogg's introduces a new ingredient in their cereals that triggers an adverse reaction only in people who have eaten sardines within the last hour. It would still be correct to hold Kellogg's entirely responsible for the adverse reactions experienced by those individuals. It wouldn't be a valid defense for Kellogg's to say, "Most people didn't experience any issues, so it can't be our product causing your adverse reactions".
Sandbox rules simply don’t apply when real money is at stake— the contracts that sit behind these relationships are all that matters + a companies ability to stop doing business with one another.
Delta probably isn’t even entitied to a pro-rated refund of their prepaid CrowdStrike subscription. If Delta has a multi-year deal contract with CrowdStrike, Delta most likely have to keep paying CS for some time In the future.
CrowdStrike breached but almost certainly cured within allowable period.
Maybe they sue for gross negligence which I think may circumvent contractual liability limits in certain situations.
Delta's IT department is in for tough times ahead, considering these cases drags on for years..
Should Delta pursue this path, Delta will have to explain.. why CrowdStrike took responsibility for its actions—swiftly, transparently, and constructively while Delta did not.. Delta would have to preserve a series of documents, including those describing its information-technology infrastructure, IT business continuity plans and its handling of outages in the past five years
Wait, were Crowdstrike seriously suggesting giving their staff physical access to Delta systems at airports across the world having just nuked those same systems through incompetence or negligence... what....
We prosecute arsonists and look towards implementing better forest management to ensure future arsonists don't have as much impact. Just like with personal crime, we prosecute the perpetrators and teach potential victims how to better protect themselves from being in vulnerable positions. Delta isn't at fault for the failure but they do need to reevaluate their systems to ensure this kind of total system failure will be less likely.
The crew location and status systems are all manual and phone based this is a chronic problem across many airlines.
When the system goes down it takes hours or days as crew all have to call in, wait on hold for hours and get their status back in the system.
If only some smart tech people gathered somewhere and someone could make a mobile app to allow crew to set their status and location instantly. They’d corner the market and save airlines billions.
45 comments
[ 4.3 ms ] story [ 70.4 ms ] threadif CrowdStrike isn't to blame, then who is?
Even if we accept that CrowdStrike’s fuckup was the root cause of all the issues that Delta customers experienced, the level of impact CS’s fuckup was able to have is entirely on Delta. Their choice of platform, their level of resilience, the processes they have in place for rapid recovery, etc, are all Delta’s responsibility.
There is zero complexity here. CrowdStrike accepted money for security services and offered a client. Then they didn't bother to test their stuff and literally caused the biggest IT outage in history to save a few bucks. They literally have a CEO with a history of not caring.
If you offer your stuff for use in critical infrastructure you cannot do stuff like that. This company needs to be sued into bankruptcy. This was plain bad engineering. If the same happened with a bridge, and it collapsed no one would even hesitate to lay blame on the engineering company/architects. You have a choice if you want to drive over a bridge or not, but when it collapses and you are on there I doubt you'll think I have to accept the consequences of my choices on your way down
Perhaps Delta had a shittier recovery plan.
> Perhaps Delta had a shittier recovery plan.
Or Delta has shittier software. News articles kept mentioning Delta's crew scheduling system couldn't keep up with its backlog after it was brought back online, and I take that to be the reason they had to cancel flights for so long.
If true, that's not Crowdstrike's fault, it's Delta's for not investing enough in their own mission-critical systems to make them resilient after an outage.
I don't think it's unreasonable to hold a company with $60 billion annual revenue, and millions of customers to be affected by any faults, more responsible for inspecting the infrastructure they choose and having contingency plans in place than an individual with little means to realistically do that. Particuarly when, as you note, there was a history of not caring to save a few bucks.
If not CrowdStrike, it seems as though something like a ransomware attack could've caused the same situation.
Delta are responsible for not having good disaster recovery systems, and for purchasing CrowdStrike's software.
It's worth checking for any warranty disclaimers, particularly disclaimers of the implied warranty of fitness for a particular purpose. If they exist, the software shouldn't be purchased. While the licenses to enterprises are private, their website's license explicitly disclaims such things as them installing viruses on your computer:
> AS BETWEEN YOU AND CROWDSTRIKE, YOUR USE OF THE WEBSITE IS AT YOUR OWN RISK. THE WEBSITE IS PROVIDED TO YOU BY CROWDSTRIKE ON AN "AS IS" AND "AS AVAILABLE" BASIS, WITHOUT ANY WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED. NEITHER CROWDSTRIKE NOR ANY PERSON ASSOCIATED WITH CROWDSTRIKE MAKES ANY WARRANTY OR REPRESENTATION WITH RESPECT TO THE COMPLETENESS, SECURITY, RELIABILITY, QUALITY, SUITABILITY, ACCURACY OR AVAILABILITY OF THE WEBSITE. WITHOUT LIMITING THE FOREGOING, NEITHER CROWDSTRIKE NOR ANYONE ASSOCIATED WITH CROWDSTRIKE REPRESENTS OR WARRANTS THAT THE WEBSITE, INCLUDING THE WEBSITE CONTENT OR ANY PRODUCTS, SERVICES OR MATERIALS OBTAINED THROUGH THE WEBSITE, WILL BE ACCURATE, RELIABLE, ERROR‑FREE OR UNINTERRUPTED, THAT DEFECTS WILL BE CORRECTED, THAT OUR SITE OR THE SERVER THAT MAKES IT AVAILABLE ARE FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS OR THAT THE WEBSITE OR ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE WILL OTHERWISE MEET YOUR NEEDS OR EXPECTATIONS.
If your AV software's publicly facing website needs to disclaim liability for "viruses or other harmful components" that their site may serve up, you really ought to think twice about giving them money. After all, if it were a useful anti-virus then they wouldn't need to worry about people getting viruses from their own site, they'd just use their own product.
Well, if you as a customer decide to install said trojan software on all your IT equipments, you should be ready to have to recover all those IT equipment at any point in time. That is that simple. If you can't do that in a reasonnable time, that is on you as well.
Yet in software this expectation is always absent.
To a certain extent I get it, although the root cause was the CrowdStrike issue, the knock-on effects were exacerbated by Delta’s duck-tape-and-prayers existing IT services.
Makes it harder to accept responsibility for that when you’re being sued.
Delta took 6 days to recover from something as impactful as this, which is much worse than any other airline. So much so, the US Transportation Department is currently reviewing why it took so long.
It is vital that any critical infrastructure such as this has a low MTTR, which Delta clearly failed to plan and test for, therefore the IMPACT that Delta felt cannot be solely attributed to CS, albeit they did trigger it, but Delta failed to mitigate the impact.
Otherwise they were forced to NOT lock down the kernel at Vista time.
Imagine if your bios vendor made it trivial for an application to brick your hardware. Yes, you'd blame the software, but you'd also be asking why the bios didn't protect the system.
In Linux eBPF is a sane interface which CS now use. MS are now developing their own eBFP interface, but it's only a early concept.
According to former Windows developer David Plummer, Microsoft does, in fact, offer a number of APIs for third-party antivirus security. “CrowdStrike defaults to kernel mode, presumably because it needs to do things that can’t be done from user mode,” Plummer said in a YouTube video.
“And to me, that’s where Microsoft could be responsible, because on the Windows platform, to the best of my knowledge, some of the CrowdStrike security functionality requires deep integration with the operating system that can only be currently achieved on the kernel side.”
Microsoft has a number of APIs including Windows Defender Application Control API and the Windows Defender Device Guard, which Plummer said provide mechanisms for controlling application execution and ensuring that only trusted code runs on the operating system.
He said that the Windows Filtering Platform (WFP) allows applications to interact with the network stack without requiring kernel level code. However, quoting sources within Microsoft, Plummer claimed that the company had actually “tried to do the right thing” by developing an advanced API designed specifically for security applications such as that from CrowdStrike.
“This API promised deeper integration with the Windows operating system, offering enhanced stability, performance and security,” he added.
But the EU 2009 ruling effectively prevented such integration as it could potentially have given Microsoft an unfair advantage.
However, Ian Brown, an independent consultant on internet regulation, argued that Microsoft should have better security controls, rather than attempting to put the blame of the CrowdStrike crash on the EU anti-competition commission.
In a blog, he wrote: “For technology-dependent societies’ resilience, OS kernel-level software and equivalents on socially critical infrastructure systems (like travel, healthcare and banking) need to be very carefully tested (and ideally run on top of a formally verified microkernel) and controlled. But OS monopolists shouldn’t be making the final decisions about precisely what those controls look like, where they have implications for competition.”
If you give third parties access to your systems with proprietary code without any checks or balances because you are too lazy to fight for the budget to deploy an in-house alternative, then whatever bad results happen are partly on you.
Yes, sue CrowdStrike, but also fire every sysadmin and IT manager that recommended or approved installing it on systems in the first place.
No one acting alone can deploy a change to mission critical systems I design.
This is just basic supply chain integrity.
Asking what the falcon agent does, how it is updated, what it has access to, and if it can be remotely controlled is part of the job of any IT engineer / manager who plans to purchase a license for crowdstrike. If you don't, you aren't doing your job.
I guess it is your fault, because you failed to do the allergic chemical test, before eating it.
CrowdStrike was clearly responsible for some of it. It’s pretty clear that Delta had other major issues that were also triggered by the CrowdStrike outage which it wasnt responsible for since other airlines didn’t face those issues.
You raise a valid point, but imo the key question (to which I don't claim to have an answer) is whether Delta's extended problems are due more to their disorganization or simply to bad luck.
Here's a somewhat silly analogy: Imagine Kellogg's introduces a new ingredient in their cereals that triggers an adverse reaction only in people who have eaten sardines within the last hour. It would still be correct to hold Kellogg's entirely responsible for the adverse reactions experienced by those individuals. It wouldn't be a valid defense for Kellogg's to say, "Most people didn't experience any issues, so it can't be our product causing your adverse reactions".
Sandbox rules simply don’t apply when real money is at stake— the contracts that sit behind these relationships are all that matters + a companies ability to stop doing business with one another.
Delta probably isn’t even entitied to a pro-rated refund of their prepaid CrowdStrike subscription. If Delta has a multi-year deal contract with CrowdStrike, Delta most likely have to keep paying CS for some time In the future.
CrowdStrike breached but almost certainly cured within allowable period.
Maybe they sue for gross negligence which I think may circumvent contractual liability limits in certain situations.
Should Delta pursue this path, Delta will have to explain.. why CrowdStrike took responsibility for its actions—swiftly, transparently, and constructively while Delta did not.. Delta would have to preserve a series of documents, including those describing its information-technology infrastructure, IT business continuity plans and its handling of outages in the past five years
It was no secret CrowdStrike updated all PCs at once. Delta could see that. It is no secret updates can nuke computers. Delta knew this could happen.
All business should have a plan for cascading/total outages.
This was a great test of humanity that Delta seems to have failed.
Deltas total incompetence makes me feel like CrowdStrike incompetence is canceled out. Delta's CEO clearly has no idea what he is talking about.
But Delta for having terrible investment in modernizing their IT infrastructure.
This is very likely to settle out of court or dropped once the CS outage falls out of the news cycle.
If push comes to shove, Delta can sue and/or stop using the product.
This is ultimately a question of contracts, liability limits— particularly if Delta secured consequential damages.
SaaS contracts are designed to defaulted to NOT allow a customer to pursue consequential damages remedies.
https://en.m.wikipedia.org/wiki/Consequential_damages
This is a question of CrowdStrike’s Deal Desk contracting hygiene.
Deal Desks are the joint finance-legal-sales teams that work on enterprise contracts in scaled enterprise SaaS startups.
This is a SaaS CFOs nightmare.
Society still blames the match based on recent legal outcomes, so Delta will probably win the argument.
If only some smart tech people gathered somewhere and someone could make a mobile app to allow crew to set their status and location instantly. They’d corner the market and save airlines billions.