> We hope he's not flying Delta to Washington DC, as the US airline has canceled more than 5,000 flights since Friday as a result of CrowdStrike's update file crashing Windows systems.
Airlines is bad, but I am more concerned about all the hospitals that shut down entirely or turned away critical surgeries that they suddenly deemed “elective”. Even big prominent hospitals like Mass General! It’s utterly ridiculous but also just a massive risk to public safety and national security that something like this can happen. None of these critical pieces of our society have any real business continuity plan, and it means nothing to them to inconvenience citizens, even if that inconvenience is actually going to result in death.
You’re assuming it’s boolean though. I guarantee all organisations with decent BC would have performed measurably better than those without it, even if both groups had consistent BSODs.
The entire industry has a monopoly problem with the expected set of related interoperability problems. There's no logical reason we should be running so much infrastructure on Microsoft's monoculture.
If "Linux" mean RHEL, and "same thing" mean "redhat eBPF implementation was buggy and caused kernel panic after a few minutes, but you could deactivate it", i'm aware.
What's absurd about it? It's absurd that I want the software operating my MRI and providing doctors with necessary information during surgeries to be different from what accountants using Excel and video game players use?
What's absurd is congress hoping to solve the fundamental problem by asking the CrowdStrike CEO to come in for a one on one. Even if we took this process at face value and assumed they were earnest it would be laughable.
Meanwhile.. the government is the largest healthcare payer in the world. For not particularly great reasons. This _is_ one of the downstream impacts of that reality.
Yes it is absurd to rewrite your entire software stack when 99% of scenarios have the same requirements: CPU, I/O, drivers. You know how many bugs you have with one stack? Multiply by 2x for each of your custom OSes. I’ve seen the fragmentation in edge devices and no thanks, I don’t want more “secure” “proprietary” software.
This has nothing to do with there being a healthy market of operating systems to choose from.
> I’ve seen the fragmentation in edge devices
How many versions and patch levels of Windows exist? How many are currently deployed? How many must stay on the current version because of fragmentation?
> You know how many bugs you have with one stack?
The scenario that started this discussion is a single bug in a single piece of software impacting a large number of businesses, some of which are safety and life critical, all of which are on a single OS.
It's not a question of "how many bugs," because you can't fully eliminate them in any system which is actually updated _ever_, it's a question of "how many people does this single bug impact?"
Will this require more time and labor? Yes, of course. Does that make it "absurd" to consider? No, that's pure hyperbole.
The problem at hand is not one of monopoly, but of awareness of what is actual infrastructure and what is not. More and more systems enter the gray area towards becoming full blown infrastructure, but at the same time nothing is done to increase their reliability accordingly. Worse, the ever increasing complexity also hides an increasing number of weak spots in the systems. This incident shows exactly that, but just look at the comments ... it is NOT the wake-up call it needs to be, the discussions are going sideways in all sorts of directions, even among those who should see the things for what they are.
For airlines there are separate risk assessments for flight ops and revenue ops.
If the website fails, that costs a lot of money, but nobody is in danger. If dispatch systems fail that has a safety risk to flight ops.
While you may not be happy that your flight gets cancelled. For airline IT there is a cost vs benefit balance between the probability of flight cancellations vs the cost of having extra systems. (And in this case extra systems would probably have been affected by the same security platform, because not having that is another risk of cyber attack to consider)
That doesn't apply to flight ops, where each operator needs to prove to a very high level that safety is always maintained. Maximum probability for catastrophic failure is 10E-9 and for hazardous is 10E-7. No commercial considerations considered, have to make those numbers to be allowed to operate.
Airlines really aren't that big deal. Once you land and park a plane. Well not too much will happen. Only thing after dust settles is to reconcile maintenance records.
Then again with systems working Boeing failed that one...
> Once you land and park a plane. Well not too much will happen.
Except for massive disruptions to tons of people, businesses, and economies worldwide, with impacts rippling out for days after services have been restored?
I don't think "CS are idiots and use their kernel-level trojan to accidentally crash every Windows computer it's running on across the whole world" was ever on anyone's threat board[0]. Maybe they have considered IT outage of their own systems, but this wasn't that - this hit everyone all at once.
Should hospitals plan on not being able to provide medical care and not able to transfer their patients elsewhere, or divert the inflow of new patients elsewhere, because every other hospital is down too, there is no one left to provide care, and 911 is shot anyway? Maybe. Right after planning for a nuclear explosion over the city.
What happened was pretty much an "act of God"-level crisis, except caused by a de-facto infrastructure provider with way more destructive power than they're equipped to handle. IMO, not only CrowdStrike should be liable for the damage and lives lost because of the outage, this incident should be a prompt to rethink the entire idea of endpoint security.
--
[0] - I think it should be, but then I think the entire business with endpoint protection is bullshit.
> I don't think "CS are idiots and use their kernel-level trojan to accidentally crash every Windows computer it's running on across the whole world" was ever on anyone's threat board.
No, but cyberattack takes out all computer infrastructure is a high impact, low risk that critical infrastructure providers must account for and would've prevented any impact from CrowdStrike's blunder. They obviously didn't do that.
Your IT department didn't choose it, I promise. Someone asked how to mitigate a risk, and the answer was spend money doing things well, or spend less money on a CYA solution and pass the buck.
That decision maker was promoted and since left for a new leadership position.
CrowdStrike is one of the products that no one wants or choses. It's a product that's forced down the throat of a business because of compliance. Hence the company has zero incentive to invest into actual security.
SentinelOne and CrowdStrike are arguably the best on the market.
Usually both are out of sight and out of mind. Curious what problems you’re having
Also, you probably don’t want to be responsible for causing your company to get ransomwared or data breached, which is the primary reason you won’t get any sympathy or special treatment.
Is it going to be like many of these hearings where they just go: "I don't recall," "I'll get back to you on that," "I’m not aware of those specifics at this time"? Hopefully not, and there will be some real answers and, more importantly, real repercussions later on.
It was just the other day I read a comment on HN on how CEOs should not be responsible for what occurs in the company they are leading. It should only be the frontline workers who should be punished never the management.
I thought I was taking crazy pills when I read that and started wondering where the world was headed. Im glad atleast the CEOs are expected to give answers.
The head of US secret service just got 'resigned' after being double body slammed by Republicans and Democrats in congress on live TV. Still that is getting off lightly for a catastrophic failure 10x worse than crowdstrike.
Yes, it's deeply worrying. And the source is Less Wrong of all places, an author with what seems like a solid track record and lots of karma. Really makes you wonder where we're heading.
Ok, do you think 1000 is way too low, or way too high?
E.g., How many life-threatening emergencies do you think happen per hour across the world? How much extra time did it take to reach the average injured person over the outage period? How was the base rate of iatrogenic death affected by losing access to all electronic records for inpatients?
If you come up with a fermi estimate more than two orders of magnitude lower than 1,000, I will be suspicious of your assumptions.
You have no idea how wired this country is. HN is a nerd cluster with zero social awareness. Half this site would die within 30 days of an insurgency kicking off hardcore.
I don't think JFK's assassination supports this though because that led to President Johnson, the Civil Rights Act, various memorials and some interesting X-Files episodes and not a civil war.
It doesn't have to be. They're disproving your claim that it will lead to civil war by showing that when that event happened previously, a civil war did not break out. It _might_ happen, sure, but it's not as sure of a thing as you're making it sound, either.
The US is very divided currently, I agree, but I don't think any of us know what would have happened had Trump been assassinated, and anyone claiming otherwise is either too confident in their theorycrafting or trying to sell you something.
You don't have to possess a crystal ball to draw a rough conclusion with confidence. Here, watch this, I'll show you: If Trump were to have been assassinated in 4K high def with his head blown off filmed perfectly from multiple angles, this country would see some of the most raw shit known to mankind (domestic terrorism). There is only one group of people who is armed up to the teeth in America and it is not the Democrats. If Trump would have been assassinated it would have been open season on Democrats and anyone not hardcore-right. Anyone who doesn't acknowledge this is simply a naïve fool or willfully lying to you. Raw power is still power, even if you hate it.
I don't see any moral reasons why 10 random people being killed wouldn't be worse than a single guy who also is 70+ yo, no matter how rich or how powerfull this guy is.
Also, you can't have a civil war as long as the military in the US is what it currently is, i.e subordinate to civilian powers. You can have riots, protests, sure, but i don't see any path for a civil war that does not somehow involve part of the military, in any country at any time, and especially now that armies are professionals.
There would have definitely been very acrimonious protests attended by people who are much more likely to own firearms than the population at large. And protests can lead to civil war, look at Syria for example.
Not to say it’s certain or even likely, but it’s definitely plausible. You are much more confident than I am that the military and police would remain 100% loyal to Biden during a protracted period of civil unrest where one side believes two general elections in a row were stolen from them and their leader assassinated.
Why was it 10 times worse? I would suspect the crowdstrike incident caused significant loss of life (emergency services in several countries were taken down, not even counting the hospitals etc.)
Shareholders will also care, or should also care, I know if I directly held shares in CrowdStrike I would want the CEO gone. The real problem is that the companies that hold shares on behalf of other people, like BlackRock, don't care.
Reform of companies like BlackRock is sorely needed, the way they operate is detrimental to the actual owners of equity, and society at large.
BlackRock's revenue is not based on the performance of the equity in their clients' portfolios, and while it may have some impact on their profit, it's not as big an impact as it has on their actual clients.
Furthermore, most of the money managed by BlackRock is not managed by them through direct choice of the equity holders, so even if the equity holders were not happy with BlackRock it would be very difficult for them to move their money elsewhere.
And then finally, the shares of BlackRock itself are also held on behalf of others by institutions like BlackRock, and some of it even by BlackRock itself. So they vote with the voting rights of others as to how they think they should be run.
The professional managerial class generally does not care about profits, as they exceed at finding ways of enriching themselves without having to actually deliver anything. They get rich while the average Joe's pension underperforms. They love things like DEI and ESG, as that allows them to define their own parameters for success in ways which in no way benefit the people they have a fiduciary duty to or society.
BlackRock's revenue comes from financial services. You pay them to invest your pension, they charge you based on how many of your assets are under their management, they don't get a performance-based fee. Currently, they have 9.101 trillion USD under management. This is not their money, this is the money of their clients, but their clients pay them, and they take the voting rights associated with the shares that they manage.
Technically, if your investment does well, they have more assets under managemnt, and they earn a larger fee, but in practice they can just increase their rate because the people using them aren't shopping around anyway, and there will always be more money invested through them.
In reality, most people are paying BlackRock without ever knowing that they are paying them, and then in addition to paying BlackRock in money they get the benefit of your voting rights, which they can then do with as they see fit. They are power brokers who are paid by the people whose power they are brokering.
"A loss of X dollars is always the responsibility of an executive whose financial responsibility exceeds X dollars."
- Weinberg’s 'First Principle of Financial Management' and 'Second Rule of Failure Prevention’
It seems to me that there's a need for the Crowdstrike CEO to take responsibility, e.g. for lax testing and for not doing canary rollouts.
And it seems to me that there's a place for the [airline|bank|hospital] CEOs and CIOs to consider implementing canary 'roll-ins' where new software is tried on a small scale before being installed everywhere.
How does this analogy work: Producers and regulators (or, if you prefer, the market) play an important role in managing food safety but they are not a replacement for our immune systems. Both are necessary.
Security software is often an security nightmare by itself. The amount of vulnerabilities you see in the products themselves is worrying. They try to offer protection but create a new huge attack vector directly inside the kernel with full permissions. Not even talking about the huge performance hit and the many false positives. IMHO way too much risk/inconvenience and too little benefits to be worth it.
Yes, Crowdstrike is yet another opportunity to feel vindicated that (when I was a CTO) I made sure our information security policies mandated people to use only the macOS/Windows built-in AV tools.
If you want security, use a secure OS. But it's not so easy, often decisions are made not to have the best technical solution, but to fulfill business agreements.
While often true, this entire comment is completely unrelated to this issue or article. The CrowdStrike failure was an issue of quality control, not susceptibility to an attack.
The position inside the kernel played a big part in making the incident worse and this part is mentioned in the article. Questioning the need for putting software there and what we could change on the technical side should be part of the discussion.
This is a subject I know nothing about. If it's known that the kernel has security flaws, why aren't they fixed? Or is it that this company is aware of specific security flaws that no one else is?
The hospitals, airlines, police, etc. that bought into CrowdStrike should have primary responsibility. I wouldn't want to live in a world where they can say "but McKinsey said it was okay" and wash their hands of this, while pinning all the blame on CS.
Yeah, why should software companies ever be held to some sort of standard? Let the buyer beware! Oh but don't call me a programmer, I'm totally a software engineer and don't you forget it.
They should pay. And then the CrowdStrike and their shareholders should pay to them. We really should just simply remove corporate veil. Make the shareholders at least pay monetarily if not in prison time.
CRWD is a member of the S&P 500. Are you truly suggesting that every holder of the S&P 500 should be put in prison? Realistically, this means that nearly every American and a very large number of international average workers just trying to save for their retirement would have to go to jail. Sounds like a good way to inflate those private prison stocks.
See, that's the way the system has been crafted. Letting theives/incompetents get away with their mistakes/crimes is tightly coupled to people getting money for retirement.
Who's gonna rock that boat?
People are now literally invested in in a system that tolerates, if not ofttimes rewards, bad behavior from companies all the way from small things right up to "deliberately poisoning/cooking the whole ecosystem to make a buck"
I am a shareholder in dozens of companies. I have zero oversight on how things are managed. The only thing I can do is sell my paltry amount of shares if I think things are not going well there.
Shareholders would be punished in this case in the sense that crippling fines and reparations that Crowdstrike should definitely be liable to pay would decimate the stock value.
And if criminal charges were to be pressed, then the people actually in charge of the company should be on the hook. For example, if someone dies because CS was horribly negligent in their duties, then that should be a possibility.
Agree with the sentiment of not hiding behind vendors for issues, but in this case there were (presumably) major procedure failures at CrowdStrike which lead to this - I believe they are 100% at fault in this case.
I don’t get it; in most professions, if you deliver crap, you are liable. Why is this different? How is the onus on the buyer who got told this product protects them and then the product actually screws them? I think CS should pay damages and go bankrupt, so others might think twice in the future.
I wouldn’t want to live in a world where people can get away with mistakes like this: as far as I have seen, if one person in CS had actually tried this file on their local windows machine, it would’ve crashed. That’s not a small thing is it? That’s a Boeing like culture which should be terminated today.
It is one thing if these updates were in hands of the IT of the companies, but that this company just could automatically push them over is just weird.
I accept shared responsibility when there is some agreed deployment and such scenarios, but here? Nope it is on CrowdStrike...
If I hand over my car keys to the same cleaning companies every month and all is fine always, except this last time, the cleaner cuts the break lines while cleaning the under side and I die in a crash.
That is a nonsensical comparison. Remote software updates and administration are common professional services that consumers of software cannot execute themselves.
The correct comparison would be handing your house keys to a licensed insect exterminator and the guy accidentally poisoning your family. You of course are not liable if a professional supplying you with a service messes up.
An other view. Tesla does over-the-air updated you don't verify. It disables break, you drive car and kill someone due to this. Do you share responsibility? What if this happens while driving?
It was a virus definition update, not a software update. CrowdStrike will obviously update their virus definition deployment pre-flight check to prevent a situation like this from happening again, but a staged roll-out is out of the question. What do you want them to do? Expose everyone to new known viruses for longer?
> a staged roll-out is out of the question. What do you want them to do? Expose everyone to new known viruses for longer?
I think trading, say, small x (15 minutes, an hour?) time, to stage rather than blast might be an acceptable tradeoff given the visible consequences of not doing so.
Because critical infrastructure doesn't get the convenience of shifting the blame. They must account for these risks in advance. Do you know what happens when IT breaks down in a bank? They've got two or more printers that print all new transactions. Database corruption? No problem, we've got the transactions right there. Printer breaks down? No problem, we've got another one printing the transactions right there. Hospitals don't get the luxury of shifting the cause of death from them failing to properly account for risks to a piece of software.
I think they need to do both. But then again, I don't think most of the IT staff in these places are very capable. I have large hospitals as clients and everything is fairly terrible there (it is how I make most of my money; I solve these type of issues for companies when shit hits the fan and hospitals are kind of the worst, but banks or gov orgs are also not great). They don't understand any of this really so it's not weird that they buy stuff that's recommended in CIO magazine. But yes, they cannot shifting cause of death indeed.
Where is the line drawn on how much is the responsibility of the "end users" (hospitals, etc).
Hypothetically, if there was a widespread 'bug' in... Intel CPUs that caused a similar issue, would you say it was the hospitals that should be held accountable, or Intel, or someone else? What about if the issue was in Windows itself - what should each hospital do differently that would have avoided this?
Let the consumers sue the hospitals and banks, and let those entities that directly interface with consumers sue Intel in return. Consumers need an interface to recover damages. This will provide the motivation not to keep around shit vendors and entertain shit contracts that basically absolve the vendors of all blame.
When the damages are in the billions we need to make sure that there are billions in liability to balance the scales. Or else there will always be a perverse incentives to not give a shit.
I think BOTH must be held liable. I’d share the blame 50/50. And if Microsoft should have checked the CS sw because msft signed it (I’m not clear on if that happened), then I’d generously make msft also 10-15% responsible.
Hospitals bought into a contract where the terms basically said "don't rely on us for anything important, especially not anything that impacts human life". Patients were certainly medically impacted, perhaps up to the point of early death.
Whose incompetence is that? If a hospital pays for an energy redundancy solution where the vendor basically communicates "please don't use us for anything important", and then a power outage costs lives, are we seriously saying that the hospital shouldn't have to pay big time?
I'm not saying that CS shouldn't be liable. I'm saying that when consumers are looking for someone to sue, and regulators are looking for someone to punish, those who directly interface with the consumer should be in the direct line of fire. Hospitals in turn should be able to sue CS.
We certainly should not prefer that the blame goes straight to the engineer as then the consumer would basically have nobody to sue. That's a world of even less accountability. Those who are most capable of taking responsibility should take responsibility, and not merely those who are most blameworthy from first principles of local causality.
There could be billions of dollars of damage here. There must be billions in liability as well to balance the economic scales. This is way past the single individual who pulled the trigger.
> The hospitals, airlines, police, etc. that bought into CrowdStrike should have primary responsibility
This is ridiculous.
None of these business can (or should) handle all of their infosec internally. They must use vendors. And because they don't have infosec expertise, they have no way to verify that a vendor has good quality control.
You're essentially arguing that a hospital should also be responsible if people die because they bought the wrong MRI machine brand or one of their trucking suppliers had a crash and couldn't bring them enough of a certain drug.
There is no universe where vendors shouldn't be responsible for failing to deliver a safety-critical service that they 100% guaranteed they could deliver. There was nothing in the CrowdStrike contract that said, "Updates may inadvertently disable all systems that receive them," because that would mean no one should ever use it.
Hospitals shouldn’t use vendors like this. They need to be able to audit and understand what they’re doing and not allow. Hospitals have CIOs and CMIOs and CHIOs to make these kinds of decisions.
Computer security is clearly not the core competency of hospitals, nor should it need to be. I'd argue, putting Internet connected systems running MS Windows in the critical path was already setting them up for failure.
I would say every company of a certain size or criticality should have a proper IT team. It does not even need to be large, but they need to have the expertise and authority to veto certain decisions. I often see vendor deals being made solely for the discount and other marketing crap, like getting another piece of service free as a bonus. IT then gets silenced because number goes up. I have seen situations where leaderhsip keeps asking IT/Engineering the same question "But why can't we just do this?" in different ways and dissecting the answers, asking for reports, and proof, and what not, so that IT is forced to make a bulletproof case against it. It then appears that IT has a personal thing against the vendor. So IT gives up. System is up. Often, part of the problem is the vendor buying lunch to key decision makers and being pals.
I am not familiar with how CS is or can be administered.
As a customer you should not have to assume the software your supplier vends is faulty to the extent of this incident.
It's reasonable for a customer to assume the software is tested according to industry best practices.
CS acted grossly negligent here, and they deserve the majority of the blame.
I agree that some blame also resides with customers, there must be disaster recovery procedures in place to allow them to function with minimal downtime in the case of emergency services.
I think the problem with DR in this case is you will need to account for every eventuality of every line of code in every sw product you use. Because say, how do you handle the case that your fleet BSODd? Usually you'd rebuild them using your bitlocker keys.
But this incident was so deep, that the SPOF here was using Windows. So now, your DR plan needs to account for some mandatory percentage of your OSs not being Windows, and your IT staff being maybe Linux experts. Cool.
But can you predict that you need to store your bitlocker keys in both platforms? And can you even do that or is it one of those things where bitlocker storage has to be on windows bc of lockin?
consultants spouting industry-standard slop, which means now everyone is on the same platform -- which we've just down doesn't really cover their butts.
and to be fair, having some sort of tool like that isn't necessarily a bad thing, the problem is when the tool gets pushed without testing and breaks the internet.
I am not sure why nobody talks about the fundamental flaw in the design of the operating systems. Monolithic kernels were chosen at a time when CPU context switching was a serious penalty on the performace.
So IMHO isn't it time to rethink this idea of putting every single shit into the kernel space?
In other words Linus won then, and its now time to think about Tanenbaum too.[0]
There are solutions in monolithic kernels for writing safe kernel space code now too. eBPF has been a thing for a while now (https://ebpf.io/what-is-ebpf/) and while it might not be as safe as microkernel it's available now and it works on both Windows and Linux.
I think a big question is why are things like CrowdStrike still written in raw C++ Kernel code? Is it the limitations of eBPF? Is it stagnations of the tech stack at these companies?
Thanks for the pointer. I will give it a read later on. If I understood correctly, eBPF is a guard rail for writing relatively bug free code. But it will still run in the processor[0] ring 0 which essentially is the same level as the kernel itself.
I went to a car rental company the other day. I looked across the counter.
“Is that a dot-matrix printer?” I asked.
“Yep. This system here—” and she tapped the monitor “— runs DOS.”
“Any downtime on Friday?”
“Nope. Solid as a rock.”
And I wonder do we really need all this complexity to run our every day stuff? I mean, yeah if you're running Photoshop or something, but basically moving numbers and demographics through a system, what the heck do you really need an ad-riddled privacy-invading behemoth like Microsoft Windows for?
Blaming one or the other is a waste of resources. Security is a shared responsibility.
Vendors must provide secure products and transparent communication, while customers need to make informed decisions and properly maintain the security measures provided.
In this case though, the vendor needs to clearly explain what happened.
As for the customers, they now need to analyze if they need to continue using the said vendor.
On a final note : never underestimate the power of mistakes - plain and simple. They exist and happen most usually.
CS made a big blunder when they rolled out an updated all at one time. Why would anyone who knows the slightest thing about risk do that? Mistakes can happen - I get that, but really all at once without knowing if you have an issue is criminal.
132 comments
[ 2.0 ms ] story [ 217 ms ] thread> We hope he's not flying Delta to Washington DC, as the US airline has canceled more than 5,000 flights since Friday as a result of CrowdStrike's update file crashing Windows systems.
I guarantee they all had BC/DR plans.
I honestly believe the whole thing in general has net negative ROI just because it never works.
What's absurd about it? It's absurd that I want the software operating my MRI and providing doctors with necessary information during surgeries to be different from what accountants using Excel and video game players use?
What's absurd is congress hoping to solve the fundamental problem by asking the CrowdStrike CEO to come in for a one on one. Even if we took this process at face value and assumed they were earnest it would be laughable.
Meanwhile.. the government is the largest healthcare payer in the world. For not particularly great reasons. This _is_ one of the downstream impacts of that reality.
This has nothing to do with there being a healthy market of operating systems to choose from.
> I’ve seen the fragmentation in edge devices
How many versions and patch levels of Windows exist? How many are currently deployed? How many must stay on the current version because of fragmentation?
> You know how many bugs you have with one stack?
The scenario that started this discussion is a single bug in a single piece of software impacting a large number of businesses, some of which are safety and life critical, all of which are on a single OS.
It's not a question of "how many bugs," because you can't fully eliminate them in any system which is actually updated _ever_, it's a question of "how many people does this single bug impact?"
Will this require more time and labor? Yes, of course. Does that make it "absurd" to consider? No, that's pure hyperbole.
If the website fails, that costs a lot of money, but nobody is in danger. If dispatch systems fail that has a safety risk to flight ops.
While you may not be happy that your flight gets cancelled. For airline IT there is a cost vs benefit balance between the probability of flight cancellations vs the cost of having extra systems. (And in this case extra systems would probably have been affected by the same security platform, because not having that is another risk of cyber attack to consider)
That doesn't apply to flight ops, where each operator needs to prove to a very high level that safety is always maintained. Maximum probability for catastrophic failure is 10E-9 and for hazardous is 10E-7. No commercial considerations considered, have to make those numbers to be allowed to operate.
Then again with systems working Boeing failed that one...
Except for massive disruptions to tons of people, businesses, and economies worldwide, with impacts rippling out for days after services have been restored?
1. https://www.npr.org/2024/07/23/nx-s1-5049792/deltas-airlines...
Should hospitals plan on not being able to provide medical care and not able to transfer their patients elsewhere, or divert the inflow of new patients elsewhere, because every other hospital is down too, there is no one left to provide care, and 911 is shot anyway? Maybe. Right after planning for a nuclear explosion over the city.
What happened was pretty much an "act of God"-level crisis, except caused by a de-facto infrastructure provider with way more destructive power than they're equipped to handle. IMO, not only CrowdStrike should be liable for the damage and lives lost because of the outage, this incident should be a prompt to rethink the entire idea of endpoint security.
--
[0] - I think it should be, but then I think the entire business with endpoint protection is bullshit.
No, but cyberattack takes out all computer infrastructure is a high impact, low risk that critical infrastructure providers must account for and would've prevented any impact from CrowdStrike's blunder. They obviously didn't do that.
Yes.
That decision maker was promoted and since left for a new leadership position.
Because we wont let you have your pet dev box naked, sorry.
Usually both are out of sight and out of mind. Curious what problems you’re having
Also, you probably don’t want to be responsible for causing your company to get ransomwared or data breached, which is the primary reason you won’t get any sympathy or special treatment.
I thought I was taking crazy pills when I read that and started wondering where the world was headed. Im glad atleast the CEOs are expected to give answers.
It's scary that people are taking these hallucinations as some sort of reasoned fact, or important opinion.
E.g., How many life-threatening emergencies do you think happen per hour across the world? How much extra time did it take to reach the average injured person over the outage period? How was the base rate of iatrogenic death affected by losing access to all electronic records for inpatients?
If you come up with a fermi estimate more than two orders of magnitude lower than 1,000, I will be suspicious of your assumptions.
Are some random gang shootings that killed 10 people worse than JFK's killing?
Don't be facetious.
The US is very divided currently, I agree, but I don't think any of us know what would have happened had Trump been assassinated, and anyone claiming otherwise is either too confident in their theorycrafting or trying to sell you something.
Also, you can't have a civil war as long as the military in the US is what it currently is, i.e subordinate to civilian powers. You can have riots, protests, sure, but i don't see any path for a civil war that does not somehow involve part of the military, in any country at any time, and especially now that armies are professionals.
Not to say it’s certain or even likely, but it’s definitely plausible. You are much more confident than I am that the military and police would remain 100% loyal to Biden during a protracted period of civil unrest where one side believes two general elections in a row were stolen from them and their leader assassinated.
If you are working for public institutions, then it probably matters immensely. But if you answer only to shareholders, then its more theatre.
Reform of companies like BlackRock is sorely needed, the way they operate is detrimental to the actual owners of equity, and society at large.
Furthermore, most of the money managed by BlackRock is not managed by them through direct choice of the equity holders, so even if the equity holders were not happy with BlackRock it would be very difficult for them to move their money elsewhere.
And then finally, the shares of BlackRock itself are also held on behalf of others by institutions like BlackRock, and some of it even by BlackRock itself. So they vote with the voting rights of others as to how they think they should be run.
The professional managerial class generally does not care about profits, as they exceed at finding ways of enriching themselves without having to actually deliver anything. They get rich while the average Joe's pension underperforms. They love things like DEI and ESG, as that allows them to define their own parameters for success in ways which in no way benefit the people they have a fiduciary duty to or society.
Thanks for your informative response
Technically, if your investment does well, they have more assets under managemnt, and they earn a larger fee, but in practice they can just increase their rate because the people using them aren't shopping around anyway, and there will always be more money invested through them.
In reality, most people are paying BlackRock without ever knowing that they are paying them, and then in addition to paying BlackRock in money they get the benefit of your voting rights, which they can then do with as they see fit. They are power brokers who are paid by the people whose power they are brokering.
- 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993
It seems to me that there's a need for the Crowdstrike CEO to take responsibility, e.g. for lax testing and for not doing canary rollouts.
And it seems to me that there's a place for the [airline|bank|hospital] CEOs and CIOs to consider implementing canary 'roll-ins' where new software is tried on a small scale before being installed everywhere.
/s
Who's gonna rock that boat?
People are now literally invested in in a system that tolerates, if not ofttimes rewards, bad behavior from companies all the way from small things right up to "deliberately poisoning/cooking the whole ecosystem to make a buck"
I am a shareholder in dozens of companies. I have zero oversight on how things are managed. The only thing I can do is sell my paltry amount of shares if I think things are not going well there.
Shareholders would be punished in this case in the sense that crippling fines and reparations that Crowdstrike should definitely be liable to pay would decimate the stock value.
And if criminal charges were to be pressed, then the people actually in charge of the company should be on the hook. For example, if someone dies because CS was horribly negligent in their duties, then that should be a possibility.
I wouldn’t want to live in a world where people can get away with mistakes like this: as far as I have seen, if one person in CS had actually tried this file on their local windows machine, it would’ve crashed. That’s not a small thing is it? That’s a Boeing like culture which should be terminated today.
I accept shared responsibility when there is some agreed deployment and such scenarios, but here? Nope it is on CrowdStrike...
If I give root to a third party, then I’m liable for everything they do with that root.
The correct comparison would be handing your house keys to a licensed insect exterminator and the guy accidentally poisoning your family. You of course are not liable if a professional supplying you with a service messes up.
It’s like paying a licensed insect exterminator that uses random chemicals. They tell you this and you hand over the keys anyways.
I don't consider them capable of preventing something like this from happening again.
I think trading, say, small x (15 minutes, an hour?) time, to stage rather than blast might be an acceptable tradeoff given the visible consequences of not doing so.
Our essential services should be built on a more secure OS, if windows isn’t suitable.
So it sounds like you agree that Crowdstrike should go bankrupt.
> Our essential services should be built on a more secure OS, if windows isn’t suitable
Oh wait you blame Windows for this?
- https://www.crowdstrike.com/blog/crowdstrike-brings-xdr-to-z...
- https://access.redhat.com/solutions/7068083
Or maybe not to boot . . .
Because critical infrastructure doesn't get the convenience of shifting the blame. They must account for these risks in advance. Do you know what happens when IT breaks down in a bank? They've got two or more printers that print all new transactions. Database corruption? No problem, we've got the transactions right there. Printer breaks down? No problem, we've got another one printing the transactions right there. Hospitals don't get the luxury of shifting the cause of death from them failing to properly account for risks to a piece of software.
Hypothetically, if there was a widespread 'bug' in... Intel CPUs that caused a similar issue, would you say it was the hospitals that should be held accountable, or Intel, or someone else? What about if the issue was in Windows itself - what should each hospital do differently that would have avoided this?
When the damages are in the billions we need to make sure that there are billions in liability to balance the scales. Or else there will always be a perverse incentives to not give a shit.
Whose incompetence is that? If a hospital pays for an energy redundancy solution where the vendor basically communicates "please don't use us for anything important", and then a power outage costs lives, are we seriously saying that the hospital shouldn't have to pay big time?
I'm not saying that CS shouldn't be liable. I'm saying that when consumers are looking for someone to sue, and regulators are looking for someone to punish, those who directly interface with the consumer should be in the direct line of fire. Hospitals in turn should be able to sue CS.
We certainly should not prefer that the blame goes straight to the engineer as then the consumer would basically have nobody to sue. That's a world of even less accountability. Those who are most capable of taking responsibility should take responsibility, and not merely those who are most blameworthy from first principles of local causality.
There could be billions of dollars of damage here. There must be billions in liability as well to balance the economic scales. This is way past the single individual who pulled the trigger.
This is ridiculous.
None of these business can (or should) handle all of their infosec internally. They must use vendors. And because they don't have infosec expertise, they have no way to verify that a vendor has good quality control.
You're essentially arguing that a hospital should also be responsible if people die because they bought the wrong MRI machine brand or one of their trucking suppliers had a crash and couldn't bring them enough of a certain drug.
There is no universe where vendors shouldn't be responsible for failing to deliver a safety-critical service that they 100% guaranteed they could deliver. There was nothing in the CrowdStrike contract that said, "Updates may inadvertently disable all systems that receive them," because that would mean no one should ever use it.
As a customer you should not have to assume the software your supplier vends is faulty to the extent of this incident.
It's reasonable for a customer to assume the software is tested according to industry best practices.
CS acted grossly negligent here, and they deserve the majority of the blame.
I agree that some blame also resides with customers, there must be disaster recovery procedures in place to allow them to function with minimal downtime in the case of emergency services.
But this incident was so deep, that the SPOF here was using Windows. So now, your DR plan needs to account for some mandatory percentage of your OSs not being Windows, and your IT staff being maybe Linux experts. Cool.
But can you predict that you need to store your bitlocker keys in both platforms? And can you even do that or is it one of those things where bitlocker storage has to be on windows bc of lockin?
and to be fair, having some sort of tool like that isn't necessarily a bad thing, the problem is when the tool gets pushed without testing and breaks the internet.
So IMHO isn't it time to rethink this idea of putting every single shit into the kernel space?
In other words Linus won then, and its now time to think about Tanenbaum too.[0]
[0] https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...
I wonder if it would be possible to make an OS that uses an inter-process communication primitive that works like io_uring to mitigate this overhead.
Why that is the case when we have faster CPU, RAM, and SSDs etc.
I think a big question is why are things like CrowdStrike still written in raw C++ Kernel code? Is it the limitations of eBPF? Is it stagnations of the tech stack at these companies?
[0] https://en.wikipedia.org/wiki/Protection_ring
https://github.com/microsoft/ebpf-for-windows/
“Is that a dot-matrix printer?” I asked.
“Yep. This system here—” and she tapped the monitor “— runs DOS.”
“Any downtime on Friday?”
“Nope. Solid as a rock.”
And I wonder do we really need all this complexity to run our every day stuff? I mean, yeah if you're running Photoshop or something, but basically moving numbers and demographics through a system, what the heck do you really need an ad-riddled privacy-invading behemoth like Microsoft Windows for?
There should never be implicit trust on critical infrastructure, period.
Vendors must provide secure products and transparent communication, while customers need to make informed decisions and properly maintain the security measures provided.
In this case though, the vendor needs to clearly explain what happened.
As for the customers, they now need to analyze if they need to continue using the said vendor.
On a final note : never underestimate the power of mistakes - plain and simple. They exist and happen most usually.
Seriously, who would run an operating system that requires trusting code by giving it ambient authority? We've known that was nuts since the 1970s.