It depends on what you mean by "open and flexible". Obviously, Windows NT isn't open source (though it has been heavily reverse engineered, and Microsoft publishes kernel symbols, which is more open than most proprietary software). But Windows does have a stable driver API and a license that permits publishing proprietary kernel-mode drivers, neither of which Linux has. Linux's position is that if you want to develop proprietary drivers, either develop them in user space or sandbox them with eBPF. This is arguably less flexible than Windows's policies, though the CrowdStrike incident certainly highlights the reliability benefits of Linux's approach.
That's actually the core of the issue. Windows gives users the capacity to run programs with Kernel access. The affected companies gave Crowdstrike kernel access. Crowdstrike crashed in Kernel, taking out the machines.
This kind of issue would not be possible in Linux/Macos unless the user configures them to allow the same kind of direct kernel access. At least for macos this is off by default. Not certain about linux but I think the default is more locked down than Windows.
Microsoft attempted to lock kernel access in 2009, the EU complained, so Microsoft kept it open in order to allow "competition in security vendors", and this is the result.
>Microsoft attempted to lock kernel access in 2009, the EU complained
And the EU complained only because the likes of McAfee complained. Microsoft here is "dammed if you do dammed if you don't" since so many of these security vendors make a living by being able to act like malware themselves in order to sell security. If Microsoft would lock them out of the kernel they'd all scream "anti-competitive" to the regulators.
The true story is that I bet some major divisions of Crowdstrike are ran by non technical people that got there through non meritocratic means. Theres generally been no repercussions for their underperformance, much like boeing. Crowdstrike business is built on relationships, not technical supremacy. And bada bing bada boom, we have a complete failure of basic technical competency (no rigourous role out process).
I used to work on Control Flow Integrity (CFI/XFI) research at places like MSR Silicon Valley and VMware, as far back as 2006. Back then, sandboxing a kernel module like ramdisk.sys was doable with a lot of binary rewriting magic, and later with custom LLVM passes, but nowadays it should be a simple matter of compiling the code with clang and the appropriate flags, to completely rule out this type of memory safety error, turning a BSOD into a polite log message and disabling the faulty driver.
... oh. and the article by Brendan Gregg in the HN link above had the telling phrase:
"Once Microsoft's eBPF support for Windows becomes production-ready"
from what I understand, CrowdStrike has essentially put a Turing-complete interpreter for their scripting language into the kernel. I doubt you can do much when something is that general purpose.
Do you have more information on that? Hadn't read anything about the CS kernel module running arbitrary code. Was it a factor in the crash?
'course, Microsoft also put turing complete scripting in ring 0 years ago for performance reasons (TTFs - XML/HTML parsing and GUI rendering too - to beat other OSes apparently) and that certainly did lead to exploited vulnerabilities...
Lua has been used in Linux kernel modules[1][2]. At least for the ZFS case I know they were satisfied with the ability to limit what the Lua scripts could do to avoid issues.
I suppose I was expecting something more authoritative here. They confirm that there was an attempted read-out-of-bounds, as CrowdStrike said, but that's not really new information at this point. I suppose we'll need to wait for more detailed analysis from CrowdStrike at some point.
This post explains why security software has historically run in kernel-mode, and really seems to be pushing new technology that Microsoft has that would push security vendors into user-mode (with APIs that attempt to assist with many of the reasons why they have historically used kernel-mode).
Crowdstrike already runs in user-mode on both Mac and Linux (from what I can tell), and it seems like running in user-mode on Windows would significantly lessen the risk of catastrophic failures like a blue-screen-of-death. I know the bulk of the failures here belong to CrowdStrike, but I can't help but think about the fact that Apple kicked security vendors out of kernel-mode a ways back, and that if Windows had done similarly, an issue like this probably wouldn't have been possible. By even offering kernel-mode options to external vendors, I believe Microsoft is creating risk for themselves.
> I can't help but think about the fact that Apple kicked security vendors out of kernel-mode a ways back, and that if Windows had done similarly, an issue like this probably wouldn't have been possible
Like others already said, Microsoft already tried to do that with PatchGuard in 2006 with the launch of Windows Vista and the likes of Symantec and McAfee complained to the EU about this would harm the sales of their products, so the EU told Microsoft to not do it in 2009[1].
Apple has the luxury of a small market share on the desktop PC space to not attract the attention of the regulators, plus a user base that's used to Apple constantly rewriting the OS, deprecating APIs, switching CPU architectures, etc. without giving a fuck about breaking backwards compatibility or cutting off developers access to OS features their products use and getting away with it, luxuries that Microsoft doesn't have.
IMHO, sticking with Window's default security and not using third party anit-malware has made Windows vastly more secure and rulabile than it was in the days when you'd be looking on installing the likes of Symantec or McAfee for your "protection" which ended up acting like malware after a while throwing dark patterns at you to milk more subsection fees, so as much as it hurts their sales, it's important for the regulators to understand that security is far more important than the regulations they put on Windows for Internet Explorer and Media Player and just like Apple's apps-store, it's sometimes better to let the original product maker handle security and not leave the product open at all points just so some of these bandits can make a living selling security for it. It's like foxes complaining to regulators how chicken wire is a threat to their existence.
Wasn’t the whole regulatory argument that Microsoft was using kernel mode in their security software, while trying to relegate third party security software to user land? In that case, regulators stepped in and made Microsoft open up kernel mode to level the playing field.
I don’t understand this world well enough to describe it well, but my understanding is that the security features in the kernel are different in purpose from the security features of something like an antivirus.
Apple provides the kernel implementation and the API. They don't offer a real competing alternative for the user mode side of the equation. They leave that to the ecosystem. It's a fair trade-off. They can focus on keeping the kernel secure and stable, and the ecosystem can work on all the different ways you can implement anti-malware policy and integrate it into a larger system of endpoint security.
MS offers a full competitive product, and they allowed that product to use features and capabilities that wouldn't be available to others. At their scale, you can't do that without drawing scrutiny from regulators. They could have at least kicked their own product out to userland, or just stopped offering one and focused on kernel stability the way Apple has.
Since this is all spitballing, perhaps they even suggested that the EU, and the EU said "no, it must be kernel access" and capitulated because this served their own interests while also letting them blame the EU for the lack of kernel stability.
If you are in the market to sell EDR tools and you can’t find a way to provide value beyond what XProtect offers, you probably shouldn’t be in the market.
The difference between XProtect and Defender, is that XProtect utilizes the same kernel interface that Apple provides to other EDR solutions, whereas Microsoft was / is only willing to implement Defender using an in-kernel solution.
As a result, it would be anti-competitive for Microsoft to deny that level of access to third party EDR solutions.
I do wish that the EU would offer a "grace period" for "platform owners" or "gatekeepers" that would allow them to use non-public APIs for their first-party solutions while they stabilize and work out the public API offering. I think there should be some requirements for them to "earn" that grace period, such as allowing interested third-parties to get early access and provide feedback on those APIs. Regardless, it takes time to stabilize those APIs, and the current EU approach of requiring all new APIs to be fully public on day one just isn't workable.
Your operating system by definition has to have complete control over things your operating system can do. Do you want third parties to have access to for instance the Secure Enclave in iPhones? The same permissions the Settings app has?
Lots of allegations here. Can you share examples with sources of other operating systems following practices which you mention here? I presume Mac allows the same level of access for CRWD through user mode access only and that’s the only way they do it too. Same goes for Linux.
I genuinely want to understand this - how everyone else got it right and this entity got it wrong.
> Can you share examples with sources of other operating systems following practices which you mention here?
Well, both dominant mobile operating systems already enforce a very strict security model. Too strict if you ask me because on neither you as the person actually owning the damn thing can become root without anything in the system throwing wrenches in your way, but hey.
The result is, most people don't even need a virus scanner, and the blast radius of a bug in an application is usually limited to whatever data that application can access. It's been ages since I heard about the last major mobile OS malware in the wild that didn't involve one of the world's secret services or their civilian suppliers - and that way, one can argue, all the security theatre actually works: it's driven the price of successful attacks so high that only nation states can afford it, and even these have to restrain themselves to only use these exploits against targets of extremely high value.
The reason mobile OS don’t have such a problem is because of the ‘walled-garden’ for App Store. Essentially, every binary being run on all Apple and likely most Android has been ‘blessed’ by the manufacturer App Store.
Is it possible to rootkit phones? Yes; does a typical user do it - heck no.
The app store does some pre-checks, yes. But it can’t catch all the security vulnerabilities that might be unknown. It’s only a small piece of the pie. Largely it exists to make sure apps work, don’t have malicious UX patterns and don’t use private APIs.
iOS (and to a lesser degree, macOS) use notarization to be able to pull apps found of being risky as needed.
Additionally they implement much stricter levels of sandboxing so that even if something slips by all detection, it’s heavily limited in what it can do without a users explicit permission or a catastrophic vulnerability.
Android supports third party app stores and sideloading out of the box without rooting. There's a separate apk scanning feature that works with sideloading.
(There's a big android user population without access to the Android app store at all)
Microsoft sells endpoint security products and it would be unfair if third party solutions couldn't leverage the same APIs, it makes a lot of sense that a regulator steps in.
I'm not aware of Apple selling security products or competing with third party security products.
Cars are sold with integrated radios and players. But at the same time
there were independent companies selling car radios back in times when
they were exchangable. Now external players are gone, everything is
integrated, and the market for custom car players is dead. And nobody
cares! One could say that car manufacturers don't offer the same API for
car player companies.
I think that Microsoft is the king of their system, and can do whatever
they please. If that doesn't sound practical or trustworthy for a
company, then maybe the company just shouldn't release the product on
their system. Use a different platform. Because if you release a product
on their platform, then you're saying that you're okay with their rules.
I work in a heavily regulated industry (healthcare) and I can tell you that if anti-virus products weren't required to pass audits we wouldn't be using them. I'm not super familiar with Windows built-in security anymore but macOS (our platform of choice) is pretty secure without any additional products. In fact, I'm pretty sure that adding A/V "solutions" makes us more vulnerable, not less.
This seems to be only partially true when I read into it. The EU said that Microsoft would need to move their security tools into user-space (or at least to use the same APIs as are available in user-space). If they did that (like Apple has done), they could kick everyone out of kernel-space if they wanted.
This is fascinating, thank you for the info! If I am understanding, it would have then been difficult/impossible for CrowdStrike to create a user-mode only sensor without these equivalent APIs.
So I guess I'm not sure I see validity in the claims of those blaming the EU here. It seems as though the EU would have allowed Microsoft to kick users out of kernel-space if they had APIs that allowed making security products in user-space. Like Linux/Mac already appear to have.
I don't think they would have had to provide those APIs in the EU, so long as their own security products were "kicked out" as well. That's kind of complicated to achieve in a permanent and provable way. Though, windows has had support for eBPF for about two years now.
I agree. Microsoft's core competency has traditionally been backwards compatibility, but if each security vendor can tamper with windows at the deepest level and is allowed to continue explore all of the ways that they can leverage that... What you end up with is a fleet of different windowses, each diverging further with time. It dilutes the benefits brought by investment into the stability of the system because whatever fights are won in one fragment must be refought in others before you can have confidence in the stability of all fragments.
> Crowdstrike already runs in user-mode on both Mac and Linux (from what I can tell),
Crowdstrike provides a Linux kernel module, and expects users to manually install an extra Secure Boot key for it, as part of their corporate laptop setup procedure.
This has always seemed inadvisable to me, but checkbox checkers gotta check checkboxes I guess.
A kernel-space watchdog (that checks integrity of the image) would be much easier than a filter that updates from the internet.
Sure, the whole thing is definitely a hard problem, but CS fucking up even the most basic QA **and** error handling ... it just shows how ridiculous their whole claim to having super fancy technology is.
Microsoft's statement implies that people choose Windows because of it's security and availability. Whereas most people end up with Windows because the software they want to run only operates on that single platform.
The security and availability, to the extent they even exist, are clearly not part of the market's decision making process.
The critical infrastructure that people actually cared about (ATC for example) had all the choice in the world. So did people designing bespoke POS systems. To rephrase the old IBM trope, "nobody got fired for choosing Windows".
> critical infrastructure that people actually cared about
Did hospitals have a choice in which OS their MRI machine runs? Are those not "critical?" Or should we just not "actually care?"
> ATC for example
Were they impacted by this outage? Isn't the reason flights were canceled is because the _carriers_ systems are the ones that had issues?
> So did people designing bespoke POS systems.
Really? I would assume most of that is down to the hardware, like cash drawers, credit card readers and order printers, which is likely third party and proprietary, and is only available and supported on Windows. Do you have evidence otherwise?
> "nobody got fired for choosing Windows".
You've recognized the same outcome I have but have gone to great lengths in an attempt to obscure the reasons for it happening. Why?
Nope, they aren't ready to pay for another platform that is their choice . If customers paid for linux or mac support there is no shortage of developers ready to cater to that.
Unwilling to pay for multiple platforms is still a choice
Vast majority of the enterprise software was built bespoke or heavily customized for a customer for a long time and still is.
SaaS / product based multi-tenant software development becoming popular is not the only revolution in software recently we also had many improvements that massively reduced the cost of developing apps in the last 30 years. Multi platform support has become exceptionally cheap with browser as a platform and electron style frameworks for native apps.
The problem is they have enough stuff where that's not an option. A lot would lack support, and Microsoft doesn't charge enough to push them over that barrier.
Yes. Only Microsoft should be allowed to crash their operating system. Like back in the good old days when only MS could use their secret high-performance APIs.
They shouldn't. Microsoft should have APIs that enable security vendors to work in userspace.
The EU didn't say that Microsoft couldn't kick vendors out of the kernel, just that they couldn't do so without having the APIs available that would let security vendors operate outside the kernel.
Mac and Linux have such APIs, so CrowdStrike operates in user-mode on those platforms, so those platforms do not give security vendors the ability to crash the operating system.
But the blame wasn't misplaced before. People keep saying that macOS does things better by forcing third parties out of the kernel and instead offering APIs to do the same work in userspace. Microsoft tried to do exactly this for security software in Windows, but the EU didn't like that this change meant that any Microsoft-developed solutions would have an advantage over third party ones.
> Microsoft tried to do exactly this for security software in Windows
Using a monopoly in one industry to capture the market in another industry is what anti monopoly laws are meant to prevent.
Microsoft was prevented because they wanted to retain a commercial business in their security products having special access while locking out everyone else.
> We plan to work with the anti-malware ecosystem to take advantage of these integrated features to modernize their approach, helping to support and even increase security along with reliability.
> Providing safe rollout guidance, best practices, and technologies to make it safer to perform updates to security products.
> Reducing the need for kernel drivers to access important security data.
They are being as diplomatic as they can, but it's definitely a slap to CS. Read as "they don't know how to roll things out, they need guidance on basic QA practices, we'll happily teach them...". Then, they list a set of facilities running in user-mode to avoid needing to run as many things in kernel mode.
I would be interested what the water cooler discussion about CS was like inside Microsoft. Especially in teams needed to respond to customers about "Your windows OS is broken, our hospital patients are suffering...".
I must disagree with that take, your last quoted sentence is in response to all the supposed self-proclaimed experts asking "why does it need kernel access", the ones before that is to limit their own liability.
What I've heard from people in the industry is not this silly "oh no, crowdstrike is so incompetent" b.s. that is being spread on sites like HN and reddit but more of an empathic "it could have been us" sentiment. In this write up as well, Microsoft knows they have caused their share of outages, it is a technical write-up but in part, it is to cover their bases for government investigations and lawsuits that will arise from this incident.
And in part, they are also responsible for recovering from third-party driver errors and repeated boot failures caused by faulty drivers.
CrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments. The bug that caused all this was present in their kernel driver for a long time.
For being such a large cybersecurity player and deploying updates to 8.5 million devices, their quality control practices are embarrasingly lacking.
I think it is past incompetence, and on into negligence. Given the stories we have heard here about emergency service failures it is likely that people died. When people die due to negligence isn't that usually criminal?
Who is negligent though? Crowdstrike, or the emergency services that are using an OS that requires third party endpoint security right out of the box in order to be safely used, or the company that makes and sells that OS?
Crowdstrike, for negligently not rolling out updates gradually.
And emergency services, if they don't have robust fallback procedures/systems for when their IT system goes down. I mean it's totally fine if regular doctor's visits get postponed, but 911 should never go down just because their computers down. Just like aircraft have redundant systems, so too should 911.
(The company that makes and sells the OS -- I don't see any negligence there, in this case. If security software fundamentally requires running at the kernel level and Microsoft allows that, I don't see how Microsoft can be at fault.)
Yeah, I don’t see how one can blame Microsoft in this scenario. If you choose to run buggy kernel-level code, that’s on you, not the publisher of the kernel/OS. Especially when the code you’re running is a replacement for functionality already provided by the OS. It’s hard to argue that MS could be negligent for “not having a good enough AV/endpoint protection solution” or “allowing customers to run kernel-level code.”
https://www.techradar.com/news/google-chrome-not-working-mic... ,not an unusual practice and they were not first av company to cause outages. and again, it was not a software update, the buggy software was deployed after testing back in march. Details matter!
How about we let the lawyers figure out who had what liability, just like with the av/edr industry, we should know when the subject matter is outside our area of knowledge and expertise.
What isn't an unusual practice? A lack of canary deployments? For av definition files, where crashing the scanner doesn't bring down the kernel I can agree. For a file read by a kernel module/driver that is primarily used by security sensitive infrastructure, I really hope it is unusual, and if it isn't lets make an example.
If failing to do unit tests to get full coverage (line and branch) on software that gets loaded on security sensitive infrastructure isn't unusual then things have sure changed in the decade since I was involved. We had to get that coverage at the assembly level and then peer review the tests to make sure they were meaningful and correct. Every time the code changed. Code that failed to properly sanitize input files without panicking should not pass normal safety critical code pipelines.
I do wonder if windows already has some disclaimer about being unfit for such use cases, in which case yeah, lawyers will have some fun, but even with that sort of protection, CS is clearly marketing to critical infrastructure, so it has no moral defense whatever lawyers and the law says.
Did Microsoft do a staged or canary roll out with that? This is not a software update, if you're making such comments then you're speaking about something outside of your field of expertise.
Every company I've ever been at rolls out updates slowly. Rolling out a change to 8.5 million computers at the same time seems ridiculous. Even the most cash strapped start-ups with every incentive to cut corners tends to get staged roll-outs more or less right. It's crazy.
Beyond crazy. I even have a small app that never makes it to production before being rolled out to internal and open testing first. And, even then, it's slowly rolled out to a percentage at each stage before being fully deployed. One would think a major company with kernel level access would do this at minimum.
When I managed ~ 15 developer’s Arch Linux workstations, I found it very beneficial to be the canary, and then rollout to a couple of the more capable of troubleshooting devs, and then the rest. I can always fix my own box.
details are always relevant in a technical discussion. look at my other comments where i pointed out microsoft performing similar immediate av signature updates and causing chaos.
We can agree on that, but it is relevant because this isn't an unusual practice. Crowdstrike didn't ignore some pre-existing best practice. Lots of things need improving but facts and details matter when you talk about RCA. it isn't about blame but fixing the root cause.
Deploying an update to 8.5 million devices without doing any form of staged roll-outs, not even just updating a few machines in the office first, is absolutely bonkers. If that's not an unusual practice it bloody should be. I don't care if it's a software update or a config update.
again, this is why I was snarky in my earlier post, this was not a software update. they should have used canary deployments still but in many cases prior to this incident, it was not acceptable to wait even a few hours because it can make the difference between companies getting ransomwared/hacked, so they focused on making the actual code/driver that interprets the channel file updates robust enough to handle real-time updates. Even if other players were doing canary deployments with behavioral detection updates, they're not the market leader, crowdstrike is for a reason.
Everyone that worked in an operational incident response role has blocked some indicator like an ip address or a domain. you don't do gradual roll outs for those either, and i've seen people cause outages by skipping a check or making a mistake. this is similar in many ways to that except it was for a named pipe. This could probably have waited for a canary deployment, but in general the class of content that is being deployed would be deployed right away, I'd be surprised if their practice is considered "bad" by any measure. I've seen Microsoft also deploy email quarantine signatures and defender updates that caused large scale impacts.
Here is a link of what Microsoft did earlier this year:
If they had canary deployments, that wouldn't have happened. I had rules that were causing chaos because of that. Now imagine if defender had a bug that caused it to crash because of a signature update. The impact would be magnitudes greater than what you saw with Crowdstrike. It's really frustrating to see the lack of technical critical thinking and arm-chair experts acting like they know what they're talking about.
Let's say the driver was "robust enough" to handle a broken channel file. How would that look exactly? Say you're responsible for writing the code which loads a new channel file. These channel files are critical; without them, your security critical product doesn't know how to do its job. The channel file parser returns a parse error. How should the driver respond? Surely you're not going to just silently disable your security critical product if someone puts a bad channel file in there?
Delete the file or mark it as corrupt so that the parser doesn't keep trying to read it, and send some telemetry back to CS to indicate there is a problem with the one of the channel files. It doesn't seem very complicated at all. There are plenty of options in between "catastrophically crash the OS" and "silently disable the entire product".
> CrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments.
Their post-incident report [1] also stated that they intend to improve testing by "using testing types such as: local developer testing". One has to wonder what, if any, testing they were doing beforehand.
Unless their developers had room temperature IQs or were actual psychopaths, I really wonder how they even managed to find developers who had the nerves to deploy to the whole world all at once like that. If it were me I'd be scared shitless, covered in sweat and probably shaking too hard to even type. Were CrowdStrike developers too stupid to even realize the magnitude of what they were doing? Or did they have cooler nerves than an open-heart surgeon? It's shocking to me that they could have done this so casually.
Microsoft should be sued, for literally having blood on their hands. There was an easily mitigated design flaw in Windows that would have greatly blunted the impact.
If "it could have been them", then I would like to read such professionals write exactly about how to avoid having a global outage like this again, rather than "showing empathy" with a corporation. Or do we just leave it up to luck, and if "it happens to them too" in a month or year, oopsies? What about which practices could be improved?
It didn't read as particularly diplomatic to me. In particular, this paragraph..
> It is possible today for security tools to balance security and reliability. For example, security vendors can use minimal sensors that run in kernel mode for data collection and enforcement limiting exposure to availability issues. The remainder of the key product functionality includes managing updates, parsing content, and other operations can occur isolated within user mode where recoverability is possible.
...was about as close to tetchy as a post like this would ever get. Basically they are saying "there was no good reason at all why CrowdStrike had to put so much code inside the actual kernel." And with the benefit of hindsight, it's a strong point.
Their business is corporate spyware to surveil employees, ofcourse they'll use any tactic to make it work, that's the why. And their EULA states there is no liability for the company:
>Their business is corporate spyware to surveil employees
What?! Anything you do on your corporate provided laptop is always gonna be logged by IT for security in every large company everywhere, that's news to nobody, but your company doesn't care that you use your corpo laptop to book your vacation, IT has better things to do than narc on you for that.
If your boss wants to actually spy on you they don't need Crowdstrike, there's other SW dedicated for that depending on the laws in your jurisdiction but that' not what Crowdstrike is for.
If you want complete privacy from your employer, just use your personal machine for your private activities instead of your work laptop, why is this so hard?
Speak for yourself. There are still companies who don't treat their employees like idiots and actually trust them. Let's not normalise pervasive surveillance.
>There are still companies who don't treat their employees like idiots and actually trust them.
Yeah sure, but wow many of those are large non-tech companies?
You massively overestimate the tech competency of the average PC user if you think it's normal in most companies to not have security monitoring solutions in place or over the internat activity. In our latest phishing test IT did, several users fell for the trap, despite it being a tech company. There's always gonna be someone careless one day and companies want insurance policies against that.
Having such solutions in place doesn't mean the company doesn't trust you, it's more like that old Russian proverb, "trust but verify", and for ticking security compliance boxing as an insurance policy.
Everyone makes mistakes, it's only human. So more like, speak for yourself, if you think your internet activity at work isn't logged anywhere.
I think there’s an inflection point where the company has grow so big it becomes impossible to trust every individual employee.
It won’t be about distrusting anyone in specific either, but something will go wrong for which you need to be monitoring every PC to find out what is going wrong.
There are better tools for spying like Teramind and Aktivtrak. Crowdstrike would make a bad spying tool. I guess there is remote CMD? And you can like, see all installed programs.
But so can SCCM/Intune from MS or another RMM like Datto that IT uses to manage PCs...
Do you happen to have a list of that "loooong history" of botched (security) updates?
I can only find a couple of examples after googling, which a bit smaller than a "loooong history" you're talking about, so unless Microsoft is paying Google to delete results, maybe you're mistaken.
This is a company whose OS could not even be installed on a live network without getting rooted within a few minutes. Anybody who was paying attention knew that you didn't use any new Windows release until at least the first service pack had come out.
Granted that was a while back but painful memories die hard.
>This is a company whose OS could not even be installed on a live network without getting rooted within a few minutes.
That was WIndows XP 20 years ago. Please bring arguments about modern Window 11 security which is the current up to date product they're selling and supporting not scenarios that haven't happened in 20 years.
Recall actually is a brilliant idea, and I dreamed of something like it for a long time, and so did plenty people here. It's just not something you can trust a third-party business with, whether it's a fly-by-night startup or an international megacorporation known to be openly promiscuous with advertisers.
This is basically "take a screenshot every 30 seconds and compile it into a timelapse", but on steroids, and the same appeal, and arguments wrt. who gets to run it on whose machines, all apply.
Ignoring Windows Insider reports is bad. However, how many endpoints having issues (out of a billion+) is ‘acceptable’ after an update? We live in a news hype cycle so clearly even the one wrong failure will make it up somewhere.
However, without metrics that show BSoDs from patches (which MS will likely never share), it’s hard to see if things have improved or regressed. If they regressed, someone up in their leadership chain is hopefully following the constructive discussion here.
Ah, well, if only things of the past were useful today, I'd still have hair, and probably millions made form right investments, but unfortunately, it's what's happening today that actually matters.
How does that impact the present? If it's no longer as vulnerable today, why would I care about the past? The point is learning from mistakes and fixing them so that doesn't happen again.
If it doesn’t matter to you, why did you ask? Are you just trying to win an argument or are you being intellectually honest? Because you asked for proof of the long history someone claimed. You could have just said “the long history doesn’t matter because I only care about the current state”. That’s fine and valid, but don’t ask questions and then shift the goalposts if you don’t like the answers.
A "loooong history" needs to have a timespan of many years.
So yes it would start in the past, but it then has to continue for a long time.
Pointing out that a company was bad 20 years ago isn't enough. You need to show they were also bad 15 years ago, and 10 years ago, and 5 and/or 25 years ago.
So complaining that the only evidence was so far in the past is valid. The original goalposts were not reached. (Well, someone in another part of the thread eventually listed every google result for a windows update making anything crash, but that doesn't really establish that microsoft is "botching" updates at a level significantly above background noise, which I think was the original intent.)
Was my implication of "every 5 years" not clear? But I already mentioned those links, they're pretty weak. I'm not calling an update that for a few people makes a handful of games crash "botched", when the original implication was quite juicy botching.
Also, if we're actually getting into this, the XP gripe had nothing to do with updates. That's moving the goalposts half a mile in the other direction.
Fair enough, but if those examples are irelevant to modern times, what's the point of bringing them up? If we want to keep the discussion relevant to modern context then let's discuss modern history, not obsolete news from 20 years ago.
That's a bit disingenuous, though. That was, as 'Rinzler89 points out, some 20 years ago. Back then, any Linux distro would've definitely been much safer option, because after installing you couldn't even connect it to the network, because it had no support for your cable modem or wireless card, and that's assuming you didn't fuck up your MBR with LiLo for the 20th time. Ask me how I know.
Both OS families have changed much since that time.
On please, if it were that tough then teenage me never would have managed it. 20 years ago, e.g. 2004 (I first installed it in 2001), installing Linux and getting networked was already user friendly. The only hitch I ever had was figuring out ndiswrapper, but my ethernet cards all worked "out of the box" and installers handled the bootloader without users even having to know what a bootloader was. It's not like 20 years ago was the 90s or something, and the dark days of Windows lasted well into the 00s.
In 2002 I wasn't yet even out of middle school when I had a Linux distro running all key hardware components "just working". At that time at my school we were taught how to search the web, so I searched the web and looked up what hardware worked. Very simple. All I had to pitch to my parents was, "this system shares its code and encourages me to study it and learn code", which made clear to them what I was asking for wasn't just another video game console. Soon after I had a refurb laptop (fortunately not x86) and a curated WiFi card that ran Linux (and soon after, BSD) with everything "just working".
When I see someone complain about unsupported/unsupportable chips in comments on online forums, especially one dubbed "Hacker News", I am puzzled how I in my middle school years acted out a pattern that is objectively smarter* than what I read in such comments. I also happen to first-hand know I am for sure not the only one with this vantage point. Those who comment about unsupported/unsupportable chips as if it is somehow an open source kernel's fault might want to take a moment to consider how others, and how many others, are viewing such drivel. For every one of us who take the time to point this out, there are 10,000 of us experiencing utter contempt, like as if we just got an unexpected whiff of some hot garbage.
[*]And, I honestly don't think I'm even that smart.
you got lucky with the hardware. there was a bunch of wifi cards that wouldn't work in Linux because there were no drivers. and then ndiswrapper came along and let you use windows drivers in Linux. now that was a user unfriendly procedure of getting it working. some chipsets eventually got native drivers like ralink or b53 but getting things working was not easy!
There was absolutely zero luck involved. As I already wrote in the previous comment, I did something very simple. I sought out a WiFi card that already had Linux drivers and then purchased that WiFi card. I didn't have to "do anything" to get the WiFi card working.
Agree.I also remember those days when it was so hard to get Linux to just boot up and get your display working correctly- it was almost like a rite of passage. It was just proving grounds for how much of an expert you were and the number of hours you spent in front of the PC, just to get things working.
My point is, good and bad memories will always stand out.
This is the main reason every IT professional I know disables auto updates of windows and manually trigger updates after testing (hopefully) on multiple dummy machines on the network.
I personally remember booting to safe mode to remove Windows updates to rescue the computers more than I can count.
Examples like that one I also found, but that's not really a "looooong list". If people can only show one single example as an argument it's kind of a moot point.
You'd experience at least 3-5 per year if you work in IT. There really is a long list but since it is not my argument, I won't list them after searching for an hour. The list starts early 2000s, not recent.
EDIT: Whatever, I will do the search for you since you cannot use google:
These are just the last quarter of 2023. There is over 2000 news but I won't link them Use keywords: Windows Update, Crash, and use the date option on google go before 2023.
Do you really expect internet strangers to sit down for hours and hours to make google searches for you just to prove you are wrong? I already spent 2 minutes of my life for you to learn something.
I don't have time for this, please learn how to use google and respect industry professionals and what they say.
I'm just asking you to provide valid arguments and facts that match the "loooong list" statement, and not extrapolate an overexaggerating from 4 google results with the excuse "you're just not googling hard enough" because that way you can also say "BigFoot evidence is real, you're just not googling hard enough".
All you could find were 4 examples in 2023? Hardly a long list, wouldn't you say?
I think my Android updates caused way more issues in one year and that's running an immutable HW that's well know and understood by the manufacturer, so 4 issues per year for Windows doesn't sound too bad, even though I had zero in 2023.
"In general, I apply Windows Defender updates as soon as they’re available. Why? Microsoft hasn’t screwed up any of them too badly. You’re better off applying those updates than letting them slide for a week or two."
Where can I find a list for all OSes? I’d assume such a list would have known issues with X11 etc. I want to ensure it’s not a case of surviviorship bias.
I don't think there is one... macOS doesn't have enough functionality-breaking updates to make a significant list, and Linux/BSD-based distros generally do cleanly segmented updates to individual apps and services rather than Microsoft's great big monolithic all-or-nothing OS update bundles that touch on dozens of services at the same time.
My own anecdote: When I got my M3 Pro in April and had to start afresh, it was stuck in a restart loop and had to take it to the Genius Bar; they asked me to answer ‘no’ to some question that I was answering differently. That was it. I have no idea on the root cause or why it was fixed this way. I don’t remember the exact screen where the answer was supposed to be different.
Attention to teacher is not equal between learners, trying to thoroughly assimilate the lesson is not everyone move, self challenging oneself with actual tests to ensure skill acquisition is rare, and going through the whole rabbit hole to figure out what untold assumptions the teacher leverage on and understanding the limits of these suggestions is the way only a few exceptional beings will follow.
And they've learned a lot from it. For example, MS no longer universally deploys updates across the world, they have a slower rollout to avoid just such an incident.
yes? that's 100x better! at the end of the day, internal testing just isn't going to catch every single permutation of customer configuration, so there's always a risk that something bad goes out. if you're that big, you'd start with .01% of the fleet instead of 1% of the fleet, so it's 100_000 before you get to 1_000_000, before going to 100% but neither Apple or Google have figured out a better way than that. It's industry standard at this point.
I can tell you they’re quite unhappy about it. Have a friend working there who frustratedly says it wasn’t their fault every-time it comes up. Which is quite often and at every social occasion since.
but it's kind of their fault? they designed the api that way, they decided what can be done in userland and what must be done via kernel. they at least _allowed_ it to happen every time.
You can't just let people do anything from userland, the performance would tank. As for restricting kernelland, EU competition regulators would not be happy if MS was the only one able to write anti virus software that runs in kernelland.
There is literally a ton of existing software out there that is keeping MS from doing exactly that. When it comes to avoiding breaking legacy applications MS scores far higher than any other operating systems out there.
And that has absolutely nothing to do with them coming up with better approaches then discussing them with industry for potential roll out, adoption, etc.
But instead, now they're in trouble they're trying to blame the EU for stopping their monopoly.
Do you honestly believe MS being unhindered by competition restraints would lead to better results?
Are you forgetting MS has already demonstrated how that goes, and been literally convicted for it?
Let me try to make it extremely simple so that maybe you might understand something.
Say I am running a shop, the EU tells me that under no circumstance can I not allow a product to be sold in my shop, even if that product is a ticking time bomb that can blow up the shop. And so hearing this, I create a document “Good approaches to sell time bombs”, and I mention helpful stuff like ensure the timer in your bomb is switched off when it is in shop. I also create an industry wide forum with all time bomb manufacturers and discuss best practices and time bomb methods with them to best sell it in the shop etc.
In spite of all this, there exists an idiot timebomb manufacturer who ignores all best practices, does not consider industry and builds a shitty time bomb that blows up the shop.
Now please educate me, apart from doing the only surefire thing and banning shitty time bomb manufacturers from selling in their shop, what should MS do?
As someone that worked at MS, on a team that worked directly on this issue (among other things) some years ago, MS did figure out better solutions and did discuss it with industry.
Kaspersky was running an SSL/TLS Proxy in the kernel IIRC and didn't want to have to move it elsewhere due to the fact it would require them to rework their product quite a bit.
The solutions MS (we) proposed were agnostic and overall better, the anti-malware industry simply doesn't want to make the changes as these things do impose technical work on existing products.
No worries. That wasn't at all evident from the above complaint.
Was the drive for this industry forum coming from dealing with the EU, or was it more from MS trying to make things better without needing the prodding?
There are ways around this that I've discussed elsewhere so I won't repeat them here.
However, think of it this way: Windows restarts, tries to load with new patch and crashes.
Question: why can't Windows be designed so that on crash it automatically restarts and loads the previous state sans patch?
Answer: Windows could be designed that way but it would require Microsoft to do many things it doesn't want to do. Some of which would require Microsoft to go back to the beginning and reengineer quarter-century or more old code from scratch, that means redesigning APIs and the underlying architecture from first principles.
Why doesn't Microsoft want to do this? It's obvious so I won't bother to spell it out.
Nevertheless, when the dust fully settles and someone outlines these alternative design strategies in great detail then it'll be obvious to everyone what a fragile stack of cards Windows has been constructed on.
Please don't post in the flamewar style to HN, such as you did here and downthread (https://news.ycombinator.com/item?id=41096774). It's not what this site is for, and destroys what it is for.
Well Microsoft did not publicly commit to using the same APIs, and no privileged access, for its own antivirus products. That's why the EU said no way; not because kernel access was revoked.
Yes, but then of course Microsoft is being obligated to open part of kernelspace to competitors, which is arguably "OK" from a competitive regulation perspective, but that then places a special burden on competitors to maintain code hygiene given the potential for crashes. It makes CrowdStrike's negligence all the more unacceptable.
I believe what philistine is suggesting is that Microsoft could have implemented their own security offering using a safer alternative like eBPF, and then opened that interface to competitors as well.
I think that would have been a proactive approach. That said, I'm not entirely convinced that the EU was right to place the restriction in the first place.
The article you shared says that Kaspersky filed a complaint, but I didn't see a clear statement there about what the outcome was. I do now see other reputable sources reporting that an agreement was reached in 2009 where Microsoft promised to allow vendors the same access to the kernel its security software had [0].
I think a proactive approach might have been for Microsoft to provide safer interfaces with the kernel, and then use those in its own security offerings.
That said, it does sound like EU competition regulation was a contributing factor here, and I think the EU is wrong on this one and that an OS vendor should not be required to provide unrestricted kernel access to allow security software vendors to compete.
Mostly unrelated, it seems somewhat interesting that this was Kaspersky insisting on kernel access... The US government seems convinced they are compromised.
> they designed the api that way, they decided what can be done in userland and what must be done via kernel
They didn’t have much of a choice - it is very hard to get adequate performance with real-time filesystem filtering without doing it in kernel mode. Not aware of any other mainstream OS which succeeds at that.
And they kind of had to provide this feature, since they’ve supported it since forever (antivirus vendors were already doing it back in the days of MS-DOS and Windows 3.x/9x/Me), and there is a lot of market demand for it. It is easy for Linux to say “no” when it never has had support for it (in official kernels)
But, as the blog post points out, it sounds like CrowdStrike is doing a lot of stuff in kernel mode that could be done in user mode instead - whether due to laziness or lack of investment or lack of sophistication of their product architects
> they at least _allowed_ it to happen every time
Microsoft, in allowing third party code to be loaded into their kernel, is no different from other major OS kernels, such as Linux or Apple XNU.
Apple is (increasingly) the most restrictive about this, and a lot of people criticise them for it.
Even Linux imposes some restrictions-which kernel symbols to export (at all or as GPL-only)—although of course being open source, you can circumvent all restrictions by changing the code and recompiling
The famous Tannenbaum-Torvalds debate happened all the way back in 1992. At the time, the most common microkernel was Mach, which had significant performance problems. NeXT/Apple solved them by transforming Mach into a monolithic kernel, making Mach (as XNU) one of the most popular kernels in the world today (powering iPhones, iPads, Macs, etc). But that doesn’t help Tannenbaum‘s side of the argument. And I don’t believe his own Minix did much better than Mach did.
Whereas, from what I hear, L4 and its derivatives have solved this problem in a way that Mach/Minix/etc could not. Yet still, it makes me wonder, if L4 has really solved it, why aren’t we all running L4? L4 has had some success in embedded applications (such as mobile basebands, Apple Secure Enclave); but as a general purpose operating system has never really taken off.
An application in which something like slow file IO wouldn’t be a problem - does it even have a filesystem? And we don’t know whether Intel has done things to make it an “impure” microkernel, like what NeXT/Apple did to XNU, or Microsoft did with win32k.sys
eBPF being able to crash the kernel is usually sign of a kernel bug. And it sounds like in this case it was even a bug specific to Red Hat kernels, introduced by a Red Hat patch.
That said, even if they are triggering a Red Hat kernel bug, CrowdStrike should be testing their software adequately enough to pick up that issue before customers do – and it sounds like they haven't been
That was more of a kernel bug than a crowdstrike bug. However, it's clear that they are pushing what you can do in kernel space to the limits, which is not a great sign.
I think if I understand the systems right Windows can roll back a bad driver update but the CS update wasn’t an update to the driver but instead updated a configuration file which CS updated outside of Windows Update. So from the Windows Update perspective the system started failing to boot with no changes to the system. Again though I don’t know if I totally understand what CS did and what capabilities Windows Update has.
No you can’t roll back bad driver updates in any OS, if you could then by definition they do not sit in the kernel space. You just want the security code to not run in kernel space, which is a decision MS could maybe make and become like Apple, though most security software would in that case rebel.
it depends on how bad. in Linux you can rmmod to get rid of the bad one if you haven't wedged it and fix your code, compile, and try again. I can't imagine that's actually different on windows if you know what you're doing. how do you think driver development happens?
The OS loads file A into the kernel. It crashes. It reboots. It decides not to load file A this time.
Wow, it's a rollback of kernel-space code.
Unless your argument is that you can't guarantee a rollback of every possible kernel driver, because it might have installed a rootkit while it had full control? Okay, cool, but this isn't a malware removal idea. It's an idea for normal drivers.
The big difference is that CS is not the user. In you analogy it's like your car allows you to drive off a cliff, and an (almost) essential part of your car (for example, the pedal) drives the car off a cliff.
Right, so a slightly better analogy would be if you wanted to install a remote starter, but then you find out that they can only be installed into Fords, because other auto manufacturers (Apple, Linux in this case) believe that tampering with the critical path (the engine, kernel) is unsafe. It isn't Ford who's at fault for allowing you to run some random engine modification, it's that mod that is at fault.
It got there because a user or administrator approved and installed it. It didn't just appear there, Microsoft didn't install it there. The user ran it.
Mmm… meaningless analogies are kind of meaningless?
More like:
If you install a security product that then prevents your car from starting; are they entirely blameless for letting you install it?
If you pull the hood up, tear off the “voids warranty” seal, ignore the “don’t open this” labels, crack the seals open and shove something into the engine… sure.
…but if you just slap a widget with the “vendor approved” sticker on your dash and it bricks your car; that’s a bit sucky right?
I do feel Microsoft is not entirely blameless in this.
It should be easier to recover from this kind of thing.
They should have been paying attention and made a fuss that one of the biggest security vendors has been doing this literally since they started.
I would bet money that until two weeks ago Microsoft was high-5ing them for best security practices.
It’s not “their fault” but they can’t just go “wasn’t us!”.
Before Microsoft comes into the picture the issues is crowdstrike pushing updates without proper testing, selling a product on which customers cannot control the update schedule, and customers for being so naives and not checking what the product they install on critical stuff do.
Microsoft tried to lock down kernel access in the Windows Vista era. Antivirus vendors went crying to the EU and they forced Microsoft to allow access to the kernel to third parties.
When a parking valet takes a car on a joy ride and crashes into a tree, we could blame the tree. We could blame the car owner for handing over the key. We could blame the auto manufacturer that didn't provide a "valet mode". We could blame the police for not detecting the joy ride before the crash.
All of these parties could do better (stupid tree!). But the real problem is the valet.
We can say that it is obvious that the electronics-heavy cars of today should anticipate rogue valets and build in protections. But we shouldn't let rogue valets off the hook for damages.
As a consumer, you could choose to only purchase cars that have "valet mode". So should we blame consumers who don't? If so, we should blame the airlines, hospitals, etc.--not Microsoft.
How about we prosecute valets unless they refuse to park cars that don't have "valet mode"?
Can Microsoft legally ban a competitor for percieved incompetence? I doubt it . partiuclarly seeing how much competence is shown with windows and MS teams software
Microsoft assigns driver levels to these guys etc. and allows them to load kernel mode components as protected etc.. If they do not allow that - CS cannot cause such damages. ofcourse, as you pointed out, this will then turn into some lawsuit blaming MS for killing competitors, even if they do it to try and protect their customers.
Just to be clear within the analogy: are you expecting the auto manufacturers to "force-eject" any hotel on Park Ave that has a record of valet mishaps? Or did you mean individual cars should force-eject the valet?
If a Caesars Entertainment property in Macao has enough incidents, should GM update the firmware on their automobiles to force-eject valets at Caesars Entertainment properties in Las Vegas?
Now imagine that GM actually operates valet services in Macao and Las Vegas. Should they be allowed to force-eject valets from competing services?
I am not a Microsoft apologist. I think they should do better. I think Linux and FreeBSD should do better. I personally avoid Microsoft products. But I place more blame on people who use MS products than I do on MS. After all, I never intend to hand my beat up old Corolla over to a valet so why should I have to pay for a "valet mode" feature that Toyota is forced to build into all their cars? Isn't it reasonable that motorcycles, 18-passenger vans, and scooters don't need "valet mode"?
In my book, the auto manufacturer is lower on the list of culprits than the valet, "the establishment that keeps a valet with an abominable record on staff", and the vehicle owner. But some place like Car and Driver could definitely prioritize encouraging GM or Toyota to develop valet modes over berating owners; so I don't mind a place like HN shooting a few arrows at MS. Unless the general public follows their lead and lets bad guys off the hook by shifting too much focus to somebody lower on the list.
> Just to be clear within the analogy: are you expecting the auto manufacturers to "force-eject" any hotel on Park Ave that has a record of valet mishaps? Or did you mean individual cars should force-eject the valet?
Not OP, but I think the analogy here is the hotel "fore-ejecting" (firing) the valet with a history of doing joy rides. That seems very reasonable.
In the analogy, it seems Microsoft is a car manufacturer. The hotel is the company that bought software from CrowdStrike. The problem is that Microsoft should not control who has access to which APIs, that is a huge can of worms, and actually called anticompetitive by the EU from what I understand. At MS level, either they publish APIs or not. If published, anyone should be able to write software for them. This is especially bad if MS themselves also sell security software that uses the same APIs. It would literally mean MS deciding who is allowed to compete with their security software.
I think it works better (please allow me to change it) if Microsoft is the hotel. Crowdstrike is the restaurant inside the hotel. The restaurant is serving poisoned food to the guests, who assume it is a decent restaurant because it is in their hotel.
Also the restaurant has their own entrance without security and questionable people are entering regularly, and they are sneaking into the hotel rooms and stealing some items, breaking the elevator.
At the same time, the hotel is in a litigation process with the restaurants association, because in the past they did not allow any restaurant on their premises. The guests, naturally, do not care about this, since their valuables have been stolen, and they have food poisoning. The reputation of the hotel is tarnished.
I don't think this works since Microsoft isn't the hotel. The hotel in your example chooses which restaurants are inside, but Microsoft doesn't. In this example, Microsoft is the builder who built the hotel building for a 3rd party. That 3rd party decides which restaurants it wants to partner with, as well as any other rules about what goes on in the building.
If the builder came around and made changes to ban the 3rd party's restaurant partner, that would cause a ton of issues and maybe get the builder sued.
Microsoft can't decide what can and can't run on their platform - the most they can do is offer certification which can't catch everything, as we just saw with Crowdstrike since they decided to take a shortcut with how they ship updates. Microsoft also had to allow for equal API access so they don't get sued by the EU.
Operating system (hotel) decides which programs run in kernel mode (Crowdstrike) but ok. Let me address the other point.
Again the reasoning of allowing equal API access to avoid getting sued is a false dichotomy: Microsoft could choose to make an OS that would not need such mechanisms to be simply usable.
They could also remove their own crowdstrike-alike offering, so that it would not be considered anti-competitive. They could also choose not to operate in EU. Of course, that would lower their profits, which is the real motive here.
Once you sum it up the reasoning goes: hospitals/flights can stop working because a company cannot lower its profits, and said company is not to blame at all. It is clearly false, the rest is sophism, and back-bending arguments IMO.
I am conceding that point (the "but ok" part). Maybe I could have expressed it better.
Please note, that in my analogy the hotel has input in which restaurant is allowed (opposite of your scenario). There are also not infinite Crowdstrike-like offerings, only a few. Same thing applies to the hotel, yes, only limited by the surface of the building and cultural norms.
I any case, the analogy cannot please everyone, and I can see how there are some errors with it in some aspects. In others, I consider it accurate. Using an analogy is an invitation to nitpick on it, so it is my fault really, but I could not resist.
There are other points in the analogy that I feel reflect very well how ridiculous it is to claim Microsoft has no responsibility whatsoever. IMO they do have at least partial responsibility. One cannot simply excuse them "because EU".
But this implies that even the guests who never went to that restaurant and have no links whatsoever to it might somehow still be directly suffering because of its presence.
In reality this doesn’t seem to be the case at all.
I'm expecting restaurant owners to fire bad valets.
Or in Microsoft's case, via regulatory, social, or software, prevent Crowdstrike from causing harm to their customers.
I'm aware it's a sticky regulatory situation, but CS has a history of these failings and the potential damage could be severe. Despite this, no effort (that I am aware of) was made by Microsoft to inform customers that Crowdstrike introduced potential risks, nor to inform regulators, nor to remove the APIs CS depends on.
I don't believe Microsoft is solely responsible, but I do believe that throwing all of the blame for the very real harm that was caused onto CS alone is missing a piece of the puzzle.
Last aside, every large corp has team(s) focused on risk. There's approximately zero chance they didn't discuss CS at some point. The only way this would not have happened is negligence.
Microsoft was required to let them have the same access their own software used. Which seems fair to me. Microsoft can remove those APIs entirely, they just can't restrict them.
Problem is that the establishment here is well the establishment. That is the state itself. Or at least one of them. As somehow MS is in position where for any slight anti-trust thing they will be prosecuted. Our system is setup to allow these actors in...
> All of these parties could do better (stupid tree!). But the real problem is the valet.
No, the operating system is supposed to provide secure access to hardware and isolate independent subsystems so they can't interfere with each other. That's its whole purpose for existing. The fact that people feel they need to deploy CS is a Microsoft failure. Windows is just not a secure OS.
You’re shifting practically the entirety of the blame to a company that at best was an accomplice to the issue.
I get that you hate Microsoft, but not everything is their fault and it’s disingenuous to pretend otherwise.
> ing. The fact that people feel they need to deploy CS is a Microsoft failure.
CS is also available and widely deployed on Mac and Linux. Is that a failure of Apple and all the distros? It literally took down Debian and Red Hat systems earlier this year, is that also not CS’s fault?
> CS is also available and widely deployed on Mac and Linux. Is that a failure of Apple and all the distros
Yes. All widely deployed commodity operating systems have terrible security designs. None of them have access control systems that enable the principle least privilege, let alone encourage or prioritize it, and none of them are written in robust languages that make verification of safety or security properties possible. Microsoft has made some headway on partial verification, but it's a far cry from what's needed.
> Yes. All widely deployed commodity operating systems have terrible security designs. None of them have access control systems that enable the principle least privilege, let alone encourage or prioritize it, and none of them are written in robust languages that make verification of safety or security properties possible. Microsoft has made some headway on partial verification, but it's a far cry from what's needed.
What, exactly, is your solution then? To never use a computer again? Because that's certainly what it sounds like.
Secure, robust operating system designs have been known since the 1970s. KeyKOS, EROS, CapROS. All commodity systems instead use classic access control lists, subject to fundamentally unsolvable access control vulnerabilities. seL4 finally implemented those lessons but it's far from a commodity operating system.
Can you point to an OS that can actually be used as a general-purpose OS? Or are you going to tell us that trying to run a web browser is actually what is fundamentally wrong with technology these days?
You could also choose to park the car yourself or plan for a secondary mode of transportation if something happened to your car.
Not the best analogy. The organization who deploys said software is responsible for the uptime of their systems. They didn't have to use CrowdStrike and if they do they should have a plan in the event of failure.
I noticed this at work and in some other contexts last week. We weren't affected by this, but most of the people that brought this up, even technical people (other fields, not security or OS or anything like that), think that this was a Microsoft and Windows issue. they all seem surprised to hear that Microsoft wasn't the root cause of this, and they all seem surprised, because no one knows or understands what Crowdstrike is or does.
Why are they being diplomatic, instead of plainly stating their contempt and revoking CS's driver/etc signing keys? Doing so would help to repair the reputational harm that CrowdStrike inflicted on Windows.
Are their lawyers telling them they can't impede CrowdStrike even though CrowdStrike is breaking Microsoft's product? They should do it anyway and dare CS to take it to court so they can publicly humiliate CS by dragging all the dirty details of their incompetence out.
this isnt even the first time its happened. Crowdstrike has killed an OS every month for the past four months.
At this point they are a threat actor. if you havent kicked their amateur-hour software out of your infrastructure by now, chances are good senior management and engineering have at least considered it formally.
That incident list is damning. Is senior leadership asleep at the wheel, or how can this many incidents possibly happen every 30 days for months on end? If leadership really cared, they'd make sure post-mortems and other best practices are in place to reduce the frequency.
Unfortunately, the executive disconnect isn't new. It's actually uncommon that they care about the reality for end users and customers (which is antithical to my entire ethos, hence why I get paid the medium bucks). Why bother waking up and going to work everyday unless you are contributing in some way to sustaining a better future for everyone? It's actually great for marketing and it's already going to be a tough 100+ years from today for our children, even with our collective care.
P.s. People can be so selfish, it kind of breaks my brain but not really. Have you seen the CO2 emissions visualization from NASA this week? It was a wakeup call for me.
“ For those who don't remember, in 2010, McAfee had a colossal glitch with Windows XP that took down a good part of the internet. The man who was McAfee's CTO at that time is now the CEO of Crowdstrike. The McAfee incident cost the company so much they ended up selling to Intel.”
so yeah, “leadership” (and that’s a loose term) doesn’t seem supremely concerned about much more than earnings
The Internet is able to transmit odors of rotting flesh????
Recently ordered an HP laptop
for some light work (not my startup),
and when placing the order said don't
include McAfee, that "I don't trust them",
all just from some odor!
CloudStrike runs in kernel mode? No wonder there are problems; kernel mode sounds like more of a threat than a protection.
Sooooo, for my Web server(s), McAfee and CloudStrike are issues I get to ignore. Problems avoided and time, money, energy saved!! Simple.
I assume T stands for [Sales and Marketing]Technology. Which makes perfect sense because these are their core departments that the whole company is dependant on.
The product itself is a secondary cost-center, probably less important than even accounting.
Have seen region-specific Field CTO roles partner with GTM teams to co-sell with customers. Product and role domain expertise without the organizational technology responsibility.
Now that's interesting. I wonder why neither here nor there anybody mentions GK's name. Fear of litigation?
IMO somebody who managed to collapse the most important infrastructure on earth twice in as many decades - not a small feat, I have to admit - should be known by name to the general public, lest he'll get another chance at it.
That was still not on of the most important piece of infrastructure on earth.
And outages were not as global as news outlets made it look to be. Crowdstrike may have been ubiquitous in some countries, but almost absent in others. And still, crowdstrike or windows windows aren't global pieces of infrastructure.
Tech needs something like the FTC that can ban someone from working in that area after multiple demonstrations of glaring incompetence. Or evil misdirection of competence.
> Presumably it doesn't matter that much and isn't worth spending money/manpower on?
Well, if they think any of the $20B of shareholder value lost recently has to do with the quality issues... Then perhaps they should reconsider. (keep in mind marketcap also represents their ability to raise capital in the future with more/less dillution)
Management often sees, “I have a dozen people on this.” When in fact the bus number was three, you laid one off, another quit and the third is sick or having life struggles.
Wikipedia is not an individual actor or a hivemind, so there is no capital-T "They". It's a system of multiple people each acting on their own accord. For a developing news story like this, I find this type of sourcing acceptable, especially because it is cited as "some person on the internet claims", not as "it is true that".
If you disagree with this choice of source, you can flag this part as needing better sources. The simplest way to do so is to just leave a comment on the talk page.
Never assume malice where incompetence will suffice. I have worked on teams where we could not get the basics like a test or integration environments signed off for months yet the managers expected us to go to production. Suffice to say production was also not signed off for half a yer and we had to improvise. I wonder is something similar was at play at CS?
I like the positivity that led to your suggestion. But I find it hard to follow the argumentation.
They know that mistakes can take out thousands and thousands of devices, therefore it is imperative they prioritize stability over rollout speed. They have more direct access to devices, than any 0-day would ever have, therefore there is a significant risk that they do more damage with an update than any 0-day ever could.
You have to remember, a 0-day could come to existence that threatens every system, but that usually happens once every couple of years (last one of that category was probably Blaster/I LOVE YOU). But CS risks of damaging the system every couple of hours with an update. Therefore, it should be tested to make absolutely sure it doesn't cause crashes.
MS should have something like Project Zero for Windows applications and drivers. Any app on more than 1-5% PC should be tested and fuzzed and ... And the vendor than pressured into fixing the issues. Even if it is not technically their fault, it is definitely optics problem for MS, half of the world refers it as Windows blue screen issue.
umm. Give a x days deadline and make after it public like Project 0 works, threaten to take away "Verified by MS" badge or create a WhatsApp group of Fortune 500 CIOs and badmouth in it.
Both of these have legal percussions: Microsoft could very well be called a competitor of CS, so they cannot force them to do something without getting accused of abusing their market position; and a publicly traded company badmouthing another publicly traded company with an awfully complex web of mutual investments is a very bad idea in general.
> However, nothing in that undertaking would have prevented Microsoft from creating an out-of-kernel API for it and other security vendors to use. Instead, CrowdStrike and its ilk run at a low enough level in the kernel to maximize visibility for anti-malware purposes. The flip side is this can cause mayhem should something go wrong.
> The Register asked Microsoft if the position reported by the Wall Street Journal was still the IT titan's stance on why a CrowdStrike update for Windows could cause the chaos it did. Redmond has yet to respond.
No commodity operating systems are designed for least privilege.
It hasn't happened because the market incentives for existing, widely deployed software are too great. Just patch and release is cheaper than starting from an actually secure foundation.
It's a little ironic they are taking the high ground on safe rollout practices when they had an Azure/365 outage caused by a bad config at the same time as the CS incident. Though to be fair, it only affected US central.
>Especially in teams needed to respond to customers about "Your windows OS is broken, our hospital patients are suffering...".
The reality is that this experience has been built into working for Microsoft since at least the 90s. The entire point of the compatibility effort they put into Windows 95 and later came from people blaming a Windows upgrade for breaking software that barely worked in the first place.
Windows Vista was known for lots of BSODs, but at least 60% of them were entirely due to nVidia GPU drivers crashing.
Reminder that Microsoft could have programmed Windows to notice if a driver has caused a blue screen three times in a row, and prompt if you want to disable the driver on boot. After all, Windows already collects how many times a driver causes a crash. This would have made recovery one click instead of heading into Safe Mode and needing BitLocker keys.
But they didn’t.
And Microsoft, I argue, also has blood on their hands for every hospital this hit. Giving users a prompt to disable the driver, after three successive failed boots, would have saved lives.
Oh I don’t know, the servers down, you go and look as a technician, and you simply see a screen saying:
“CSAgent.sys has caused a failure to boot three times in a row. Do you want to disable this driver? <Yes> <No>.”
You click “Yes.” Server reboots with CloudStrike driver disabled. The day is saved in 5 minutes instead of building a custom ISO image or going on a BitLocker key recovery spree.
At the same time though, imagine you woke up and CloudStrike hit your organization.
For most users, they’ll try clicking “Yes.” And then it’s back to work. After all, “No” just causes a blue screen again, might as well try the other path.
This would have been the difference between the IT department handling 10,000+ calls or a few hundred (plus sending out a bulletin) in many, many organizations. It also could have saved billions at this point.
Instead of prompting on the screen, disable the driver and boot directly into a recovery state that has networking enabled so sysadmins can push scripts and fixes? As long as it's not a network driver you'd be okay.
With modern developer tooling, it kind of is. Pick the right language for the right job. You can catch the vast majority of crashes in your code at compile time, to the point where the potential of 'making a mistake' is a statistical improbability. But so many companies just stick with C, and even ignore compiler and static code analysis warnings, because the machismo of their devs protects them much better. Software by companies failing basic risk analysis.
When parsing the config, the code crashed causing a BSOD. It wasn't even known it was malformed at that stage. A safer language would not let you crash when attempting to parse or validate malformed config. Unsafe languages require remembering and bothering to run extra steps to check your code is safe
Yes, it is a desired feature to bugcheck when flagrant violations of the kernel rules are detected.
CrowdStrike isn't special because it's security software, but there are good reasons you might not want to have the OS start disabling drivers randomly when the system isn't in a stable state.
One reason is partly because, depending on what happened when, you might disable the wrong driver. It so happens in this case that CrowdStrike's hand was caught in the cookie jar immediately because it did something blindingly obvious to the kernel core as wrong, but there are scenarios where heap or other memory corruption happens in your CD player driver and 5 minutes later when you start using the network, it crashes because the network driver was executing off in la la land.
In addition, sometimes it's just hardware failure that is a proximate cause, rather than a bad driver (I've seen bugchecks attributed to NTFS.sys that were ultimately the result of a dying hard disk).
If you're blindly installing software system wide, that has kernel access no less, and not accounting for failure of that software in your risk analysis then you are to blame more so than the vendor.
Certainly I expect some SLA is in place but that's only of monetary benefit and irrelevant to keeping critical infra online.
I think sueing MS for the behavior that ensued when people installed a rootkit directly into the kernel and opened all the ports on their network to let that rootkit get used, is... excessive. Both MS and CS should have had a fail to previous good kernel ability, but the negligence here is clearly with CS for not even trying a blank data file in the automated tests for a piece of safety critical software, and then not using canary deployments before pushing to millions of devices.
But do I think Microsoft was negligent in not building that?
No, I think that's going too far. Windows already has Safe Mode -- as you note -- to allow for manual recovery, which is what people are using.
I don't think it makes sense for it to be Microsoft's legal responsibility to protect its users from software with a critical bug that wasn't written by Microsoft. Otherwise, where would it end? If a third-party program tries to delete all your user data, is it Microsoft's legal responsibility to check whenever a process is deleting a lot of data, and intervene with a confirmation dialog? Is it Microsoft's responsibility to protect you from all malware and ransomware, no matter how cleverly written? Is it Microsoft's responsibility to constantly cache program state on disk so that when a third-party program crashes, you don't lose your data since your last save?
I think that's going too far, in terms of legal obligation.
Microsoft may be negligent in selling a product unsuitable for these applications. Windows is unsuitable precisely because it can be brought down by third party updates, such that it cannot recover without manual intervention by technical experts. Third party vendors are forced into writing unsafe kernel drivers because Microsoft does not provide sufficient user mode APIs.
Windows has a dated design and a security model no longer fit for purpose. As for your other example, it could be protecting users from malicious programs that may delete data, simply by having a better security model, like Android and iOS.
I don't think Microsoft can be negligent here, because Windows isn't being brought down by Microsoft updates.
Somebody bought Windows, and bought CrowdStrike. CrowdStrike is negligent, and possibly also the person/org who chose to rely on Windows+CrowdStrike without a backup plan if that resulted in further damages to others.
Third party vendors are absolutely not "forced into writing unsafe kernel drivers". They can properly test things to write safer code (which CrowdStrike infamously didn't). And kernel mode is fundamentally required for security software like this, as far as I understand.
And using app-based mobile OS's is not necessarily a useful comparison point. They are limited in all sorts of ways that desktop OS's are not -- and don't you hear people here on HN constantly complaining about that? A better comparison point is macOS and Linux. CrowdStrike also crashed Linux, and macOS still lets you bypass SIP if you want to.
Imagine I've installed CrowdStrike under the assumption that it makes my system more secure. Why would I want the OS to allow the system to boot up in a less secure state by providing a prompt for that? Most users will just click whichever option gets them back up and running and IT will have no control over that.
This is very much a “easier said than done” situation that I would think Hacker News of all places would be better about when it comes to “just” doing something in code.
First Windows already does something similar. After 3 it is supposed to boot into WindowsRE which gives you options to revert to a previous version, uninstall updates, and I believe also reverts configurations like recent driver installations.
The problem here though, CrowdStrike itself didn’t update. It updated a definition file (last I saw at least) and that likely would not have been caught by Windows as a new version.
Also frankly, not super thrilled at the idea of Windows just deciding to disable/uninstall something except for rolling back (so a previously working config) due to how things could interact. This situation could have been far worse and harder to recover from.
In this case maybe Windows could have noticed that the configuration update is what was causing it and rolled that back, but it’s possible it would have just re-downloaded the file when it started back up anyways.
Regarding saved lives, do we actually know that anyone’s lives were lost due to this? My local hospitals were still performing emergency surgery.
Windows does do that by default. If it fails to boot, it will start an "Automatic Repair" screen and it will offer to disable drivers (ie: Safe Mode), or sometime just disable the driver itself.
The problem is that CrowdStrike doesn't want to let you start the computer without it running. It's the reason why it's an ELAM driver - it's marked as required for boot, so Windows won't try to boot without it, much like it won't if you remove crucial hardware drivers. I guess what they are trying to avoid is malware crashing the kernel driver which then gets disabled letting the malware roam free, without realizing that the cure is worse than the disease.
So attackers just have to find a way to trick the drivers into crashing 3 times and they have full access to the system without the pesky security systems in place? Nice!
I do have to wonder how many agonising layers of review this went through with the marketing and legal departments as part of shifting the blame.
If you want to decide which OS/distros to avoid for critical stuff, look to see who's learning from the incident (even if not bitten by it) compared to those saying "it wasn't our fault" (and that's not just MS).
Oh I like this breakdown a lot. Fairly technical, links to resources used, flow of debug process, didn’t get lost in a the weeds of details and how clever they were. I wish more debug retrospectives were like this. It seems like you end up with 100 pages of analysis or a couple of vague paragraphs.
I'm going to be the controversial one here and say that, as bad as CrowdStrike was, the alternative of having only Microsoft be able to decide what people can do is far worse. I've already seen many others trying to use this incident to advocate for digital totalitarianism.
Microsoft as the OS vendor will always be a potential source of updates that crash computers. Now with a third party, you’re adding another level of risk.
Interesting that they bill Defender as "requiring frequent OS updates," alleging that their solution is somehow better in that regard. Are the suggesting that by installing CrowdStrike, you don't need to update Windows anymore? It really reads like that.
Did either release from MS or Crowdstrike explain how this crash bypassed QC? I'm still baffled that a 100% repro crash even made it anywhere near the later stages of QC. This is something easily caught by the earliest CI phases , at the developer and at least first build automation phase, let alone human QC.
I heard something similar. that they deploy content separately from code, but they don't test all of the combinations of code + content. This crash was from "stable" code in the driver mixed with a corrupt or incomplete content file (config, etc) , triggering the null-ptr exception .
Sounds like one of those companies where you get hired and are shocked by the sausage factory you just stepped into
In February they added new code that allows to spy/block named pipes.
Named pipes are pipes of communication that processes can use to talk to each other, as an alternative to sockets.
For example Chrome uses them between the user interface and the actual page renderer.
In March they tested it in staging, said it was fine, pushed to prod with few rules in April, still looked fine.
In July they added a new rule, which was deployed to 100% immediately, as from their perspective, a new entry in a database definition doesn't need testing nor canary deploy
(which is still irresponsible, because bad rules could cause damage as well like any security/antivirus software, even if the parser didn't crash, but it could have blocked legitimate actions or files)
Right, so it seems two egregious errors: no (or highly lacking) fuzzing of kernel modules acceping arbitrary input, and no testing of configuration changes given it's ingested by a kernel module.
My understanding is that they ship precompiled, templated scripts. The "content" updates fill in these templates with configured values. They test the templated scripts, and they validate the content, but they don't validate the content bound to the script. The garbage content was apparently valid but its behavior when used was not.
Their language for describing this design is obtuse and confusing.
“Windows has announced a commitment around the Rust programming language as part of Microsoft’s Secure Future Initiative (SFI) and has recently expanded the Windows kernel to support Rust.”
Telling that there’s no mention of eBPF, which is standard on Linux and available on Windows, but hasn’t been brought into the main Windows OS. Static analysis might or might not have caught the Blue Friday bug, but it certainly increases the protection level over the current do-as-you-wish model for kernel modules.
So how did this kernel level driver get through WHQL verification? The Static Driver Verifier should have caught this.[1] Do some security vendors get to bypass that? Microsoft is very quiet about that.
That's the sort of thing a negligence lawyer focuses on. Partner at Brown Rudrick: "The most likely legal theory will be one of negligence. [Congress] will drag the guy over the coals, they'll maybe implicate him and his company and put in place a negligence action. There'll maybe be a couple of plaintiffs lawyers who dig up some exceptional theory on negligence, and get some class action lawsuits going. Again, we still don't know all the facts in this case, and there are other dimensions which have not yet been fully explored, including how CrowdStrike had access to kernel level updates on the Microsoft operating system? How come Microsoft didn't have any control over these updates being pushed on their kernel?"
Not all potential null dereferences are covered by the verifier, they even give an example where the rule is not triggered, but null may be dereferenced by the code.
I am still to this day gobsmacked how a company the size of Microsoft doesn't do all of it's security in-house like Apple, which locked down kernel access to macos some time ago. The blame is mostly on CrowdStrike, but Microsoft does share responsibility in allowing third-parties to pepper the kernel with whatever code they want to.
The EU didn't make you click 'Accept cookies' on every website, websites decided to interpret EU regulations as 'Let's make it super annoying for users to not accept, so they accept out of laziness'.
It does matter because you spin things to make it sound like the EU is anti-consumer when they are frequently the ones improving consumer protections and privacy.
the trade off is either you make your own hardware, or make exceptions to integrate with other OEMs, which causes the fragmentation problem that security vendors exist to provide solutions for. Apple doesn't have that fragmentation problem, but they also don't have as rich an enterprise ecosystem. Android is the definition of a fragmentation problem, where instead of resolving it, Google manages it and calls it an ecosystem, and they're right.
Fragmentation makes consistent governance/security impossible, but the heterogeneity also limits the scope of incidents. Apple balances its greater monoculture risk with deep control of the underlying hardware, to where as a user or developer you're only ever interacting with a very, very high level abstraction.
What I heard is that CrowdStrike normally rate limits pushing a fix. This is so that if the fix is bad, the damage is limited. But for some reason, the rate limiter was turned off and the update went out to everyone.
> What I heard is that CrowdStrike normally rate limits pushing a fix. This is so
that if the fix is bad, the damage is limited. But for some reason, the rate
limiter was turned off and the update went out to everyone.
Is this true though? They've released a Post Incident article:
> How Do We Prevent This From Happening Again? Implement a staggered
deployment strategy for Rapid Response Content in which updates are
gradually deployed to larger portions of the sensor base, starting with
a canary deployment.
So it seems, if I understand this correctly, they've just implemented
the rate limiter as a response to this incident.
one thing from this whole fiasco that i wished bring to conversation was the fact that (crucial/market-dominant) digital/IT services don't have the same level of liability as mundane, physical goods.
a simple plastic covering of your new dyson has more legal scrutiny and action (see the "children may choke" warnings they all need to come with) than software that we otherwise block in the name of "national security".
given how much overvalued tech companies are in this region, i believe it is high time to start legally recognizing the real-life impact of digital tech. to hell with the "but muh innovation" argument.
437 comments
[ 0.12 ms ] story [ 569 ms ] threadWhich, I can't fault them for trying, considering Crowdstrike isn't their product...
This kind of issue would not be possible in Linux/Macos unless the user configures them to allow the same kind of direct kernel access. At least for macos this is off by default. Not certain about linux but I think the default is more locked down than Windows.
Microsoft attempted to lock kernel access in 2009, the EU complained, so Microsoft kept it open in order to allow "competition in security vendors", and this is the result.
And the EU complained only because the likes of McAfee complained. Microsoft here is "dammed if you do dammed if you don't" since so many of these security vendors make a living by being able to act like malware themselves in order to sell security. If Microsoft would lock them out of the kernel they'd all scream "anti-competitive" to the regulators.
In a more fair world (that also valued economic productivity/growth more) companies which completely ignore that wouldn't survive, though.
I'd hope security products in the future leverage this more than custom kernel-mode sensors.
'course, Microsoft also put turing complete scripting in ring 0 years ago for performance reasons (TTFs - XML/HTML parsing and GUI rendering too - to beat other OSes apparently) and that certainly did lead to exploited vulnerabilities...
https://googleprojectzero.blogspot.com/2016/07/a-year-of-win... https://gist.github.com/Nevor/ed3719dad0cf66893e42a9ba024c91... https://learn.microsoft.com/en-us/security-updates/securityb... https://www.fortinet.com/blog/threat-research/one-bit-to-rul... https://learn.microsoft.com/en-us/security-updates/SecurityA... https://news.ycombinator.com/item?id=9769099 (this comment in particular https://news.ycombinator.com/item?id=9783863)
[1]: https://lwn.net/Articles/830154/
[2]: https://openzfs.github.io/openzfs-docs/man/master/8/zfs-prog...
This post explains why security software has historically run in kernel-mode, and really seems to be pushing new technology that Microsoft has that would push security vendors into user-mode (with APIs that attempt to assist with many of the reasons why they have historically used kernel-mode).
Crowdstrike already runs in user-mode on both Mac and Linux (from what I can tell), and it seems like running in user-mode on Windows would significantly lessen the risk of catastrophic failures like a blue-screen-of-death. I know the bulk of the failures here belong to CrowdStrike, but I can't help but think about the fact that Apple kicked security vendors out of kernel-mode a ways back, and that if Windows had done similarly, an issue like this probably wouldn't have been possible. By even offering kernel-mode options to external vendors, I believe Microsoft is creating risk for themselves.
Like others already said, Microsoft already tried to do that with PatchGuard in 2006 with the launch of Windows Vista and the likes of Symantec and McAfee complained to the EU about this would harm the sales of their products, so the EU told Microsoft to not do it in 2009[1].
Apple has the luxury of a small market share on the desktop PC space to not attract the attention of the regulators, plus a user base that's used to Apple constantly rewriting the OS, deprecating APIs, switching CPU architectures, etc. without giving a fuck about breaking backwards compatibility or cutting off developers access to OS features their products use and getting away with it, luxuries that Microsoft doesn't have.
IMHO, sticking with Window's default security and not using third party anit-malware has made Windows vastly more secure and rulabile than it was in the days when you'd be looking on installing the likes of Symantec or McAfee for your "protection" which ended up acting like malware after a while throwing dark patterns at you to milk more subsection fees, so as much as it hurts their sales, it's important for the regulators to understand that security is far more important than the regulations they put on Windows for Internet Explorer and Media Player and just like Apple's apps-store, it's sometimes better to let the original product maker handle security and not leave the product open at all points just so some of these bandits can make a living selling security for it. It's like foxes complaining to regulators how chicken wire is a threat to their existence.
[1] https://stratechery.com/2024/crashes-and-competition/
It also isn’t a separate product, it is bundled with the OS.
MS offers a full competitive product, and they allowed that product to use features and capabilities that wouldn't be available to others. At their scale, you can't do that without drawing scrutiny from regulators. They could have at least kicked their own product out to userland, or just stopped offering one and focused on kernel stability the way Apple has.
Since this is all spitballing, perhaps they even suggested that the EU, and the EU said "no, it must be kernel access" and capitulated because this served their own interests while also letting them blame the EU for the lack of kernel stability.
The difference between XProtect and Defender, is that XProtect utilizes the same kernel interface that Apple provides to other EDR solutions, whereas Microsoft was / is only willing to implement Defender using an in-kernel solution.
As a result, it would be anti-competitive for Microsoft to deny that level of access to third party EDR solutions.
I do wish that the EU would offer a "grace period" for "platform owners" or "gatekeepers" that would allow them to use non-public APIs for their first-party solutions while they stabilize and work out the public API offering. I think there should be some requirements for them to "earn" that grace period, such as allowing interested third-parties to get early access and provide feedback on those APIs. Regardless, it takes time to stabilize those APIs, and the current EU approach of requiring all new APIs to be fully public on day one just isn't workable.
I genuinely want to understand this - how everyone else got it right and this entity got it wrong.
Well, both dominant mobile operating systems already enforce a very strict security model. Too strict if you ask me because on neither you as the person actually owning the damn thing can become root without anything in the system throwing wrenches in your way, but hey.
The result is, most people don't even need a virus scanner, and the blast radius of a bug in an application is usually limited to whatever data that application can access. It's been ages since I heard about the last major mobile OS malware in the wild that didn't involve one of the world's secret services or their civilian suppliers - and that way, one can argue, all the security theatre actually works: it's driven the price of successful attacks so high that only nation states can afford it, and even these have to restrain themselves to only use these exploits against targets of extremely high value.
Is it possible to rootkit phones? Yes; does a typical user do it - heck no.
iOS (and to a lesser degree, macOS) use notarization to be able to pull apps found of being risky as needed.
Additionally they implement much stricter levels of sandboxing so that even if something slips by all detection, it’s heavily limited in what it can do without a users explicit permission or a catastrophic vulnerability.
(There's a big android user population without access to the Android app store at all)
Cars are sold with integrated radios and players. But at the same time there were independent companies selling car radios back in times when they were exchangable. Now external players are gone, everything is integrated, and the market for custom car players is dead. And nobody cares! One could say that car manufacturers don't offer the same API for car player companies.
I think that Microsoft is the king of their system, and can do whatever they please. If that doesn't sound practical or trustworthy for a company, then maybe the company just shouldn't release the product on their system. Use a different platform. Because if you release a product on their platform, then you're saying that you're okay with their rules.
This doesn't exist. It's briefly hinted at in their conclusion, but right now it's simply not there.
There is no userspace equivalent of filesystem minifilters, ObRegisterCallbacks, etc.
So I guess I'm not sure I see validity in the claims of those blaming the EU here. It seems as though the EU would have allowed Microsoft to kick users out of kernel-space if they had APIs that allowed making security products in user-space. Like Linux/Mac already appear to have.
I would be delighted if their long-term solution is eBPF which provides full anti-malware hooks, but again it's unfortunately not there yet.
It seems like madness to me.
Crowdstrike provides a Linux kernel module, and expects users to manually install an extra Secure Boot key for it, as part of their corporate laptop setup procedure.
This has always seemed inadvisable to me, but checkbox checkers gotta check checkboxes I guess.
Sure, the whole thing is definitely a hard problem, but CS fucking up even the most basic QA **and** error handling ... it just shows how ridiculous their whole claim to having super fancy technology is.
Yep. You just have to pretend that everyone who deployed Windows had an actual competitive choice available to them.
> A second benefit of loading into kernel mode is tamper resistance.
I guess availability is negotiable after all.
Could you elaborate? How is that related to security and availability being non negotiable?
The security and availability, to the extent they even exist, are clearly not part of the market's decision making process.
Did hospitals have a choice in which OS their MRI machine runs? Are those not "critical?" Or should we just not "actually care?"
> ATC for example
Were they impacted by this outage? Isn't the reason flights were canceled is because the _carriers_ systems are the ones that had issues?
> So did people designing bespoke POS systems.
Really? I would assume most of that is down to the hardware, like cash drawers, credit card readers and order printers, which is likely third party and proprietary, and is only available and supported on Windows. Do you have evidence otherwise?
> "nobody got fired for choosing Windows".
You've recognized the same outcome I have but have gone to great lengths in an attempt to obscure the reasons for it happening. Why?
There were reports of airports disallowing landings of incoming planes due to controllers being unable to provide separation.
There are card readers available for Linux, it has been long standardized. Cash drawer is a single solenoid.
Major supermarket chains order and supervise delivery of custom POS solutions integrated the way they want from companies like NCR.
NCR offers Windows, Android and Linux POS bases, but supermarkets tend to choose Windows.
Nope, they aren't ready to pay for another platform that is their choice . If customers paid for linux or mac support there is no shortage of developers ready to cater to that.
Unwilling to pay for multiple platforms is still a choice
If enough of them do. A single customer can't convince companies to add that support by paying a reasonable amount.
Since there's no collective bargaining happening, this does not show a lack of willingness to pay.
Also, you're ignoring all the software that is already paid for and often doesn't have developers any more.
Vast majority of the enterprise software was built bespoke or heavily customized for a customer for a long time and still is.
SaaS / product based multi-tenant software development becoming popular is not the only revolution in software recently we also had many improvements that massively reduced the cost of developing apps in the last 30 years. Multi platform support has become exceptionally cheap with browser as a platform and electron style frameworks for native apps.
The EU didn't say that Microsoft couldn't kick vendors out of the kernel, just that they couldn't do so without having the APIs available that would let security vendors operate outside the kernel.
Mac and Linux have such APIs, so CrowdStrike operates in user-mode on those platforms, so those platforms do not give security vendors the ability to crash the operating system.
Microsoft could have come up with a kernel API that their own malware (and everyone elses) product could make use of. They did not.
If Defender was also kicked out, it would have been fine, but it wasn't.
Using a monopoly in one industry to capture the market in another industry is what anti monopoly laws are meant to prevent.
Microsoft was prevented because they wanted to retain a commercial business in their security products having special access while locking out everyone else.
> Providing safe rollout guidance, best practices, and technologies to make it safer to perform updates to security products.
> Reducing the need for kernel drivers to access important security data.
They are being as diplomatic as they can, but it's definitely a slap to CS. Read as "they don't know how to roll things out, they need guidance on basic QA practices, we'll happily teach them...". Then, they list a set of facilities running in user-mode to avoid needing to run as many things in kernel mode.
I would be interested what the water cooler discussion about CS was like inside Microsoft. Especially in teams needed to respond to customers about "Your windows OS is broken, our hospital patients are suffering...".
What I've heard from people in the industry is not this silly "oh no, crowdstrike is so incompetent" b.s. that is being spread on sites like HN and reddit but more of an empathic "it could have been us" sentiment. In this write up as well, Microsoft knows they have caused their share of outages, it is a technical write-up but in part, it is to cover their bases for government investigations and lawsuits that will arise from this incident.
And in part, they are also responsible for recovering from third-party driver errors and repeated boot failures caused by faulty drivers.
For being such a large cybersecurity player and deploying updates to 8.5 million devices, their quality control practices are embarrasingly lacking.
That goes even further, because apparently they were fully blind and didn't have crash metrics.
"Ok we push the update, and pray".
Crowdstrike, for negligently not rolling out updates gradually.
And emergency services, if they don't have robust fallback procedures/systems for when their IT system goes down. I mean it's totally fine if regular doctor's visits get postponed, but 911 should never go down just because their computers down. Just like aircraft have redundant systems, so too should 911.
(The company that makes and sells the OS -- I don't see any negligence there, in this case. If security software fundamentally requires running at the kernel level and Microsoft allows that, I don't see how Microsoft can be at fault.)
How about we let the lawyers figure out who had what liability, just like with the av/edr industry, we should know when the subject matter is outside our area of knowledge and expertise.
If failing to do unit tests to get full coverage (line and branch) on software that gets loaded on security sensitive infrastructure isn't unusual then things have sure changed in the decade since I was involved. We had to get that coverage at the assembly level and then peer review the tests to make sure they were meaningful and correct. Every time the code changed. Code that failed to properly sanitize input files without panicking should not pass normal safety critical code pipelines.
I do wonder if windows already has some disclaimer about being unfit for such use cases, in which case yeah, lawyers will have some fun, but even with that sort of protection, CS is clearly marketing to critical infrastructure, so it has no moral defense whatever lawyers and the law says.
https://www.techradar.com/news/google-chrome-not-working-mic...
Did Microsoft do a staged or canary roll out with that? This is not a software update, if you're making such comments then you're speaking about something outside of your field of expertise.
8.5M all at once feels insane.
I'm more than comfortable labelling parts of Microsoft as incompetent as well.
Everyone that worked in an operational incident response role has blocked some indicator like an ip address or a domain. you don't do gradual roll outs for those either, and i've seen people cause outages by skipping a check or making a mistake. this is similar in many ways to that except it was for a named pipe. This could probably have waited for a canary deployment, but in general the class of content that is being deployed would be deployed right away, I'd be surprised if their practice is considered "bad" by any measure. I've seen Microsoft also deploy email quarantine signatures and defender updates that caused large scale impacts.
Here is a link of what Microsoft did earlier this year:
https://www.techradar.com/news/google-chrome-not-working-mic...
If they had canary deployments, that wouldn't have happened. I had rules that were causing chaos because of that. Now imagine if defender had a bug that caused it to crash because of a signature update. The impact would be magnitudes greater than what you saw with Crowdstrike. It's really frustrating to see the lack of technical critical thinking and arm-chair experts acting like they know what they're talking about.
Hours... Wouldn't a 15 minute canary have found this problem about 14 minutes before it hit wider deployment?
Their post-incident report [1] also stated that they intend to improve testing by "using testing types such as: local developer testing". One has to wonder what, if any, testing they were doing beforehand.
[1]: https://www.crowdstrike.com/blog/falcon-content-update-preli...
The update literally crashed the system it was used on.
There’s no way they couldn’t know that unless they never ran it. Right?
Is this one of those things that only happened to 10% of users? Because I haven’t seen that reported anywhere.
As far as I'm aware, it affected all systems using Crowdstrike.
More likely they were following a playbook to the letter, and were therefore 100% of success.
https://news.ycombinator.com/item?id=41095788
Some companies could have a severe and readily reproducible bug get through testing.
A few of those companies have a hand-rolled update mechanism, and can accidentally break their ability to roll back a bad release.
A few of those companies are in a position to push a release that breaks not only their own software, but the entire OS.
Very few companies in that position would roll out to 100% of client machines in a single worldwide deployment.
> It is possible today for security tools to balance security and reliability. For example, security vendors can use minimal sensors that run in kernel mode for data collection and enforcement limiting exposure to availability issues. The remainder of the key product functionality includes managing updates, parsing content, and other operations can occur isolated within user mode where recoverability is possible.
...was about as close to tetchy as a post like this would ever get. Basically they are saying "there was no good reason at all why CrowdStrike had to put so much code inside the actual kernel." And with the benefit of hindsight, it's a strong point.
Their business is corporate spyware to surveil employees, ofcourse they'll use any tactic to make it work, that's the why. And their EULA states there is no liability for the company:
https://www.crowdstrike.com/terms-conditions/
Dirty policies on top of dirty practices.
What?! Anything you do on your corporate provided laptop is always gonna be logged by IT for security in every large company everywhere, that's news to nobody, but your company doesn't care that you use your corpo laptop to book your vacation, IT has better things to do than narc on you for that.
If your boss wants to actually spy on you they don't need Crowdstrike, there's other SW dedicated for that depending on the laws in your jurisdiction but that' not what Crowdstrike is for.
If you want complete privacy from your employer, just use your personal machine for your private activities instead of your work laptop, why is this so hard?
Yeah sure, but wow many of those are large non-tech companies?
You massively overestimate the tech competency of the average PC user if you think it's normal in most companies to not have security monitoring solutions in place or over the internat activity. In our latest phishing test IT did, several users fell for the trap, despite it being a tech company. There's always gonna be someone careless one day and companies want insurance policies against that.
Having such solutions in place doesn't mean the company doesn't trust you, it's more like that old Russian proverb, "trust but verify", and for ticking security compliance boxing as an insurance policy.
Everyone makes mistakes, it's only human. So more like, speak for yourself, if you think your internet activity at work isn't logged anywhere.
It won’t be about distrusting anyone in specific either, but something will go wrong for which you need to be monitoring every PC to find out what is going wrong.
But so can SCCM/Intune from MS or another RMM like Datto that IT uses to manage PCs...
Microsoft has a loooong history of botched (security) updates, so I'm not hopeful they can teach Crowdstrike much.
I can only find a couple of examples after googling, which a bit smaller than a "loooong history" you're talking about, so unless Microsoft is paying Google to delete results, maybe you're mistaken.
Granted that was a while back but painful memories die hard.
That was WIndows XP 20 years ago. Please bring arguments about modern Window 11 security which is the current up to date product they're selling and supporting not scenarios that haven't happened in 20 years.
[0] https://www.tomsguide.com/news/windows-11-update-causing-blu...
This is basically "take a screenshot every 30 seconds and compile it into a timelapse", but on steroids, and the same appeal, and arguments wrt. who gets to run it on whose machines, all apply.
However, without metrics that show BSoDs from patches (which MS will likely never share), it’s hard to see if things have improved or regressed. If they regressed, someone up in their leadership chain is hopefully following the constructive discussion here.
So yes it would start in the past, but it then has to continue for a long time.
Pointing out that a company was bad 20 years ago isn't enough. You need to show they were also bad 15 years ago, and 10 years ago, and 5 and/or 25 years ago.
So complaining that the only evidence was so far in the past is valid. The original goalposts were not reached. (Well, someone in another part of the thread eventually listed every google result for a windows update making anything crash, but that doesn't really establish that microsoft is "botching" updates at a level significantly above background noise, which I think was the original intent.)
Also, if we're actually getting into this, the XP gripe had nothing to do with updates. That's moving the goalposts half a mile in the other direction.
??? You specifically asked for it! What are you doing.
That didn't run their own enhanced visibility on their own cloud platform.
https://www.csoonline.com/article/564499/3-leaked-nsa-exploi...
You have no idea the cruft and technical debt Windows has in order to maintain its backwards compatibility.
Both OS families have changed much since that time.
That'd be cool if one day I can get the laptop running on battery and not just on sector.
Let me just setup it.
Wait a second, how do I wake up the screen again and get out of this hibernation stage ?
Why are all the fans stuck in 100% now ?
Errr, first let's see if I can get the trackpad working.
When I see someone complain about unsupported/unsupportable chips in comments on online forums, especially one dubbed "Hacker News", I am puzzled how I in my middle school years acted out a pattern that is objectively smarter* than what I read in such comments. I also happen to first-hand know I am for sure not the only one with this vantage point. Those who comment about unsupported/unsupportable chips as if it is somehow an open source kernel's fault might want to take a moment to consider how others, and how many others, are viewing such drivel. For every one of us who take the time to point this out, there are 10,000 of us experiencing utter contempt, like as if we just got an unexpected whiff of some hot garbage.
[*]And, I honestly don't think I'm even that smart.
My point is, good and bad memories will always stand out.
https://www.techradar.com/computing/windows/windows-11-updat...
This is the main reason every IT professional I know disables auto updates of windows and manually trigger updates after testing (hopefully) on multiple dummy machines on the network.
I personally remember booting to safe mode to remove Windows updates to rescue the computers more than I can count.
EDIT: Whatever, I will do the search for you since you cannot use google:
https://www.pcgamer.com/an-odd-bug-in-this-months-windows-10...
https://www.windowslatest.com/2023/10/22/windows-11-october-...
https://www.bleepingcomputer.com/news/microsoft/windows-10-e...
https://www.windowslatest.com/2023/02/09/microsoft-confirms-...
https://www.windowslatest.com/2023/07/16/windows-11-kb502818...
These are just the last quarter of 2023. There is over 2000 news but I won't link them Use keywords: Windows Update, Crash, and use the date option on google go before 2023.
I don't have time for this, please learn how to use google and respect industry professionals and what they say.
I think my Android updates caused way more issues in one year and that's running an immutable HW that's well know and understood by the manufacturer, so 4 issues per year for Windows doesn't sound too bad, even though I had zero in 2023.
OK, please show us the proof then. If it's as regularly indeed like you claim then it must be documented somewhere as a greppable list.
Tech blogs would have a field day getting traffic on their site by keeping track and documenting on such regular mistakes if they exist.
[1] https://www.askwoody.com/ms-defcon-system/
"In general, I apply Windows Defender updates as soon as they’re available. Why? Microsoft hasn’t screwed up any of them too badly. You’re better off applying those updates than letting them slide for a week or two."
It's the other Windows Updates that they botch frequently enough to make people wary of patching immediately.
https://www.manageengine.com/patch-management/resources/micr...
- https://www.macworld.com/article/671831/macos-wont-install-f...
- https://askubuntu.com/questions/1231849/how-to-fix-update-pr...
My own anecdote: When I got my M3 Pro in April and had to start afresh, it was stuck in a restart loop and had to take it to the Genius Bar; they asked me to answer ‘no’ to some question that I was answering differently. That was it. I have no idea on the root cause or why it was fixed this way. I don’t remember the exact screen where the answer was supposed to be different.
https://www.forbes.com/sites/daveywinder/2024/07/27/microsof...
> Microsoft has a loooong history of botched (security) updates, so I'm not hopeful they can teach Crowdstrike much.
Experience is the best teacher
If they are, then you could be right. :)
But instead, now they're in trouble they're trying to blame the EU for stopping their monopoly.
Do you honestly believe MS being unhindered by competition restraints would lead to better results?
Are you forgetting MS has already demonstrated how that goes, and been literally convicted for it?
https://en.wikipedia.org/wiki/Microsoft_Corp._v._Commission
(there are plenty of other examples)
Say I am running a shop, the EU tells me that under no circumstance can I not allow a product to be sold in my shop, even if that product is a ticking time bomb that can blow up the shop. And so hearing this, I create a document “Good approaches to sell time bombs”, and I mention helpful stuff like ensure the timer in your bomb is switched off when it is in shop. I also create an industry wide forum with all time bomb manufacturers and discuss best practices and time bomb methods with them to best sell it in the shop etc.
In spite of all this, there exists an idiot timebomb manufacturer who ignores all best practices, does not consider industry and builds a shitty time bomb that blows up the shop.
Now please educate me, apart from doing the only surefire thing and banning shitty time bomb manufacturers from selling in their shop, what should MS do?
That doesn't seem to be a good faith representation of what the EU was requiring.
> ... so that maybe you might understand something.
It looks like there's literally no getting through to you nor other MS apologists. :( :( :(
sigh
This is and has been a thing for quite some time. Windows is a highly regulated OS.
It doesn't really seem like an example of MS coming up with a better solution then discussing it with industry, unless I'm misunderstanding it?
Instead it seems a lot like MS figuring out a solution that advantages themselves then just rolling it out, at the expense of others. (?)
MS has an entire forum for discussing these things with industry (https://learn.microsoft.com/en-us/defender-xdr/virus-initiat...) and has had variants of said forum for some time (I think the first effort was in 2010).
Kaspersky was running an SSL/TLS Proxy in the kernel IIRC and didn't want to have to move it elsewhere due to the fact it would require them to rework their product quite a bit.
The solutions MS (we) proposed were agnostic and overall better, the anti-malware industry simply doesn't want to make the changes as these things do impose technical work on existing products.
Was the drive for this industry forum coming from dealing with the EU, or was it more from MS trying to make things better without needing the prodding?
I do not know enough to properly answer on the concrete reasons why, only that it was external. Sorry.
(Your comment would be fine without that first bit.)
https://news.ycombinator.com/newsguidelines.html
However, think of it this way: Windows restarts, tries to load with new patch and crashes.
Question: why can't Windows be designed so that on crash it automatically restarts and loads the previous state sans patch?
Answer: Windows could be designed that way but it would require Microsoft to do many things it doesn't want to do. Some of which would require Microsoft to go back to the beginning and reengineer quarter-century or more old code from scratch, that means redesigning APIs and the underlying architecture from first principles.
Why doesn't Microsoft want to do this? It's obvious so I won't bother to spell it out.
Nevertheless, when the dust fully settles and someone outlines these alternative design strategies in great detail then it'll be obvious to everyone what a fragile stack of cards Windows has been constructed on.
If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.
Isn't the point of userland that you can (try to) do anything from there?
It seems like MacOS and Linux provide substantially safer alternatives that are still performant?
> As for restricting kernelland, EU competition regulators would not be happy
I keep seeing people say this. Is there a basis for that assertion, or is that mere speculation? Again, hasn't MacOS already deprecated kexts?
Via Google: https://www.techtarget.com/searchsecurity/news/450420491/Mic...
(Also via myself, as I was at MS when we wanted to make this change and the EU said no.)
I think that would have been a proactive approach. That said, I'm not entirely convinced that the EU was right to place the restriction in the first place.
I think a proactive approach might have been for Microsoft to provide safer interfaces with the kernel, and then use those in its own security offerings.
That said, it does sound like EU competition regulation was a contributing factor here, and I think the EU is wrong on this one and that an OS vendor should not be required to provide unrestricted kernel access to allow security software vendors to compete.
Mostly unrelated, it seems somewhat interesting that this was Kaspersky insisting on kernel access... The US government seems convinced they are compromised.
[0]: https://www.ft.com/content/60dde560-194a-40d1-8c98-1d96d6d01...
They didn’t have much of a choice - it is very hard to get adequate performance with real-time filesystem filtering without doing it in kernel mode. Not aware of any other mainstream OS which succeeds at that.
And they kind of had to provide this feature, since they’ve supported it since forever (antivirus vendors were already doing it back in the days of MS-DOS and Windows 3.x/9x/Me), and there is a lot of market demand for it. It is easy for Linux to say “no” when it never has had support for it (in official kernels)
But, as the blog post points out, it sounds like CrowdStrike is doing a lot of stuff in kernel mode that could be done in user mode instead - whether due to laziness or lack of investment or lack of sophistication of their product architects
> they at least _allowed_ it to happen every time
Microsoft, in allowing third party code to be loaded into their kernel, is no different from other major OS kernels, such as Linux or Apple XNU.
Apple is (increasingly) the most restrictive about this, and a lot of people criticise them for it.
Even Linux imposes some restrictions-which kernel symbols to export (at all or as GPL-only)—although of course being open source, you can circumvent all restrictions by changing the code and recompiling
Whereas, from what I hear, L4 and its derivatives have solved this problem in a way that Mach/Minix/etc could not. Yet still, it makes me wonder, if L4 has really solved it, why aren’t we all running L4? L4 has had some success in embedded applications (such as mobile basebands, Apple Secure Enclave); but as a general purpose operating system has never really taken off.
Whereas this Windows outage rather obviously was.
eBPF being able to crash the kernel is usually sign of a kernel bug. And it sounds like in this case it was even a bug specific to Red Hat kernels, introduced by a Red Hat patch.
That said, even if they are triggering a Red Hat kernel bug, CrowdStrike should be testing their software adequately enough to pick up that issue before customers do – and it sounds like they haven't been
or at least provided some level of protection against crashes in third party kernel code.
drivers and kernel binaries are typically installed and maintained by user space programs that run with some sort of elevated privileges.
"kernel space" is just a runtime context, what gets loaded into there typically comes ordinary (protected) files on the disk.
The OS loads file A into the kernel. It crashes. It reboots. It decides not to load file A this time.
Wow, it's a rollback of kernel-space code.
Unless your argument is that you can't guarantee a rollback of every possible kernel driver, because it might have installed a rootkit while it had full control? Okay, cool, but this isn't a malware removal idea. It's an idea for normal drivers.
Kind of weird that anyone is blaming Microsoft for any part of this, imo
It got there because a user or administrator approved and installed it. It didn't just appear there, Microsoft didn't install it there. The user ran it.
More like:
If you install a security product that then prevents your car from starting; are they entirely blameless for letting you install it?
If you pull the hood up, tear off the “voids warranty” seal, ignore the “don’t open this” labels, crack the seals open and shove something into the engine… sure.
…but if you just slap a widget with the “vendor approved” sticker on your dash and it bricks your car; that’s a bit sucky right?
I do feel Microsoft is not entirely blameless in this.
It should be easier to recover from this kind of thing.
They should have been paying attention and made a fuss that one of the biggest security vendors has been doing this literally since they started.
I would bet money that until two weeks ago Microsoft was high-5ing them for best security practices.
It’s not “their fault” but they can’t just go “wasn’t us!”.
It was them.
It wasn’t macOS. It wasn’t *nix.
Suck it up. They should’ve done better.
Before Microsoft comes into the picture the issues is crowdstrike pushing updates without proper testing, selling a product on which customers cannot control the update schedule, and customers for being so naives and not checking what the product they install on critical stuff do.
Basically IOS which is so locked you can't even run apps not expressively approved by Apple.
Pick one. If I build a bike and you remove the breaks to save weight don't get mad at me when you crash.
All of these parties could do better (stupid tree!). But the real problem is the valet.
We can say that it is obvious that the electronics-heavy cars of today should anticipate rogue valets and build in protections. But we shouldn't let rogue valets off the hook for damages.
As a consumer, you could choose to only purchase cars that have "valet mode". So should we blame consumers who don't? If so, we should blame the airlines, hospitals, etc.--not Microsoft.
How about we prosecute valets unless they refuse to park cars that don't have "valet mode"?
Microsoft took no steps to force-eject them from their ecosystem, despite their long history of issues.
wonderful world.
If a Caesars Entertainment property in Macao has enough incidents, should GM update the firmware on their automobiles to force-eject valets at Caesars Entertainment properties in Las Vegas?
Now imagine that GM actually operates valet services in Macao and Las Vegas. Should they be allowed to force-eject valets from competing services?
I am not a Microsoft apologist. I think they should do better. I think Linux and FreeBSD should do better. I personally avoid Microsoft products. But I place more blame on people who use MS products than I do on MS. After all, I never intend to hand my beat up old Corolla over to a valet so why should I have to pay for a "valet mode" feature that Toyota is forced to build into all their cars? Isn't it reasonable that motorcycles, 18-passenger vans, and scooters don't need "valet mode"?
In my book, the auto manufacturer is lower on the list of culprits than the valet, "the establishment that keeps a valet with an abominable record on staff", and the vehicle owner. But some place like Car and Driver could definitely prioritize encouraging GM or Toyota to develop valet modes over berating owners; so I don't mind a place like HN shooting a few arrows at MS. Unless the general public follows their lead and lets bad guys off the hook by shifting too much focus to somebody lower on the list.
Not OP, but I think the analogy here is the hotel "fore-ejecting" (firing) the valet with a history of doing joy rides. That seems very reasonable.
Also the restaurant has their own entrance without security and questionable people are entering regularly, and they are sneaking into the hotel rooms and stealing some items, breaking the elevator.
At the same time, the hotel is in a litigation process with the restaurants association, because in the past they did not allow any restaurant on their premises. The guests, naturally, do not care about this, since their valuables have been stolen, and they have food poisoning. The reputation of the hotel is tarnished.
I don't think this works since Microsoft isn't the hotel. The hotel in your example chooses which restaurants are inside, but Microsoft doesn't. In this example, Microsoft is the builder who built the hotel building for a 3rd party. That 3rd party decides which restaurants it wants to partner with, as well as any other rules about what goes on in the building.
If the builder came around and made changes to ban the 3rd party's restaurant partner, that would cause a ton of issues and maybe get the builder sued.
Microsoft can't decide what can and can't run on their platform - the most they can do is offer certification which can't catch everything, as we just saw with Crowdstrike since they decided to take a shortcut with how they ship updates. Microsoft also had to allow for equal API access so they don't get sued by the EU.
Again the reasoning of allowing equal API access to avoid getting sued is a false dichotomy: Microsoft could choose to make an OS that would not need such mechanisms to be simply usable.
They could also remove their own crowdstrike-alike offering, so that it would not be considered anti-competitive. They could also choose not to operate in EU. Of course, that would lower their profits, which is the real motive here.
Once you sum it up the reasoning goes: hospitals/flights can stop working because a company cannot lower its profits, and said company is not to blame at all. It is clearly false, the rest is sophism, and back-bending arguments IMO.
So the hotel can have an infinite number of restaurants which can move and move out as they please with not input from the hotel itself?
Please note, that in my analogy the hotel has input in which restaurant is allowed (opposite of your scenario). There are also not infinite Crowdstrike-like offerings, only a few. Same thing applies to the hotel, yes, only limited by the surface of the building and cultural norms.
I any case, the analogy cannot please everyone, and I can see how there are some errors with it in some aspects. In others, I consider it accurate. Using an analogy is an invitation to nitpick on it, so it is my fault really, but I could not resist.
There are other points in the analogy that I feel reflect very well how ridiculous it is to claim Microsoft has no responsibility whatsoever. IMO they do have at least partial responsibility. One cannot simply excuse them "because EU".
In reality this doesn’t seem to be the case at all.
Or in Microsoft's case, via regulatory, social, or software, prevent Crowdstrike from causing harm to their customers.
I'm aware it's a sticky regulatory situation, but CS has a history of these failings and the potential damage could be severe. Despite this, no effort (that I am aware of) was made by Microsoft to inform customers that Crowdstrike introduced potential risks, nor to inform regulators, nor to remove the APIs CS depends on.
I don't believe Microsoft is solely responsible, but I do believe that throwing all of the blame for the very real harm that was caused onto CS alone is missing a piece of the puzzle.
Last aside, every large corp has team(s) focused on risk. There's approximately zero chance they didn't discuss CS at some point. The only way this would not have happened is negligence.
I’m pretty sure anti trust law doesn’t allow Microsoft to go anywhere near that kind of action, even if they wanted to be more Apple like.
No, the operating system is supposed to provide secure access to hardware and isolate independent subsystems so they can't interfere with each other. That's its whole purpose for existing. The fact that people feel they need to deploy CS is a Microsoft failure. Windows is just not a secure OS.
They don't need to deploy shit. Only reason it's deployed because it's a whole racket.
I get that you hate Microsoft, but not everything is their fault and it’s disingenuous to pretend otherwise.
> ing. The fact that people feel they need to deploy CS is a Microsoft failure.
CS is also available and widely deployed on Mac and Linux. Is that a failure of Apple and all the distros? It literally took down Debian and Red Hat systems earlier this year, is that also not CS’s fault?
I don't.
> CS is also available and widely deployed on Mac and Linux. Is that a failure of Apple and all the distros
Yes. All widely deployed commodity operating systems have terrible security designs. None of them have access control systems that enable the principle least privilege, let alone encourage or prioritize it, and none of them are written in robust languages that make verification of safety or security properties possible. Microsoft has made some headway on partial verification, but it's a far cry from what's needed.
What, exactly, is your solution then? To never use a computer again? Because that's certainly what it sounds like.
Not the best analogy. The organization who deploys said software is responsible for the uptime of their systems. They didn't have to use CrowdStrike and if they do they should have a plan in the event of failure.
Are their lawyers telling them they can't impede CrowdStrike even though CrowdStrike is breaking Microsoft's product? They should do it anyway and dare CS to take it to court so they can publicly humiliate CS by dragging all the dirty details of their incompetence out.
At this point they are a threat actor. if you havent kicked their amateur-hour software out of your infrastructure by now, chances are good senior management and engineering have at least considered it formally.
https://en.wikipedia.org/wiki/CrowdStrike#Severe_outage_inci...
Unfortunately, the executive disconnect isn't new. It's actually uncommon that they care about the reality for end users and customers (which is antithical to my entire ethos, hence why I get paid the medium bucks). Why bother waking up and going to work everyday unless you are contributing in some way to sustaining a better future for everyone? It's actually great for marketing and it's already going to be a tough 100+ years from today for our children, even with our collective care.
P.s. People can be so selfish, it kind of breaks my brain but not really. Have you seen the CO2 emissions visualization from NASA this week? It was a wakeup call for me.
'Tremendous' NASA Video Shows CO2 Spewing from US into Earth's Atmosphere https://www.newsweek.com/nasa-video-carbon-dioxide-co2-emiss...
It's concerning.. and caught no traction.. http://news.ycombinator.com/item?id=41064029
“ For those who don't remember, in 2010, McAfee had a colossal glitch with Windows XP that took down a good part of the internet. The man who was McAfee's CTO at that time is now the CEO of Crowdstrike. The McAfee incident cost the company so much they ended up selling to Intel.”
so yeah, “leadership” (and that’s a loose term) doesn’t seem supremely concerned about much more than earnings
https://archive.is/20240724213623/https://www.barrons.com/ar...
Also what the fuck is a sales-facing CTO??
https://www.businessinsider.com/john-mcafee-tweet-said-his-s...
Recently ordered an HP laptop for some light work (not my startup), and when placing the order said don't include McAfee, that "I don't trust them", all just from some odor!
CloudStrike runs in kernel mode? No wonder there are problems; kernel mode sounds like more of a threat than a protection.
Sooooo, for my Web server(s), McAfee and CloudStrike are issues I get to ignore. Problems avoided and time, money, energy saved!! Simple.
I’m on #2 now and it’s been great. It’s like a breakup. “What was I thinking?”
Of course if it is representing your values and your values are purely mercenary, it’s really not going to change anything.
Is that what happens when a company has so many Sales Engineers that they become a parallel department from regular Engineering?
The product itself is a secondary cost-center, probably less important than even accounting.
Perhaps of a symptom of the "Everyone is in sales" brain damage so pervasive in companies now.
IMO somebody who managed to collapse the most important infrastructure on earth twice in as many decades - not a small feat, I have to admit - should be known by name to the general public, lest he'll get another chance at it.
And outages were not as global as news outlets made it look to be. Crowdstrike may have been ubiquitous in some countries, but almost absent in others. And still, crowdstrike or windows windows aren't global pieces of infrastructure.
Presumably it doesn't matter that much and isn't worth spending money/manpower on?
If the usefulness/quality of their software has no influence on their potential customers decision making process. why bother?
It would make much more sense to allocate any excess resources to the departments that do actually matter like sales and marketing.
Well, if they think any of the $20B of shareholder value lost recently has to do with the quality issues... Then perhaps they should reconsider. (keep in mind marketcap also represents their ability to raise capital in the future with more/less dillution)
Management often sees, “I have a dozen people on this.” When in fact the bus number was three, you laid one off, another quit and the third is sick or having life struggles.
> A Hacker News user claimed that
Nice to see Wikipedia has devolved even further into a dumpster fire in that they are now citing random HN posts as authoritative sources of facts.
If you disagree with this choice of source, you can flag this part as needing better sources. The simplest way to do so is to just leave a comment on the talk page.
Yeah, but doesn't MS have to sign every kernel mode driver? They've allowed Crowdstrike's foot gun to continue to live in the kernel.
Not a CS fanboy, but just wanted to suggest an alternative to sheer incompetence
1. They run inside a kernel.
2. They have many clients. (Incl. hospitals?)
This implies serious severity and impact of potential incidents, so in turn it should imply certain level of quality assurance process.
As we witnessed, it has glaring holes.
They know that mistakes can take out thousands and thousands of devices, therefore it is imperative they prioritize stability over rollout speed. They have more direct access to devices, than any 0-day would ever have, therefore there is a significant risk that they do more damage with an update than any 0-day ever could.
You have to remember, a 0-day could come to existence that threatens every system, but that usually happens once every couple of years (last one of that category was probably Blaster/I LOVE YOU). But CS risks of damaging the system every couple of hours with an update. Therefore, it should be tested to make absolutely sure it doesn't cause crashes.
IMHO, it was sheer negligence and incompetence.
https://youtu.be/6m_Im7J9Iaw?si=q8jLBefEdgm-PrrZ
How would Microsoft apply pressure? Short of publicly shaming them what power do they have?
It’s not that easy.
Microsoft has (somewhat correctly IMNSHO) pointed at the EU agreement that forced them to open the kernel up to third parties as being a factor in the CrowdStrike catastrophe. <https://www.theregister.com/2024/07/22/windows_crowdstrike_k...>
> However, nothing in that undertaking would have prevented Microsoft from creating an out-of-kernel API for it and other security vendors to use. Instead, CrowdStrike and its ilk run at a low enough level in the kernel to maximize visibility for anti-malware purposes. The flip side is this can cause mayhem should something go wrong.
> The Register asked Microsoft if the position reported by the Wall Street Journal was still the IT titan's stance on why a CrowdStrike update for Windows could cause the chaos it did. Redmond has yet to respond.
How would a better designed Windows eliminate the business & compliance need for installing software like CS? And why hasn't that already happened?
I would think Microsoft and CS' customers have an incentive to not have such third party software on their system if possible.
It hasn't happened because the market incentives for existing, widely deployed software are too great. Just patch and release is cheaper than starting from an actually secure foundation.
The reality is that this experience has been built into working for Microsoft since at least the 90s. The entire point of the compatibility effort they put into Windows 95 and later came from people blaming a Windows upgrade for breaking software that barely worked in the first place.
Windows Vista was known for lots of BSODs, but at least 60% of them were entirely due to nVidia GPU drivers crashing.
But they didn’t.
And Microsoft, I argue, also has blood on their hands for every hospital this hit. Giving users a prompt to disable the driver, after three successive failed boots, would have saved lives.
“CSAgent.sys has caused a failure to boot three times in a row. Do you want to disable this driver? <Yes> <No>.”
You click “Yes.” Server reboots with CloudStrike driver disabled. The day is saved in 5 minutes instead of building a custom ISO image or going on a BitLocker key recovery spree.
For most users, they’ll try clicking “Yes.” And then it’s back to work. After all, “No” just causes a blue screen again, might as well try the other path.
This would have been the difference between the IT department handling 10,000+ calls or a few hundred (plus sending out a bulletin) in many, many organizations. It also could have saved billions at this point.
Heck, it would have saved lives in hospitals.
Wouldn’t malware just use that as an attack vector?
Malware can’t crash a well-written or memory-safe driver, so it will never be unloaded. Problem solved.
The issues seem to be operational not technical.
CrowdStrike isn't special because it's security software, but there are good reasons you might not want to have the OS start disabling drivers randomly when the system isn't in a stable state.
One reason is partly because, depending on what happened when, you might disable the wrong driver. It so happens in this case that CrowdStrike's hand was caught in the cookie jar immediately because it did something blindingly obvious to the kernel core as wrong, but there are scenarios where heap or other memory corruption happens in your CD player driver and 5 minutes later when you start using the network, it crashes because the network driver was executing off in la la land.
In addition, sometimes it's just hardware failure that is a proximate cause, rather than a bad driver (I've seen bugchecks attributed to NTFS.sys that were ultimately the result of a dying hard disk).
If you're blindly installing software system wide, that has kernel access no less, and not accounting for failure of that software in your risk analysis then you are to blame more so than the vendor.
Certainly I expect some SLA is in place but that's only of monetary benefit and irrelevant to keeping critical infra online.
Yes, absolutely. It's a clever idea.
But do I think Microsoft was negligent in not building that?
No, I think that's going too far. Windows already has Safe Mode -- as you note -- to allow for manual recovery, which is what people are using.
I don't think it makes sense for it to be Microsoft's legal responsibility to protect its users from software with a critical bug that wasn't written by Microsoft. Otherwise, where would it end? If a third-party program tries to delete all your user data, is it Microsoft's legal responsibility to check whenever a process is deleting a lot of data, and intervene with a confirmation dialog? Is it Microsoft's responsibility to protect you from all malware and ransomware, no matter how cleverly written? Is it Microsoft's responsibility to constantly cache program state on disk so that when a third-party program crashes, you don't lose your data since your last save?
I think that's going too far, in terms of legal obligation.
Windows has a dated design and a security model no longer fit for purpose. As for your other example, it could be protecting users from malicious programs that may delete data, simply by having a better security model, like Android and iOS.
Somebody bought Windows, and bought CrowdStrike. CrowdStrike is negligent, and possibly also the person/org who chose to rely on Windows+CrowdStrike without a backup plan if that resulted in further damages to others.
Third party vendors are absolutely not "forced into writing unsafe kernel drivers". They can properly test things to write safer code (which CrowdStrike infamously didn't). And kernel mode is fundamentally required for security software like this, as far as I understand.
And using app-based mobile OS's is not necessarily a useful comparison point. They are limited in all sorts of ways that desktop OS's are not -- and don't you hear people here on HN constantly complaining about that? A better comparison point is macOS and Linux. CrowdStrike also crashed Linux, and macOS still lets you bypass SIP if you want to.
> And kernel mode is fundamentally required for security software like this, as far as I understand.
These are conflicting points. They cannot both be true.
First Windows already does something similar. After 3 it is supposed to boot into WindowsRE which gives you options to revert to a previous version, uninstall updates, and I believe also reverts configurations like recent driver installations.
The problem here though, CrowdStrike itself didn’t update. It updated a definition file (last I saw at least) and that likely would not have been caught by Windows as a new version.
Also frankly, not super thrilled at the idea of Windows just deciding to disable/uninstall something except for rolling back (so a previously working config) due to how things could interact. This situation could have been far worse and harder to recover from.
In this case maybe Windows could have noticed that the configuration update is what was causing it and rolled that back, but it’s possible it would have just re-downloaded the file when it started back up anyways.
Regarding saved lives, do we actually know that anyone’s lives were lost due to this? My local hospitals were still performing emergency surgery.
The problem is that CrowdStrike doesn't want to let you start the computer without it running. It's the reason why it's an ELAM driver - it's marked as required for boot, so Windows won't try to boot without it, much like it won't if you remove crucial hardware drivers. I guess what they are trying to avoid is malware crashing the kernel driver which then gets disabled letting the malware roam free, without realizing that the cure is worse than the disease.
If you want to decide which OS/distros to avoid for critical stuff, look to see who's learning from the incident (even if not bitten by it) compared to those saying "it wasn't our fault" (and that's not just MS).
https://www.crowdstrike.com/compare/crowdstrike-vs-microsoft...
That is, there was a post-test pre-distribution packaging stage, and that's where the distributed file(s) got f'ed up.
If true that would explain how it got past their testing, but would also be an incredible lack of competence IMHO.
But yeah, curious if there's been some more concrete details there.
Sounds like one of those companies where you get hired and are shocked by the sausage factory you just stepped into
Named pipes are pipes of communication that processes can use to talk to each other, as an alternative to sockets.
For example Chrome uses them between the user interface and the actual page renderer.
In March they tested it in staging, said it was fine, pushed to prod with few rules in April, still looked fine.
In July they added a new rule, which was deployed to 100% immediately, as from their perspective, a new entry in a database definition doesn't need testing nor canary deploy
(which is still irresponsible, because bad rules could cause damage as well like any security/antivirus software, even if the parser didn't crash, but it could have blocked legitimate actions or files)
(obligatory "Parse, don't validate" post
https://news.ycombinator.com/item?id=35053118
and some more discussion on the topic
https://news.ycombinator.com/item?id=39322551 )
Their language for describing this design is obtuse and confusing.
?!?!
That's the sort of thing a negligence lawyer focuses on. Partner at Brown Rudrick: "The most likely legal theory will be one of negligence. [Congress] will drag the guy over the coals, they'll maybe implicate him and his company and put in place a negligence action. There'll maybe be a couple of plaintiffs lawyers who dig up some exceptional theory on negligence, and get some class action lawsuits going. Again, we still don't know all the facts in this case, and there are other dimensions which have not yet been fully explored, including how CrowdStrike had access to kernel level updates on the Microsoft operating system? How come Microsoft didn't have any control over these updates being pushed on their kernel?"
The first two class actions are already starting.
[1] https://learn.microsoft.com/en-us/windows-hardware/drivers/d...
[2] https://www.channele2e.com/analysis/crowdstrike-legal-and-li...
https://www.youtube.com/watch?v=ZHrayP-Y71Q https://www.youtube.com/watch?v=wAzEJxOo1ts
That verification is for interactions with the OS. Its not going to catch driver specific exceptions.
[1] https://learn.microsoft.com/en-us/windows-hardware/drivers/d...
https://www.neowin.net/news/microsoft-points-finger-at-the-e...
Does it matter? Ultimately, the impact to the citizenry is what matters.
Fragmentation makes consistent governance/security impossible, but the heterogeneity also limits the scope of incidents. Apple balances its greater monoculture risk with deep control of the underlying hardware, to where as a user or developer you're only ever interacting with a very, very high level abstraction.
Is this true though? They've released a Post Incident article:
https://www.crowdstrike.com/blog/falcon-content-update-preli...
in which they state:
> How Do We Prevent This From Happening Again? Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment.
So it seems, if I understand this correctly, they've just implemented the rate limiter as a response to this incident.
You are the clown's of the world, that's all ... xD
a simple plastic covering of your new dyson has more legal scrutiny and action (see the "children may choke" warnings they all need to come with) than software that we otherwise block in the name of "national security".
given how much overvalued tech companies are in this region, i believe it is high time to start legally recognizing the real-life impact of digital tech. to hell with the "but muh innovation" argument.