Is anyone really gullible enough to blame the EU for CrowdStrike rolling out a faulty update? What is Microsoft PR thinking? I also find it amusing that George Kurtz (CrowdStrike CEO) was perfectly capable of causing a global IT disaster in 2010 as McAfee CTO without kernel access (the update just deleted critical Windows XP system files).
That's if allowed to continue booting, the correct procure would be on fault, reload without the patch. There are additional measures too that Microsoft could have adopted but this would have required the rewriting of code that goes back to Windows 2000 or earlier.
But that would cost and thus eat into MS's profits.
Oh sorry, I forgot doing that would be the ultimate sin.
This 100%. I could see a fault tolerance threshold where a module has to fail say three times in a row before bring disbled, but the OS should be able to handle failing third party kernel modules.
I don't find that particularly credible. If Microsoft would boot all security products (including their own) out of the kernel and force them to use specific interfaces for this there is no anti-trust concern at all there. It would only be a problem if they wanted Windows Defender to remain having privileged access while taking it away from everyone else.
The true reason is a regulatory environment where snake-oil peddling corporations lobby for governments to mandate the use of their crapware in the name of security and reliability.
This causes senior management to push for the installation of this crapware/malware on their systems, overriding the objection of their system administrators who know better.
Senior management want to cover their asses, and the administrators shrug their shoulders and respond "if you say so".
In sum it is mostly a regulatory racket that is profitable for the peddlers of this crapware and for management who can claim that they did what they were required to do to stop shit from happening. Everybody wins except shareholders, customers and IT staff who have to fix this mess without getting the righteously justified overtime and bonuses.
Why do you think insurance companies have backing out of the business of insuring against these kind of disasters? They've clearly learned better.
Why do you think I stopped using Norton, MacAfee, etc, etc, and etc and opted for just decent backups and Windows own built-in stuff?
The headline, as always, is disingenuous. They were asked why they couldn't lock third parties out of this level of unprotected system access, and said that the 2009 ruling prevented them from doing so. Which is simply factually correct.
While that is correct they could at any point in the last 15 years have implemented an API for security software which does not require to load executable code into kernel space. Kind of like Apple did: https://developer.apple.com/documentation/endpointsecurity
Then Microsoft would have proceeded to only use this documented user land API themselves for their own Defender product and thus have no undocumented API or access advantage over other security software. The EU ruling only cares about a level playing field and not about the implementation details.
I am reasonably confident that if Microsoft publically communicated that CrowdStrike breaks their recommendations and decreases security of their systems, CrowdStrike would have way fewer customers
Microsoft has a very solid point here. MS has wanted to kick AV vendors out of kernel space for a long time because it isn't necessary, and can lead to the type of incident we are talking about here.
MS provides a userspace interface[0] for AV vendors to do what they need to do, but they can't be forced to use it.
So yes, due to EU regulations, AV vendors can still play in kernel space, and can bring much of the world to a halt when they make a mistake as a result.
If they provide a userspace interface but use their own set of kernel accesses for Windows Defender then they are not competing fairly, don't you think? They just need to make Windows Defender use those userspace interfaces as well and all is good :)
The real cause of mistake was a single point of failure where there was no A/B testing, no gradual deployment nothing. To say that banning AV from kernel would prevent this is not just hilarious but disingenuous and shows complete lack of knowledge of operations and deployment. There's no golden rule which says that Windows cannot make these errors.
That is a solution, but not the cause. The cause is not having a culture that evaluates failure scenarios. From what I have read:
* Updates are not vetted or sanity checked.
* Updates are not slow-rolled to production.
* Updates are not signed to prevent corruption or alteration.
* Updater does not sanitize or validate inputs.
* Updater does not have a reversion process to previously known good position on faulty boot.
* Updater should mark itself as Unnecessary For Boot on faulty boot at some point.
Finally, its high adoption means it creates a mono-culture. There should be another version built independently where one is running on a machine and another sits in a ready state. If there is a fault in one, it becomes disabled and the second takes over. Good ol' NASA style redundancy.
- could have kicked AV vendors out of kernel space, including themselves, ensuring level playing field (which is the point of EU regulations). But then they couldn't sell their product that's "isn't necessary".
- could have created other, less critical APIs to use for everyone
- could have enabled anything mandated by the EU only for the EU market
or, the users could kick CS and McAfee out of the kernel space themselves - by not using those products or features that require these dodgy kernel modules.
the CTOs and engineering staff are the ones in control of their machines, not M$. or at least they should be. problem is that they thought they were solving a problem by installing this kind of software, but instead were simply handing over their responsibility to a 3rd party that was totally irresponsible - and certainly doesn't accept it. they took a compliance short-cut with "box checking" software. that's where the problem lies - lack of responsibility and engineering rigor in the IT orgs.
If microsoft is content with being out of kernel space why don't they just do that and compete with their own AV on the same API? Oh, they want kernel level access with their AV and want nobody else to have it? Oh. Maybe no, they don't have a solid point.
Oh yeah, if Microsoft was so worried about other companies tinkering with its kernel then why didn't it introduce routines that would ensure that a reboot would actually occur on a boot load error? (Upon error, a reload would then omit the faulty code as well as tell users there was a problem with the update.)
I'd suggest that there is no reason a BSOD—Blue Screen of Death—should ever occur on a system that was already working as the OS should be constructed in such a way that it can undo a faulty patch. As you'd know, there is already such a thing as Volume Shadow Copy, VSS, in MS Windows. Microsoft could have adopted this and similar techniques to ensure that the system either stayed up or rebooted.
Yes, I can hear Microsoft's retort now that doing that would make Windows more vulnerable to viruses, infiltration, etc.
To that I'd say utter bullshit, the real problem—as it has always been with Microsoft—is that it doesn't properly finish or bootstrap its code against errors before it releases it to the public. Microsoft is thus doing cheapskate engineering as it's much more profitable.
Hopefully, eventually regulators will require hardening of such software together with guarantees against such faults—guarantees that if not honored would result in enforceable financial penalties.
Only loss of income/profit is likely to fix this problem.
EU, for everyone's sake quickly debunk that deliberately misleading PR crap from Microsoft before it takes hold.
32 comments
[ 4.7 ms ] story [ 87.6 ms ] thread"Just regulate us less, seriously!".
Their solution to this situation is quite literally monopoly, which is hilarious.
But that would cost and thus eat into MS's profits.
Oh sorry, I forgot doing that would be the ultimate sin.
This causes senior management to push for the installation of this crapware/malware on their systems, overriding the objection of their system administrators who know better.
Senior management want to cover their asses, and the administrators shrug their shoulders and respond "if you say so".
In sum it is mostly a regulatory racket that is profitable for the peddlers of this crapware and for management who can claim that they did what they were required to do to stop shit from happening. Everybody wins except shareholders, customers and IT staff who have to fix this mess without getting the righteously justified overtime and bonuses.
Why do you think insurance companies have backing out of the business of insuring against these kind of disasters? They've clearly learned better.
Why do you think I stopped using Norton, MacAfee, etc, etc, and etc and opted for just decent backups and Windows own built-in stuff?
Then Microsoft would have proceeded to only use this documented user land API themselves for their own Defender product and thus have no undocumented API or access advantage over other security software. The EU ruling only cares about a level playing field and not about the implementation details.
MS provides a userspace interface[0] for AV vendors to do what they need to do, but they can't be forced to use it.
So yes, due to EU regulations, AV vendors can still play in kernel space, and can bring much of the world to a halt when they make a mistake as a result.
[0] https://learn.microsoft.com/en-us/windows/win32/amsi/antimal...
Precisely the point I made in my comment. If Windows can initiate a BSOD then it can also initiate a reboot without said patch.
What Microsoft's PR department said is personified bullshit and needs debunking ASAP.
- could have kicked AV vendors out of kernel space, including themselves, ensuring level playing field (which is the point of EU regulations). But then they couldn't sell their product that's "isn't necessary".
- could have created other, less critical APIs to use for everyone
- could have enabled anything mandated by the EU only for the EU market
the CTOs and engineering staff are the ones in control of their machines, not M$. or at least they should be. problem is that they thought they were solving a problem by installing this kind of software, but instead were simply handing over their responsibility to a 3rd party that was totally irresponsible - and certainly doesn't accept it. they took a compliance short-cut with "box checking" software. that's where the problem lies - lack of responsibility and engineering rigor in the IT orgs.
I'd suggest that there is no reason a BSOD—Blue Screen of Death—should ever occur on a system that was already working as the OS should be constructed in such a way that it can undo a faulty patch. As you'd know, there is already such a thing as Volume Shadow Copy, VSS, in MS Windows. Microsoft could have adopted this and similar techniques to ensure that the system either stayed up or rebooted.
Yes, I can hear Microsoft's retort now that doing that would make Windows more vulnerable to viruses, infiltration, etc.
To that I'd say utter bullshit, the real problem—as it has always been with Microsoft—is that it doesn't properly finish or bootstrap its code against errors before it releases it to the public. Microsoft is thus doing cheapskate engineering as it's much more profitable.
Hopefully, eventually regulators will require hardening of such software together with guarantees against such faults—guarantees that if not honored would result in enforceable financial penalties.
Only loss of income/profit is likely to fix this problem.
EU, for everyone's sake quickly debunk that deliberately misleading PR crap from Microsoft before it takes hold.