HN Explorer

CrowdStrike Update: Windows Bluescreen and Boot Loops (old.reddit.com)

4489 points by BLKNSLVR ↗ HN
My workplace has a number of people reporting Windows blue-screening and going into a boot loop. The IT Department have a number of servers recently gone offline and have said there's a chance that the two issues are related, and potentially due to a Crowd Strike application update.

My laptop blue-screened and rebooted, but is working fine after the reboot.

A local radio station has also said they've got the same issues with their laptops and their phone system is down as a result.

Not seeing anything on news sites yet. Anyone else seeing similar?

Above is all based in Australia.

4,017 comments

[ 3.8 ms ] story [ 567 ms ] thread
sammy2255
Yep happened to us too. Its global. And it just started happening.
VargaLand
Same in Adelaide. Reports coming in from Gov agencies, utilities etc.
scriptsmith
It's crowdstrike: https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e...

> 7/18/24 10:20PT - Hello everyone - We have widespread reports of BSODs on windows hosts, occurring on multiple sensor versions. Investigating cause. TA will be published shortly. Pinned thread.

> SCOPE: EU-1, US-1, US-2 and US-GOV-1

> Edit 10:36PT - TA posted: https://supportportal.crowdstrike.com/s/article/Tech-Alert-W...

> Edit 11:27 PM PT:

> Workaround Steps:

> Boot Windows into Safe Mode or the Windows Recovery Environment

> Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

> Locate the file matching “C-00000291*.sys”, and delete it.

> Boot the host normally.

ineptech
Can you post a summary? We're affected but I don't have access to that portal.
scriptsmith
Someone posted this in the thread, but I also can't log in to verify

> Summary

> CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

> Details

> Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.

> Current Action

> Our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket.

> Status updates will be posted below as we have more information to share, including when the issue is resolved.

> Latest Updates

> 2024-07-19 05:30 AM UTC | Tech Alert Published.

> Support

> Find answers and contact Support with our Support Portal

nofinator
They've bumped this support info to a blog post that's linked from their home page: https://www.crowdstrike.com/blog/statement-on-falcon-content...

It includes PDFs of some relevant support pages that someone printed with their browser 5 hours ago. That's probably the right thing to do in such a situation to get this kind of info publicly available ASAP, but still, oof. Looks like lots of people in the Reddit thread had trouble accessing the support info behind the login screen.

unixhero
"Start your free trial now." Hahahahah you have got to ne kidding me :)
Neil44
Right after you enter the bit locker recovery key.

You do have your bit locker recovery key, right? .....right?

tamimio
I doubt most of the clients who use CS know what BitLocker is, let alone how to back it up, assuming it wasn’t backed up automatically by Windows.
Fire-Dragon-DoL
Doesn't that get backed up automatically to the Microsoft account?
dist-epoch
That's opt-in.

In Enterprise setups the key should be backed somewhere in Active Directory.

mrhhaacckk
Yes you should be able to pull it from your domain controllers. Unless they're also down, which they're likely to be seeing as Tier 0 assets are most likely to have crowdstrike on them. So you're now in a catch 22.
mavhc
Log into hypervisor, rollback VM
olyjohn
Microsoft shops gonna be running Hyper-V. Probably also got hosed.
tremon
Rolling back an Active Directory server is a spectacularly bad idea. Better make doubly sure it's not connected to any network before you even attempt to do so.
ryanjshaw
In theory. I've seen it not happen twice. (The worst part is that you can hit the Bitlocker recovery somewhat randomly because of an irrelevant piece of hardware failing, and now you have to rebuild the OS because the recovery key is MIA.)
tamimio
I know it does for personal accounts once linked to your machine. Years ago, I used the enterprise version and it didn’t, probably because it was “assumed” that it should be done with group policies, but that was in 2017.
acdha
Most people installed CrowdStrike because an audit said they needed it. I find it exceedingly unlikely that the same audit did not say they have to enable Bitlocker and backup its keys.
toomuchtodo
I can confirm this. EDR checkbox for CrowdStrike, BitLocker enabled for local disk encryption checkbox. BitLocker backups to Entra because we know reality happens, no checkbox for that.
paganel
This was particularly interesting (from the reddit thread posted above):

> A colleague is dealing with a particularly nasty case. The server storing the BitLocker recovery keys (for thousands of users) is itself BitLocker protected and running CrowdStrike (he says mandates state that all servers must have "encryption at rest").

> His team believes that the recovery key for that server is stored somewhere else, and they may be able to get it back up and running, but they can't access any of the documentation to do so, because everything is down.

radiator
I find that hilarious
lukan
Me too, as I am also not affected. But I do pity those guys who now try to solve that deadlock.
lordnacho
> but they can't access any of the documentation to do so, because everything is down.

One of my biggest frustrations with learning networking was not being able to access the internet. Nowadays you probably have a phone with a browser, but back in the day if you were sitting in a data room and you'd configured stuff wrong, you had a problem.

RobotToaster
Nobody, not one person, thought that documentation should be stored in hard copy?
whynotmaybe
I'm guessing someone somewhere said that "it must be stored in hard copy in a safe" and the answer was in the range of "we don't have a safe, we'll be fine".

Or worse, if it's like where I worked in the past, they're still in the buying process for a safe (started 13 months ago) and the analysts are building up a general plan for the management of the safe combination. They still have to start the discussions with the union to see how they'll adapt the salary for the people that will have to remember the code for the safe and who's gonna be legally responsible for anything that happens to the safe. Last follow-up meeting summary is "everything's going well but we'll have to modify the schedule and postpone the delivery date of a few months, let's say 6 to be safe"

KineticLensman
Not just financial / process barriers. I worked for a company in the early 90's that needed a large secure safe to store classified documents and removable hard drives. A significant part of the delay in getting it was figuring out how to get it into the upstairs office where it would be located. The solution involved removing a window and hiring a crane.

When we later moved to new offices, somebody found a solution that involved a 'stair-walking' device that could supposedly get the safe down to the ground floor. This of course jammed when it was halfway down the stairs. Hilarity ensued.

sobellian
Just hope there is no mutual recursion, i.e. recovery key A is stored on machine B, recovery key B is stored on machine A!
JCharante
Isn’t that what office safes are for? I don’t know the location, but all the old guard at my company knew that room xyz at Company Office A held a safe with printed out recovery keys and the root account credentials. No idea where the key to the safe is or if it’s a keypad lock instead. Almost had to use it one time.
sebazzz
BitLocker for Business stores the bitlocker key centrally. Still, it is a huge manual undertaking fixing every system.
tgshaik
Absolutely correct. Unfortunately, there is no other solution to this issue. If the laptops were powered down overnight, there might be a stroke of luck. However, this will be one of the most challenging recoveries in IT history, making it a highly unpleasant experience.
teeheelol
Yeah in context we have about 1000 remote workers down. We have to call them and talk through each machine because we can't fix them remotely because they are stuck boot looping. A large proportion of these users are non-technical.
chrisjj
How fortunate the phone system is not vulnerable to CrowdStrike...
CoastalCoder
Man, talk about a mass-phishing opportunity.
immibis
I heard the central system was on Azure, running CrowdStrike.
mschuster91
MS Windows Recovery screen (or the OS installer disk) might ask you for the recovery key only, but you can unlock the drive manually with the password as well! I had to do that a week ago after a disk clone gone wrong, so in case someone steps on the same issue (this here is tested with Win 10, but it should be just the same for W11 and Server):

1. Boot the affected machine from the Windows installer disk

2. Use "Repair options"

3. Click through to the option to spawn a shell

4. It will now ask you for unlocking the disk with a recovery key. SKIP THAT.

5. In the shell, type: "manage-bde -unlock C: -Password", enter the password

6. The drive is unlocked, now go and execute whatever recovery you have to do.

Good luck.

sam_lowry_
On my corporate Windows 11 22H2 "manage-bde -unlock C: -Password" does not unlock the disk with the user key. I guess it needs recovery key as well.
plonk
Don’t you need more options if the key is in a TPM, or there is a password but it’s only part of the key?

Can you even get the secret from the TPM in recovery mode?

mschuster91
> Can you even get the secret from the TPM in recovery mode?

Given that you can (relatively trivially) sniff the TPM communication to obtain the key [1], yes it should be possible. Can't verify it though as I've long ago switched to Mac for my primary driver and the old cheesegrater Mac I use as a gaming rig doesn't have a hardware TPM chip.

[1] https://pulsesecurity.co.nz/articles/TPM-sniffing

toast0
TPMs embedded in the processor (fTPM) are pretty popular and it's a lot harder to sniff communications that stay inside the cpu.
plonk
yea I don't need an attack on a weak system, I mean the authorized legal normal way of unlocking BL from Windows when you have the right credentials. Windows might not be able to unlock BitLocker with just your password.

I don't know how common it is to disable TPM-stored keys in companies, but on personal licenses, you need group policy to even allow that.

Although this is moot if Windows recovery mode is accepted as the right system by the TPM. But aren't permissions/privileges a bit neutered in that mode?

sva_
Happy weekend to everyone who works there.
[deleted]
(comment deleted)
2OEH8eoCRo0
They had me at "crowdstrike engineering"

So engineer-like.

jamesbfb
Our company is in panic mode. 15 machines blue screened for no apparent reason and stuck in boot loop. I’m a gloating Linux user :)

Also in Australia

jdleel
I think there are companies with 10000+ or probably more
dabiged
Year of the Linux desktop at last!
VargaLand
According to Reddit It's hitting Croatia, Philippines, US, Germany, Mexico, India, Japan. SAP servers dropping like flies, that's Defence,Banks, Payroll all affected. Major Retail Chains like Big W down.
aenis
We have outages across whole APAC and most EMEA. Despite being a very big client of CS, we do not have an official resolution yet, an hour into the incident.
[deleted]
(comment deleted)
ajdlinux
Starting to get media coverage: https://www.abc.net.au/news/2024-07-19/technology-shutdown-a...
gamma032
Australian news reporting this has hit hospitals, fire and rescue, banking, media, airlines and many other companies worldwide.
yen223
Newsreaders on the ABC are reading off paper notepads lmao
jaza
I was listening to Triple J (one of ABC's radio stations), they said: "welcome to our first and possibly last ever Triple J's USB Fridays, we can't play any of our usual music because the computers are all down, all we can play is the songs that happen to be on the USB stick that one of us had in our pocket". LOL!
thrdbndndn
I assume you have to install "CrowdStrike" yourself (i.e. not bundled with Windows by default)? I have no idea what it is before.
sammy2255
Its paid antivirus software, they cater to businesses
alams
Anyone found any fixes, while Crowdstrike comes up with a fix?
jimmy_lbts
we will found a troubleshooting , we start in revovery mode and rename the path C:\windows\system32\drivers\Crowdstrike
h4kunamata
Do this at your own risk!! Apparently there are two NON-OFFICIAL solutions:

1. Rename csagent.sys ( the file causing the BSOD ) 2. Rename c:\windows\System32\Drivers\Crowdstrike

Again, do this at your own risk. Both workaround have been reported as "working". I am Linux user so I cannot tell.

michelevr
Husband is a deputy in California. His department and many others here are down as well (including PDs, jails, ambulance companies, etc.)
mcast
This seems like a pretty severe point of failure.
aenis
We have ~50 thousand laptops in reboot loop and ~1.5k servers as well. No resolution yet.
[deleted]
(comment deleted)
kswap
Faced the same issue few minutes back after few loops of reboot my system is up
aenis
We had a few machines come out of the boot loop - only to re-enter it 20 mins later. I am sure CS pulled the patch from their CDNs but ...maybe some cached versions still linger?
7am00dee
A system restore helps. But obviously not when you’ve got an environment of ~500 or more clients
zcretu
Quick fix that worked for us, In safe mode:

1.enter in drive C: 2.system 32 folder 3. Drivers 4. Rename crowdstrike folder to something else doesent matter what.

jimbob45
Could you potentially do the same by just attaching the HDD to another computer as a secondary drive and renaming the folder if safe mode falls through?
nrr
Not likely, not unless the disk is also not BitLockered.
_kyran
Was just using the energy vic website and thought I'd been rate limited when their API stopped working. Seems like it could be this.
BLKNSLVR
In terms of analysing risk factors to minimise something like this happening again, what are the factors at play here?

A Crowdstrike update being able to blue-screen Windows Desktops and Servers.

Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system. This makes me think that, maybe, Crowdstrike were unlucky enough to have accidentally discovered a bug that affects multiple versions of Windows (ie. it's a Windows bug, maybe more-so than it is a Crowdstrike bug).

There also seems to have been a ball-dropped in regards to auto-updating all the things. Yes, gotta keep your infrastructure up to date to prevent security incidents, but is this done in test environments before it's put into production?

Un-audited dependence on an increasingly long chain of third-parties.

All the answers are difficult, time consuming, and therefore expensive, and are only useful in times like now. And if everyone else is down, then there's safety in the crowd. Just point at "them too", and stay the path. This isn't a profitable differentiation. But it should be! (raised fists towards the sky).

aloha2436
> an application shouldn't be able to do this kind of damage to an operating system

Antivirus software by its nature probably needs the kind of access that would let it bluescreen your computer.

BLKNSLVR
Wading out my depth here, so forgive any stupidity following.

And there's a certain amount of sense to that, it has to get "under" the layer that viruses can typically get to, but I still think there should be another layer at which the OS is protected from misbehaving anti-virus software (which has been known to happen).

eddythompson80
It's a kernel mode driver. There aren't layers in kernel drivers. Any kernel module/driver can crash your system if it wants to.
quietbritishjim
You're taking about how things are, the comment you're replying to is talking about how things could be. There's not a contradiction there.

Originally, x86 processors had 4 levels of hardware protection, from ring 0 up to ring 3 (if I remember right). The idea was indeed that non-OS drivers could operate at the intermediate levels. But no one used them and they're effectively abandoned now. (There's "level -1" now for hypervisors and maybe other stuff but that's besides the point.)

Whether those x86 were really suitable or not is not exactly important. The point is, it's possible to imagine a world where device drivers could have less than 100% permissions.

lnx01
It runs at Ring 0, there's no lower ring (besides maybe IME and the like).
BLKNSLVR
The problem I have with this is that anti-virus software has never felt like the most reliable, well-written, trustworthy software that's deserving of it's place in Ring 0.

I understand I'm yelling into the storm here, because anti-virus also requires that level of system access due to the nature of what it's trying to detect. But then again, does it only need Ring 0 access for the worst of the worst? Can it run 99% of the time in Ring 1, or user space, and only instantiate it's Ring 0 privileges for regular but infrequent scans or if it detects something else may be 'off'?

Default Ring 0? Earn it.

This turns into a "what's your threat level" discussion.

DaoVeles
Need something like a hypervisor OS/hardware that isnt IME.
dist-epoch
Modern Windows installs already run under a hypervisor. It's called Core Isolation or Virtualization Based Security.
prmoustache
That usually makes it a port of entry for attacks. Antivirus are really malwares waiting to be exploited.
fulafel
This is not the case. There are many possible AV architectures, with or without kernel drivers and/or administrator level permissions.
zsims
> Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system.

It doesn't operate in user space, they install a kernel driver.

Reason077
> "they install a kernel driver"

And therein lies the problem!

Tech-Untangler
Not like they have an option. Kernel drivers are required.
viraptor
There's a grey area between "kernel drivers are required for crowdstrike" and "windows is not modular enough to expose necessary functionality to userspace". It could be solved differently given enough motivation.
Reason077
Are they? Apple has pretty much banned kernel drivers (kexts) in macOS on Apple Silicon. When they were still used, they were a common cause of crashes and instability, not to mention potential gaping security holes.

Most things that third-party kernel drivers used to do (device drivers, file systems, etc) are now done just as well, and much more safely, in userspace. I'm surprised if Microsoft isn't heading in this direction too?

Presumably, Crowdstrike runs on macOS without a kernel extension?

cywick
> Presumably, Crowdstrike runs on macOS without a kernel extension?

That's correct: CrowdStrike now only installs an "Endpoint Security" system extension and a "Network" system extension on macOS, but no kernel extension anymore.

lucasRW
Whatever protection is implemented in user-land can be removed from user-land too. This is why most EDR vendors are now gradually relying on kernel based mechanisms rather than doing stuff like injecting their DLL in a process, hooking syscalls, etc...
JackSlateur
This is wrong, there are many facilities that, once applied, cannot be modified (unless reboot)
lucasRW
Such as ?
JackSlateur
Random example: https://man.openbsd.org/OpenBSD-7.3/msyscall

This is a syscall used by userspace to tell the kernel which memory portion is allowed to do syscalls

This syscall can only be used once : once the linker has done it, the kernel will refuse extra calls (so allowing more memory pages is not possible)

lucasRW
First, we were talking about EDR in Windows usermode.

Second, still, that doesn't change anything. You can make your malware jmp to anywhere so that the syscall actually comes from an authorized page.

In fact, in windows environment, this is actively done ("indirect syscalls"), because indeed, having a random executable directly calling syscalls is a clear indicator that something is malicious. So they take a detour and have a legitimate piece of code (in ntdll) do the syscall for them.

tankenmate
One would hope that Crowdstrike does a similar thing on Linux and relies on fanotify and/or ebpf instead of using a kernel module. The other upside to this would be not having to wait for Crowdstrike to be constantly updating their code for newer kernels.
wjdp
I believe so but would like better details. We used to use another provider that depended on exact kernel versions whereas the falcon-sensor seems quite happy with kernel updates.
mrweasel
The original Windows NT had microkernel architecture, where a driver/server could not crash the OS. So no, Crowdstrike didn't have an option really, but Microsoft did.

As PC got faster, Microsoft could have returned to the microkernel architecture, or at least focused on isolating drivers better.

DaoVeles
Yep, this is technical legacy in action.
jsheard
They've done it to a degree but only for graphics drivers, Windows is (AFAIK) unique amongst the major OSes in that it can nearly always recover from a GPU driver or hardware crash without having to reboot. It makes sense that they would focus on that since graphics drivers are by far the most complex ones on most systems and there are only 3 vendors to coordinate API changes with, but it would be nice if they broadened it to other drivers over time.
nullindividual
NT was never a true microkernel. Most drivers are loaded into the kernel. Display drivers being a huge pain point, subsequently rolled back to user space in 2000, and printer drivers being the next pain point, but primarily with security -- hence moving to a Microsoft-supplied universal print driver, finally in Windows 11.
tyho
Required for crowdstrike to do what crowdstrike does. Which is mostly useless security theatre.
adrianN
The people installing crowdstrike have an option: Don't install it.
simfoo
Yep. We can't migrate our workstations to Ubuntu 24.04 because Crowdstrikes falcon kernel modules don't support the kernel version yet. Presumably they wanted to move to EBPF but I'm guessing that hasn't happened yet. Also: I can't find the source code of those kernel modules - they likely use GPL-only symbols, wouldn't that be a GPL violation?
actionfromafar
Probably only a violation if you distribute the linked result. Not if you only install it.
saagarjha
How would you install it without them distributing it?
diffeomorphism
Two different "it". As an analogy: selling pizza Hawaii is dicey, but you can sell pineapple slices and customers can add those to their pizza themselves.
tpush
They mean distributing Linux + the module together. Like e.g. shipping the Nvidia kernel module alone is fine, but shipping a Linux distro with that module preinstalled is not fine.
rkangel
The generally accepted (but not well tested) legal position is that it's ok to have a proprietary kernel module that is dynamically loaded.

You can, for instance, ask a running kernel if it is "tainted" by having loaded a non-GPL module.

immibis
GPL exported symbols are the ones that are thought to be so tightly coupled to the kernel implementation that if you are using them, you are writing a derivative work of the kernel.
simfoo
Yeah that was also my understanding, and I can't imagine a av module able to intercept filesystem and syscalls to be only using non-core symbols. But of course you never know without decompiling the module
yjftsjthsd-h
> and I can't imagine a av module able to intercept filesystem and syscalls to be only using non-core symbols.

I can, considering that you can do that from user space using strace. Or ebpf which is probably the actual right way to do this kind of thing.

vbezhenar
Last time I dealt with HP, I had to use their fakeraid proprietary kernel module which "tainted" the kernel. Of course they never open-sourced it. I guess it's not necessary.
pmontra
Why would you use Crowdstrike on Ubuntu? Is because of a real security concern, or abiding to regulations (thou shalt have an antivirus) or else?
simfoo
Ask my IT dep. AFAIK it's audit related, safety-critical software
jonathanlydall
I think "compliance" would be a better word to use that "safety" when it comes to a lot of "security" software on computers.

And I bring up the distinction because while compliance is "sometimes" about safety, it's also very often about KPIs of particular individuals or due to imaginary liability for having not researched every possible "compliance" checkbox conceivable and making sure it's been checked.

Some computer security software is completely out of hand because its primary purpose is to have the appearance of effectiveness for the exec whose job is to tick off as many safety checkboxes as they can find, as opposed to being actually pragmatically effective.

If the same methodologies were applied to car safety, cars would be so weighed down by safety features, that they wouldn't be able to go faster than 40km/h.

slow_typist
Just to be safe, of course! In my org they try to rollout sentinel one on every ‘endpoint’ regardless of operating system.
thyrsus
I was given to understand that Crowdstrike provided some protection from unvetted export of data. I'm not sure that data would be useful without the rare domain expertise to use it, but I wasn't shown the risk analysis. And then someone else demands and gets ssh access to GitHub. Sigh.
qwertox
> We can't migrate our workstations to Ubuntu 24.04 because Crowdstrikes

Should you upgrade before 24.04.1 is released? It's scheduled for August 15.

CoastalCoder
IIRC, about a 12-18 months ago CrowdStrike was recruiting for a development with eBPF skills.
j-krieger
So what? Crowdstrike is a kernel AV. How else would you solve this?
mr_mitm
It's a design decision. People want the antivirus to protect them even if an attacker exploits a local privilege escalation vulnerability or if an attacker that compromised an admin account (which happens all the time in Windows environments) wants to load malicious software. That's kind of the point of these things. Somebody exploits a memory vulnerability of one of the hundreds of services on a system, the antivirus is supposed to prevent that, and to their benefit, Crowdstrike is very good at this. If it didn't run in the kernel, an attacker with root can deactivate the antivirus. Since it's a kernel module, the attacker needs to load a signed kernel module, which is much harder to achieve.
jsheard
Presumably Crowdstrikes driver also has the ELAM flag which guarantees it will be loaded before any other third party drivers, so even if a malicious driver is already installed they have the opportunity to preempt it at boot.

https://learn.microsoft.com/en-us/windows-hardware/drivers/i...

chrisjj
> guarantees it will be loaded before any other third party drivers

Point of information. "Guarantee" and "any" are unsubstantiated by that MS article.

jsheard
If we are being pedantic then an ELAM driver can't be guaranteed to load before another ELAM driver of course, but only a small list of vetted vendors are able to sign ELAM drivers so it is very unlikely that malware would be able to gain that privilege. That's the whole point.
chrisjj
Not pedantic. Just accurate.

> an ELAM driver can't be guaranteed to load before another ELAM driver of course,

Thanks for the correction.

gamma032
At the end of the day, if you give an application a deep set of permissions, that's on you as an administrator, not the OS. This unchecked global rollout appears to just be a violation of every good software engineering practice we know.
BLKNSLVR
> This unchecked global rollout appears to just be a violation of every good software engineering practice we know.

Yeah, this is what surprises me. Corporate infrastructure policy seems to have been matched to smart phone default settings.

freetanga
I have worked in Finance for 25 years, and the amount of pressure I had to stand from Auditing on "Why do we have a 20-day-window on applying most updates as we get them from suppliers? We are not best practice!" is gruelling.

These people report to the Board Chairman, don't understand any real implication of their work, and believe the world is a simplistic Red - Amber - Green grid.

I understand most CIOs / CTOs / CISOs in Corporate would buckle.

chii
So the silver lining from this incident would be that you can simply point to it, and tell those auditors to fuck off.
vbezhenar
I'm pretty sure Apple does gradual rollouts of upgrades, so default smartphone settings are better than that.
strunz
It's actually worse than phone updates. Ever looked at your phone and noticed it hasn't updated to the new OS despite it having been out for a few days already? This is why.
bzzzt
Administrators are to blame because management (and a lot of 'cybersecurity policies') demand there's a virus scanner on the machines?

While virus scanners might pick up some threats not addressed by OS updates yet every one of them I've seen is a rootkit in disguise wanting full system privileges. There are numerous incidents with security holes and crashes caused by these security products. They also aren't that clever: repeatedly scanning the same files 'on access' over and over again wasting CPU and IO is not going to give you any extra security.

attentive
Not so much in disguise.

CS has official RCE root/admin access on all the clients. Which skips any normal auth of the OS. Yes, on all windows, mac and linux.

mycall
I often watch Crowdstrike thrash my laptop's resources, making it slow to do compiles. Cybersecurity won't let me disable it either, so I just set it to lower priority process.
aeyes
You might have more luck asking Cybersecurity to add a path like ~/code which contains your source code to the exclusion list.
mrweasel
As someone who worked for a company, who's a Crowdstrike partner, I assure you that Crowdstrike does not sell to administrators. It is very much a product sold to management and company auditors.

Where you're correct is that it's on the administrators to rollout the updates, but I'm not sure that's how Crowdstrike works. It's a managed solution and updates are done for you, maybe that can be disabled, but I honestly don't know.

tomohawk
This should clue you in.

CS is not sold to SA or technical types. It's sold to management as a risk reduction.

The whole point is that if you are technical, you are so untrusted that management is willing to require circumvention of known good practices and force installation of this software against technical advice.

nikau
I feel the need to make a mustang car crowdsrike meme
CoastalCoder
Could you elaborate?
nikau
Mustangs are famous for their high power and poor handling - there are lots of videos showing drivers doing burnouts, losing control, and striking the crowd they are showing off to.
Arrath
Mustangs have a reputation as being 'crowd (or streetlight) seeking' missiles.

This is due to their price making them relatively more available to the enthusiasts than say Hellcats, enthusiasts who may not be experienced enough to deal with having that much power available to them in a RWD car. This confluence of power, confidence and lack of skill often comes to a head when the enthusiast goes to a car meet to show off and meet with like minded folks. At the conclusion of the meet, or during a group drive, they'll often pull a sick burnout as they pull out of the parking lot on to a street.

A sick burnout they haven't practiced, and will often cause them to lose the back end sending the car into the curb, a tree, or a crowd of like minded attendees at the car meet. Therefore, the reputation.

For example: https://www.youtube.com/watch?v=DPx5aBI8UTQ

ncr100
I think it was a Dodge charger. That pro-trump KKK guy in the gray car who drove through the crowd while the crowd was I think it was a George Floyd protest?

Also heard today crowd stroke.

ric2b
There's a better joke, Crowdstrike sponsors the Mercedes Formula 1 team and in 1955 Mercedes was involved in the worst motorsport accident ever, killing over 80 people watching from the stands when parts of the cars flew off and... striked the crowd...
bratwurst3000
maybe they installed crowdstrike because they wanted updated without testing. and crowdstrike failed at testing them in their environment.

sounds like they didnt test all cases and stumbled on a windows bug

AmericanChopper
My experience working with Crowdstrike was that they were super arrogant about these risks. I was working on a ~50k enterprise rollout, and our CS guy was very belligerent about how long we were taking to do it, how much testing we wanted to do, the way that we were staggering roll outs and managing rollback plans. He didn’t think any of this was necessary, that we should roll it out in one fell swoop, have everything to auto-update all the time, and constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks.

He actually threatened to fire us as a client because he claimed he didn’t want the CS brand associated with an org that wasn’t “fully protected” by CS. By far the worst vendor contact I’ve ever had. I’ve had nicer meetings with Oracle lawyers than I was having with this guy. I hope this sort of thing humbles them a little.

BLKNSLVR
> constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks.

Evidence is pointing towards him actually being right about this, despite likely being wrong about everything else.

It'd be worth giving him a call, just to check in how he's going, and take him up on the offer to fire you as a client.

AmericanChopper
I was just a contractor there, and don’t work with them at the moment. But I’m a customer of theirs and they’re definitely having an outage right now, so I’m guessing it’s all still in place.
[deleted]
(comment deleted)
CoastalCoder
Mind rephrasing? I don't understand what you're saying.
AmericanChopper
I don’t work there any more. But they were having an outage, so I’m guessing they never got fired as a client (guessing that they’re still using Crowdstrike) and could still take that offer (of being fired as a client) if they wanted to.
ric2b
What evidence are you referring to? Was there a company that was breached for taking a few days or weeks to update crowdstrike?
DaoVeles
Ouch, considering the devil works under Oracles lawyers, thats bad!
shiroiushi
>I hope this sort of thing humbles them a little.

Hopefully not. It would be better that this company is sued into oblivion by all the customers that were affected by this huge outage.

mym1990
Maybe humbles all the other surviving companies? We can only dream
locococo
sounds like a very inexperienced person.

If their mission is to protect businesses they should understand your concerns.

Speed is useless without control.

AmericanChopper
He was pretty senior for his role, but really I have no idea whether he was representative of the wider company culture.

We had a buggy client release during the rollout which consumed all the CPU in one of our test environments (something he assured us could never happen), and he calmed down a bit after that. Prior to that though he was doing stuff like finding our CISO on LinkedIn to let him know how worried he was about our rollout pace, and that without CS protection a major breach could be imminent.

CoastalCoder
I'm guessing he's not typical, only because of he was, CrowdStrike would be known far and wide for this behavior.

For example, the way Oracle's lawyers are known.

radiator
> I hope this sort of thing humbles them a little.

What I hope, is that they stop to exist as a product and as a company. They have caused inconvenience, economic damage in global scale and probably also loss of life, given that many hospitals, ER units had outages. It has been proven that their whole way of working is wrong, from the very foundation to the top.

midtake
Maybe it's time that critical systems switch to Linux. The major public clouds are already primarily running Linux. Emergency services, booking, and traditional point-of-sale have no strong reason to run Windows. In the past 10 years, the technological capability differences between Windows and Linux have widened considerably, with Linux being the most advanced operating system in the world without question.

Concerns about usability between Windows and Linux in the modern day are disingenuous at best and malicious at worst. There is no UX concern when everything runs off a webapp these days.

Just use Linux. You will save money and time, and your system will be supported for many years, you won't be charged per E-Core, you won't suffer BSoDs in 2024. Red Hat is a trustworthy American company based out of Raleigh, NC, in case you have concerns of provenance.

Really there's no downside. If you were building your own company you would base your tech stack on Linux and not Windows.

Critical systems cannot go down; therefore they cannot run Windows. If they do, they are being mismanaged and run negligently. Management should have no issue finding Linux engineers, they are everywhere. I could make a killing right now as a consultant going from company to company and just swapping out Windows backends for Linux. And quite frankly I might just do that, literally starting right now.

DaoVeles
All good points but Windows didnt win because it had the best tech or user interface. Merely the most developer support thus user numbers. Legacy momentum is an incredibly difficult thing to sway. It has taken Apple decades an potentially hundreds of billions of dollars of marketing and good will to carve out its share of the market. Linux doesn't have that despites its clear technical advantages.

It is an incredibly frustrated battle akin to Sisyphus.

attentive
oh, you think security won't mandate to run CS on linux.

Granted it didn't down linux this time but nothing is stopping it.

snotrockets
It’s not security, it’s compliance. The two are sometimes aligned, sometimes less so.
chippiewill
Yeah, they definitely would mandate it.

My work laptop is running Ubuntu, and corporate IT requires Symantec Antivirus to be running on it

hrrsn
We've had production outages caused by Microsoft Defender our RHEL boxes :(
dist-epoch
> Linux being the most advanced operating system in the world without question.

Only if you don't need a GUI/Desktop.

prmoustache
I rate Linux DE higher than I do windows and Mac desktop tbh. Better ergonomics, better user experience and less bloat.
blippitybleep
I could never get smooth scrolling to work on Linux in any mainstream web browser, most people don’t seem to see it, but I’m sensitive to things like that.
prmoustache
Imho that was somewhat true on x11 but on wayland I feel everything is much smoother. I am more a pgup/pgdown user though.
cjbillington
Like with a laptop trackpad? I'm smooth-scrolling through these comments right now, and don't remember when scrolling wasn't smooth by default on any trackpad.
blippitybleep
It’s smooth to a point, but not smooth like OS X is. It might have improved (I think I last tried desktop Linux a year ago). I do enjoy using Linux as my default headless OS.
commandersaki
I need a few accessibility settings and Mac just excels in this regard.
ric2b
I have a GUI/Desktop on Linux, not sure what you're referring to?
lelanthran
> Only if you don't need a GUI/Desktop.

I not only need a GUI/Desktop, it's my daily driver!

And there are precious few things that Windows GUI/Desktop provides which I don't have on Linux, while the reverse is never true.

When I used Mac (Big Sur, I think?) until a year ago, I was absolutely miserable about having to use such a primitive GUI.

AdeptusAquinas
I don't understand this sort of blindness? Linux fails all the time, with rather terrible nobody to root vulns because some idiot failed to use the right bounds check. Ye gods, XZ utils was barely a few months ago!
zik
But no damage actually ended up happening with the xz utils exploit. It didn't even get released because someone picked it up pre-release.

Every system gets attacked, but I think your point shows that even with state-level attacks Linux handles it better than other platforms.

AdeptusAquinas
Hmm? It was released for two plus months? 5.6.0 and 5.6.1

I'd also say this wasn't a good example of 'linux handling it better': usually when a mess like this occurs on windows all the corps get a quiet tap on the shoulder that they need to immediately patch when MS releases it, then a few days later it hits the news. In XZ's case, the backdoor was published before the team knew about it, huge mess.

adrian17
You’re right that it went noticed for a long time, just one clarification

> all the corps get a quiet tap on the shoulder that they need to immediately patch when MS releases it, then a few days later it hits the news

AFAIK, distros were notified and released a patched version of xz like a week before it hit the news, so at least a lot of machines received it via automatic updates.

AdeptusAquinas
Depends which news you're talking about. MS guy who discovered it found it March 29th, published to oss. It was in infosec news same day as redhat, others pushed out critical advisories. Patch didn't come til a day or two later.
zik
You're half right - people who compiled it from source could theoretically get those releases, but no, it wasn't released in any distros. So in practice since no linux distro released it, no-one relying on linux distros was exposed to it.
prmoustache
Crowdstrike has a linux version. It is mandatory in our linux servers in my company so that is not the solution.

I would say issue 1 is management/compliance forcing admins to install malwares like crowdstrike. But issue 1 is because of issue 2 which is about admins / app devs / users aren't smart enough to not have their machines compromised on a regular basis in the first place. And issue 2 is because issue 3 of the software industry not focusing on quality and making bug free software.

All in all this should be mitigated by more diversity in OS, software and "said security solution". Standardization and monopolies works well until they don't and you get this kind of shit.

fsflover
> Crowdstrike has a linux version

But would it crash the OS?

baq
Why wouldn't it? This particular bug wouldn't, but another one...
TheFlash
Yes, the CS Falcon agent caused a kernel panic on RHEL about a month ago.
sterlind
and yet everyone is blaming Windows sigh.

Windows actually runs a lot of drivers in user-mode, even GPU drivers. largely this is because third-party drivers were responsible for the vast majority of blue screens, but the users would blame Microsoft. which makes sense; Windows crashes so they blame Windows, but I doubt anyone blamed Linux for the kernel panic.

fsflover
Linux is blamed for bad device drivers all the time, even on HN.
prmoustache
I think windows can be blamed on how badly you can fix that kind of issues. I mean on linux or any bsd admins would build an iso image that would automatically run a script that would take care of optionnally decrypting the system drive, then remove crowdstrike. Or alternatively simply building a live system that take an address via dhcp and start an ssh server. and admins would remotely and automatically run a playbook that mount that iso on the hypervisor, boot it, remotely apply the fix, then boot back the system on the system drive.

Maybe this is just my ignorance about windows and its ecosystem but it seems most admins this morning were clueless on how to fix that automatically and remotely on n machines and would resort to boot in safe mode and remove a file manually on each single server. This is just insane to think that supposed windows sysadmins / cloudops have no idea how to deploy a fix automatically on that platform.

nickjj
One place I'm at recently required us to install it in our Kubernetes cluster which powers a bunch of typical web apps.

Falcon sensor is the most CPU intensive app running in the cluster and produces a constant stream of disk activity (more so than any of our apps).

It hasn't crashed anything yet but it definitely leaves me feeling iffy about running it.

I don't like CrowdStrike at all. I got contacted by our security department because I used curl to download a file from GitHub on my dev box and it prompted a severe enough security warning that it required me to explain my intent. That was the day I learned I guess every command or maybe even keystroke I type is being logged and analyzed.

nevon
We were also forced to run that until the agent had introduced a memory leak that ate almost all the memory on all the hosts. Thankfully we managed to convince our compliance people that we could run an immutable OS rather than deploy this ~~malware~~ XDR agent.
prmoustache
It can kill process based on memory scanning. Imagine systemd was getting killed at every boot?

An issue might not be as universal as on windows, because some distros do things differently like not using glibc, or systemd, or whatever. Yet there are some baselines common to the most popular ones.

guax
I think we don't do enough to fight back this requests in a language that is understood by management. Ask them to sign a security waiver assuming risks for installing software techs would classify as a malware and RCE risk.

Companies like CS live on reputation, it should be dragged down.

NexRebular
> Maybe it's time that critical systems switch to Linux.

I switched critical systems to illumos and BSD years ago and it's been smooth sailing ever since. Nowadays there really is no need to contribute to linux monoculturization whatsoever.

bdd8f1df777b
I too want to see Linux more widely adopted, but it won't prevent this from happening. People will install corrupted kernel modules on Linux too for anti-virus purposes.
jsmith99
Do Linux systems not crash if a third party kernel module crashes? Or was your comment sarcastic?
MatejKafka
The discussed issue is not related to any meaningful difference between Windows and Linux – Crowdstrike used a kernel driver, apparently containing a serious bug, which took down the system, which is something any kernel driver can do, no matter which kernel you use. At least Windows have a well-developed framework for writing userspace drivers, unlike Linux.

> Linux being the most advanced operating system in the world without question.

Very strong and mostly unfounded claim; there are specific aspects where Linux is "more advanced", and others where Windows come out ahead (e.g. almost anything related to hardware-based security and virtualization).

> your system will be supported for many years

Windows Server 2008 was supported until earlier this year, longer than any RHEL release.

> you won't suffer BSoDs in 2024

Until you install a shitty driver for a dubious (anti)malware service.

mhh__
Other than a lack of redundant systems, it should be illegal to roll out updates like this to more than x% of any gov stuff at a time. Brute force way of avoiding correlational Armageddon.
JonChesterfield
Don't root kit yourself then cry about it when it falls over. Problem solved.
btbuildem
Crowdstrike is basically corporate malware - the failure is in large part with security dept deciders who signed off on policies that compel people to install these viruses on their work machines.
tru3_power
Crazy- wasn’t Azure having an outage earlier today? Is this related?
BLKNSLVR
I'd guess that it's related. Azure, OneDrive, and Office 365 were having issues this morning Australian time (about 6 hours ago).
jdleel
Doesn't actually look like. Noone really figured out the solution yet
tru3_power
If that’s the case I wonder why we’re only now seeing everyone’s systems BSODing. Should be an interesting write up (if they release one)
cpf_au
This is effecting our company. A colleague visited her local supermarket (Woolworths) and all the self-service checkouts were effected.
[deleted]
(comment deleted)
vinura
Anyone has a good news? or still it's on the BSOD loop?
tgshaik
use windows recovery to boot into safe mode and update latest crowdstrike. That is the only option at this time.
ali2724
Harvey Norman’s system is down
l0g4n_me
c:\system32\drivers\csagent.sys renaming this file on server with safe mode boot fixes the issue but disables agent
jmcgough
My entire emergency department got knocked offline by this. Really scary when you have ambulances coming in and are trying to stabilize a heart attack.

Update: 911 is down in Oregon too, no more ambulances at least.

sexy_seedbox
But at least the best instant messaging app in the world Microsoft Teams and the best web browser in the world Microsoft Edge are working fine, right?
jmcgough
It's a bsod loop, so not really.
jeffrallen
"Always look on the bright side of life!" - M Python
spike021
Do you have offline backup processes at least? Nasty situation.
jmcgough
We're really prepared for epic to go down and have an isolated cluster that we access in emergencies. I transitioned from software engineering so I've only been in the ED for a year, but from what I could see there didn't seem to be a plan for what to do if every computer in the department bluescreened at once.