I have been thinking about this. How do I make my git setup on my laptop secure? Currently, I have my ssh key on the laptop, so if I want to push, I just use git push. And I have admin credentials for the org. How do I make it more secure?
My SSH keys aren't on my computer: they're safely hidden on a hardware token, behind a secure element, like a Yubikey.
Devices like the Yubikey do precisely exist because computers aren't things to be trusted. So their reason for being is to offer a minimal attack surface.
When I git fetch/pull/push I just do it. But it requires me to physically use my Yubikey. It's not 100% foolproof but it's way better than having SSH keys only protected by a password.
So Git over SSH, on a Git/SSH server that supports Yubikeys.
The approach the attacker took makes little sense to me, perhaps someone else has an explanation for it? At first they monitored what's going on and then silently exfiltrated credentials and private repos. Makes sense so far. But then why make so much noise with trying to force push repositories? It's Git, surely there's a clone of nearly everything on most dev machines etc.
It's most likely two or more separate attackers operating. The first malware, Shai Hulud 2, exfiltrates credentials from the infected dev machine to new public GitHub repositories. As the repositories are public and searchable via GitHub's interfaces, any malicious attacker aware of the attack can easily grab the credentials and launch any attack, whether it's a noisy destructive script or some sophisticated ransomware.
That’s weird, pnpm no longer automatically runs lifecycle scripts like preinstall [1], so unless they were running a very old version of pnpm, shouldn’t they have been protected from Shai-Hulud?
> This incident involved one of our engineers installing a compromised package on their development machine, which led to credential theft and unauthorized access to our GitHub organization.
The org only has 4-5 engineers. So you can imagine the impact a large org will have.
> stored in our database which was not compromised
Personally I don't really agree with "was not compromised"
You say yourself that the guy had access to your secrets and AWS, I'd definitely consider that compromised even if the guy (to your knowledge) didn't read anything from the database. Assume breach if access was possible.
Given that all the stolen credentials were made public, I was hoping that someone would build a haveibeenpwned style site. We know we were compromised on at least a few tokens, but it would be nice to be able to search using a compromised token to find out what else leaked. We’ve rotated everything we could think of but not knowing if we’ve missed something sucks.
It was a really noisy worm though, and it looked like a few actors also jumped on the exposed credentials making private repos public and modifying readmes promoting a startup/discord.
>Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently.
No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. You only have "secutity" to the extent that you can trust the people who control those packages to act both competently and in good faith ad infinitum.
Also the OP seemingly implies credentials are stored on-filesystem in plaintext but I might be extrapolating too much there.
>> The security failure is in an ecosystem that allows packages to run arbitrary code silently.
> No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight.
How about both? It’s conceptually straightforward to build a language in which code cannot do anything other than read its inputs, consume resources, and produce correctly typed output.
This would not fully solve the supply chain problem — malicious code could produce maliciously incorrect output or exploit side channels, but the exposure would be much, much less than it is now.
> This is one of the frustrating realities of these attacks: once the malware runs, identifying the source becomes extremely difficult. The package doesn't announce itself. The pnpm install completes successfully. Everything looks normal.
Sounds like there’s no EDR running on the dev machines? You should have more to investigate if Sentinel One/CrowdStrike/etc were running.
We don't have a clear explanation of the destructive behavior, right? It looks like it had no real purpose, and there were much more effective ways of destroying their repos. Very script kiddie-like, which does not really fit the main complexity of the virus. Very surprising.
> """
I'm strongly in favor of blocking post-install scripts by default. :+1:
This is a change that will have a painful adjustment period for our users, but I believe in ~1 year everyone will look back and be thankful we made it. It's nuts that a [pnpm|yarn|npm] install can run arbitrary code in the first place.
"""
It’s almost like Microsoft sells security products and runs the most insecure JavaScript package manager to build those security products and couldn’t switch off of it even if the engineers in the org recommended a more secure JavaScript execution context— and that’s realistically why anthropic bought an engine.
I'm wondering why storing creds in env variables as plain text is acceptable - e.g. they better be dynamically fetched from a secret manager with 2FA in the way
"The simultaneous activity from US and India confirmed we were dealing with a single attacker using multiple VPNs or servers, not separate actors."
Did it really? It's not clear to me why the possibility that the exfiltrated credentials were shared with other actors, each acting independently, is ruled out.
35 comments
[ 2.9 ms ] story [ 61.2 ms ] threadI'm curious was the exfiltration traffic distinguishable from normal developer traffic?
We've been looking into stricter egress filtering for our dev environments, but it's always a battle between security and breaking npm install
My SSH keys aren't on my computer: they're safely hidden on a hardware token, behind a secure element, like a Yubikey.
Devices like the Yubikey do precisely exist because computers aren't things to be trusted. So their reason for being is to offer a minimal attack surface.
When I git fetch/pull/push I just do it. But it requires me to physically use my Yubikey. It's not 100% foolproof but it's way better than having SSH keys only protected by a password.
So Git over SSH, on a Git/SSH server that supports Yubikeys.
1: https://github.com/pnpm/pnpm/pull/8897
The org only has 4-5 engineers. So you can imagine the impact a large org will have.
There has to be a tool that allows you (or an AI) to easily review post-install scripts before you install the package.
Personally I don't really agree with "was not compromised"
You say yourself that the guy had access to your secrets and AWS, I'd definitely consider that compromised even if the guy (to your knowledge) didn't read anything from the database. Assume breach if access was possible.
It was a really noisy worm though, and it looked like a few actors also jumped on the exposed credentials making private repos public and modifying readmes promoting a startup/discord.
No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. You only have "secutity" to the extent that you can trust the people who control those packages to act both competently and in good faith ad infinitum.
Also the OP seemingly implies credentials are stored on-filesystem in plaintext but I might be extrapolating too much there.
> No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight.
How about both? It’s conceptually straightforward to build a language in which code cannot do anything other than read its inputs, consume resources, and produce correctly typed output.
This would not fully solve the supply chain problem — malicious code could produce maliciously incorrect output or exploit side channels, but the exposure would be much, much less than it is now.
Sounds like there’s no EDR running on the dev machines? You should have more to investigate if Sentinel One/CrowdStrike/etc were running.
> Total repos cloned: 669
How big is this company? All the numbers I can find online suggest well below 100 people, and yet they have over 600 repos? Is that normal?
* Spikes/Demo project
* Smaller projects that might have gone live, but have since been migrated elsewhere
* Core services
* Forks of certain supply chain dependencies that we've made improvements to.
I beg to differ and look forward to running my own fiefdom where interpreter/JIT languages are banned in all forms.
- a pnpm maintainer 1 year ago
https://github.com/pnpm/pnpm/pull/8897
Convenience trumps security every time. With people who allegedly know better.
Did it really? It's not clear to me why the possibility that the exfiltrated credentials were shared with other actors, each acting independently, is ruled out.
I think if you look at all the actions, and see that they don't overlap, then it might make sense.
If access was: Location: Repo USA: 1,2,3,4 India: 5,6,7,8
The it is reasonable to assume that the access was from that same actor, as it is coordinated in some way.