What Happened?
• The compromised Action executes a Python script that dumps CI/CD secrets from the Runner Worker process.
• Multiple v35 tags were modified four hours ago, indicating a recent supply chain attack.
• The malicious behavior can be observed in StepSecurity Harden-Runner insights, showing the Action downloading and executing an unauthorized script.
Wow that's scary, they updated tons of tags to an offending random commit. With the way repositories are included in automation and the fact that this adjusted the tags of older versions (so not requiring an upgrade) this sort of attack can have a huge impact very quickly :(.
Maybe GitHub should have some kind of security setting a repo owner can make that locks-down things like old tags so after a certain time they can't be changed.
Tags are a git concept, not a GitHub concept. Tags provide human-readable names to commits. They're intended to be changeable, to allow things like a "latest" tag pointing to the latest release. Tags aren't versions, commit hashes are.
StepSecurity Harden-Runner detected this security incident by continuously monitoring outbound network calls from GitHub Actions workflows and generating a baseline of expected behaviors. When the compromised tj-actions/changed-files Action was executed, Harden-Runner flagged it due to an unexpected endpoint appearing in the network traffic—an anomaly that deviated from the established baseline. You can checkout the project here: https://github.com/step-security/harden-runner
The advertising in this article is making it actively difficult to figure out how to remediate this issue. The "recovery steps" section just says "start our 14 day free trial".
The security industry tolerates self-promotion only to the extent that the threat research benefits everyone.
Thank you, cyrnel, for the feedback! We are trying our best to help serve the community. Now, we have separate recovery steps for general users and our enterprise customers.
In recent years, it's started to feel like you can't trust third-party dependencies and extensions at all anymore. I no longer install npm packages that have more than a few transitive dependencies, and I've started to refrain from installing vscode or chrome extensions altogether.
Time and time again, they either get hijacked and malicious code added, or the dev themselves suddenly decides to betray everyone's trust and inject malicious code (see: Moq), or they sell out to some company that changes the license to one where you have to pay hundreds of dollars to keep using it (e.g. the recent FluentAssertions debacle), or one of those happens to any of the packages' hundreds of dependencies.
Are there examples of these types of actions in other circles outside of the .NET ecosystem? I knew about the FluentAssertions ordeal, but the Moq thing was news to me. I guess I've just missed it all.
node-ipc is a recent example from the Node ecosystem. The author released an update with some code that made a request to a geolocation webservice to decide whether to wipe the local filesystem.
Where do I stand on the war? I stand with Ukraine.
Where do I stand on software supply chain issues? I stand with not fucking around with the software supply chain.
I'd also emphasize out that there's nothing safe about it being "only dev", given how many attacks use employee computers (non-prod) as a springboard elsewhere.
The original .NET (and I think Java?) had an idea in them of basically library level capability permissions.
That sort of idea seems increasingly like what we need because reputation based systems can be gamed too easily: i.e. there's no reason an action like this ever needed network access.
It was only recently removed in Java and there was a related concept (adopted from OSGi) designed to only export certain symbols -- not for security but for managing the surface area that a library vendor had to support
But I mentioned both of those things because [IMHO] they both fell prey to the same "humanity bug": specifying permissions for anything (source code, cloud security, databases, Kubernetes, ...) is a lot of trial and error, whereas {Effect: Allow, Action: ["*:*"]} always works and so they just drop a "TODO: tighten permissions" and go on to the next Jira
I had high hopes for the AWS feature "Make me an IAM Policy based on actual CloudTrail events" but it talks a bigger game than it walks
When using NuGet packages I usually won't even consider ones with non-Microsoft dependencies, and I like to avoid third-party dependencies altogether. I used to feel like this made me a weird conspiracy theorist but it's holding up well!
It also has led to some bad but fun choices, like implementing POP3 and IMAP directly in my code, neither of which worked well but taught me a lot?
Yes. Same with browser plugins. I've heard multiple free-plugin authors say they're receiving regular offers to purchase their projects. I'm sure some must take up the offer.
No I don’t. But Wladimir Palant is where I get most of my information on the topic (and is probably where I got this link). His blog might have a post (or a comment) that links to similar lists: https://palant.info/categories/security/
This is why I fork the extensions I use, with the exception of uBlock. Basically just copy the extension folder, if I can't find it on GitHub. That way I can audit the code and not have to worry about an auto-update sneaking in something nefarious. I've had two extensions in the past suddenly start asking for permissions they definitely did not need, and I suspect this is why.
Btw, here's a site where you can inspect an extension's source code before you install it: https://robwu.nl/crxviewer/
Yeah, and thx for the link to the neat crx explorer.
Close to what you do, I started writing my own addon to replace a couple addons whose featureset I use only partially.
For example, when I use Chromium I want 1. to customize the New Tab page, and 2. to add a keyboard shortcut to pin/unpinTab. These two features are absolutely part of extensions, but in addition to the security risk I find them heavy (I don’t need the kitchen sink, just need 2 micro-features!). And so, I have my little personal addon with zero resource usage with just these two features. It’s tiny (20 lines of code!), git-versioned, and never changes / gets pwned. When I need an extra micro-feature, it’s easy enough to add it by searching addons docs, of asking an LLM.
You shouldn’t need an extension just to add a keyboard shortcut for a menu item. Doesn’t your OS let you map that? On macOS you can in Keyboard Settings
I have long since stopped using any extension that doesn’t belong to an actual company (password managers for example). Even if they aren’t malware when you installed them, they will be after they get sold.
Did you turn off updates on your phone as well? Because 99.999% of people have app auto-updates and every update could include an exploit.
I'm not saying you're wrong not to trust package managers and extensions but you're life is likely full of the same thing. The majority of apps are made from 3rd party libraries which are made of 3rd party libraries, etc.... At least on phones they update constantly, and every update is a chance to install more exploits.
The same is true for any devices that get updates like a Smart TV, router, printer, etc.... I mostly trust Apple, Microsoft, and Google to check their 3rd party dependencies, (mostly), but don't trust any other company - and yet I can't worry about it. Don't update and I don't get security vulnerabilities fixed. Do update and I take the chance that this latest update has a 3rd party exploit buried in a 3rd party library.
I don't trust apps. I trust Apple (enough) that they engineered iOS to have a secure enough sandbox that a random calculator app can't just compromise my phone.
Most developer packages have much higher permission levels because they integrate it with your code without a clear separation of boundaries. This is why attackers now like to attack GitHub Actions because if you get access to secrets you can do a lot of damage.
If this were “the solution”, then the many, many smart individuals and teams tasked with solving these problems throughout the software industry would’ve been out of work for some time now.
It’s obviously more complicated than that.
Signed public builds don’t inherently mean jack. It highly depends on the underlying trust model.
—
Malicious actor: “we want to buy your browser extension, and your signing credentials”.
Plugin author: “Well, OK”.
—
Malicious actor: hijacks npm package and signs new release with new credentials
The vast majority of dependent project authors: at best, see a “new releaser” warning from their tooling, which is far from unusual for many dependencies. ignores After all, what are they going to do?.
—
Hacker News, as usual, loves to pretend it has all the answers to life’s problems, and the issue is that nobody has listened to them.
> Hacker News, as usual, loves to pretend it has all the answers to life’s problems, and the issue is that nobody has listened to them.
eh, it’s not just HN.
like, there’s no single technical/material solution to something as complex and widespread as humanity’s apparent base need to “get more stuff”. which is the root cause for acting maliciously — it’s just “getting more stuff” in a way that’s harmful to others.
but that won’t stop people from claiming that they can come up with a technical solution. whether that’s politicians, tech bros, HN commentators or that guy down the pub on a thursday evening.
—
that being said, signing software is better than doing nothing… so, a better way of phrasing it from the GP would probably have been it is a partial mitigation for the problem in some cases.
We need better capabilities.
E.g. when I run `fd`, `rg` or similar such tool, why should it have Internet access?
IMHO, just eliminating Internet access for all tools (e.g. in a power mode), might fix this.
The second problem is that we have merged CI and CD.
The production/release tokens should ideally not be on the same system as the ones doing regular CI.
More users need access to CI (especially in the public case) than CD.
For example, a similar one from a few months back https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-inj...
You also need to block write access, so they can’t encrypt all your files with an embedded public key. And read access so they can’t use a timing side channel to read a sensitive file and pass that info to another process with internet privileges to report the secret info back to the bad guy. You get the picture, I’m sure.
> You also need to block write access, so they can’t encrypt all your files with an embedded public key. And read access so they can’t use a timing side channel to read a sensitive file and pass that info to another process with internet privileges to report the secret info back to the bad guy. You get the picture, I’m sure.
Indeed.
One can think of a few broad capabilities that will drastically reduce the attack surface.
1. Read-only access vs read-write
2. Access to only current directory and its sub-directories
3. Configurable Internet access
Docker mostly gets it right.
I wish there was an easy way to run commands under Docker.
E.g.
If I am running `fd`
1. Mount current read-only directory to Docker without Internet access (and without access to local network or other processes)
2. Run `fd`
3. Print the results
4. Destroy the container
> 1. Mount current read-only directory to Docker without Internet access (and without access to local network or other processes) 2. Run `fd` 3. Print the results 4. Destroy the container
Systemd has a lot of neat sandboxing features [1] which aren't well known but can be very useful for this. You can get pretty far using systemd-run [2] in a script like this:
Which creates a blank filesystem with no network or device access and only bind mount the specified files.
Unfortunately TemporaryFileSystem require running as a system instance of the service manager rather than per-user instance, so that will generally mean running as root (hence sudo). One approach is to create a suid binary that does the same without needing sudo.
This is exactly what the tool bubblewrap[1] is built for. It is pretty easy to wrap binaries with it and it gives you control over exactly what permissions you want in the namespace.
I get the picture, yes, namely that probably 99% of project dependencies don't need I/O capabilities at all.
And even if they do, they should be controlled in a granular manner i.e. "package org.ourapp.net.aws can only do network and it can only ping *.aws.com".
Having finer-grained security model that is enforced at a kernel level (and is non-circumventable barring rootkits) is like 20 years overdue at this point.
I’m not sure I can recommend Qubes entirely due to the usability aspect.
I’ve used Qubes several times for a week at a time over the last few years. It’s gotten better, but they really need someone to look at the user experience of it all for it to be a compelling option.
I’m regularly questioning myself if what I’m doing is making it less secure because I don’t understand exactly everything Qubes is doing. I know how all the pieces work individually (Xen, etc).
Outside of configuration, I believe I’d have to ditch any hope of running 3D-anything with any expectation of performance. That’s simply a non-starter as someone who has written off “nation-state actor targeting me, specifically” as something I can defend against.
And lastly, I’m deeply skeptical of anything that loudly wears the Snowden-badge-of-approval as that seems to follow grifts.
My main workstation is a Mac and I’m doing this on Parallels. Would Qubes probably be more secure? Maybe. But it comes at a massive usability hit.
OpenBSDs pledge[0] system call is aimed at helping with this. Although, it's more of a defense-in-depth measure on the maintainers part and not the user.
> The pledge() system call forces the current process into a restricted-service operating mode. A few subsets are available, roughly described as computation, memory management, read-write operations on file descriptors, opening of files, networking (and notably separate, DNS resolution). In general, these modes were selected by studying the operation of many programs using libc and other such interfaces, and setting promises or execpromises.
How so? Obviously this is ineffective at the package level but if the thing spawning these processes, like the GitHub runners or Node itself added support to enter a "restricted" mode and pledged then that would help, no?
As far as I see its purpose is mostly a mitigation/self-defence for vulnerabilities in C-based apps, so basically limiting what happens once the attacker has exploited a vulnerability. Maybe it has other uses.
It could be used defending against bugs in the Node runtime itself, as you say, but as I understand vulnerabilities in the Node runtime itself are quite rare, so more fine-grained limitations could be implemented within itself.
But that's what firejail and docker/podman are for. I never run any build pipeline on my host system, and neither should you. Build containers are pretty good for these kind of mitigations of security risks.
For CI/CD using something like ArgoCD let's you avoid giving CI direct access to prod - it still needs write access to a git repo, and ideally some read access to Argo to check if deployment succeeded but it limits the surface area.
bubblewrap is a safer alternative to firejail because it does not use setuid to do its job, and it is used by flatpak (so hopefully has more eyes on it, but I have no idea).
You do have to assemble isolation scripts by hand though, it's pretty low level. Here is a decent comment which closely aligns to what I'm using to isolate npm/pnpm/yarn/etc, I see no need to repeat it:
Great points! Harden-Runner (https://github.com/step-security/harden-runner) is similar to Firejail and OpenSnitch but purpose-built for CI/CD context. Harden-Runner detected this compromise due to an anomalous outbound network request to gist.githubusercontent.com.
> We need better capabilities. E.g. when I run `fd`, `rg` or similar such tool, why should it have Internet access?
Yeah!! We really need to auto sandbox everything by default, like mobile OSes. Or the web.
People browse the web (well, except Richard Stallman) all the time, and run tons of wildly untrusted code, many of them malicious. And apart from zero days here and there, people don't pay much attention to it, and will happily enter any random website in the same machine they also store sensitive data.
At the same time, when I open a random project from Github on VSCode, it asks whether the project is "trusted". If not, it doesn't run the majority of features like LSP server. And why not? Because the OS doesn't sandbox stuff by default. It's maddening.
FreeBSD has Capsicum [0] for this. Once a process enters capability mode, it can't do anything except by using already opened file descriptors. It can't spawn subprocesses, connect to the network, load kernel modules or anything else.
To help with things that can't be done in the sandbox, e.g. DNS lookups and opening new files, it provides the libcasper library which implements them using helper processes.
Not all utilities are sandboxed, but some are and hopefully more will be.
Linux recently added Landlock [1] which seems sort of similar, although it has rulesets and doesn't seem to block everything by default, as far as I can tell from quickly skimming the docs.
I don't think it would help in this case, when the entire process can be replaced with malicious version. It just won't make the Capscium call.
What you really want is something external and easily inspectable, such as systemd per-service security rules, or flatpak sandboxing. Not sure if FreeBSD has somethingike this.
npm supply chain attacks are the lone thing that keeps me up at night, so to speak. I shudder thinking about the attack surface.
I go out of my way to advocate for removing dependencies and pushing against small dependency introductions in a large ruby codebase. Some dependencies that suck and impose all sorts of costs, from funky ass idiosyncratic behavior or absurd file sizes (looking at you any google produced ruby library, especially the protocol buffer dependent libraries) are unavoidable, but I try to keep fellow engineers honest about introducing libraries that do things like determine the underlying os or whatever and push towards them just figuring that out themselves or, at the least, taking "inspiration" from the code in those libraries and reproducing behavior.
A nice side effect of AI agents and copilots is they can sometimes write "organic" code that does the same thing as third party libraries. Whether that's ethical, I don't know, but it works for me.
This isn't new - Thompson warned us 40 years ago (and I believe others before him) in his Reflections on Trusting Trust paper.
It's something I've been thinking about lately because I was diving into a lot of discussion from the early 90s regarding safe execution of (what was, at the time, called) "mobile code" - code that a possibly untrustworthy client would send to have executed on a remote server.
There's actually a lot of discussion still available from w3 thankfully, even though most of the papers are filled with references to dead links from various companies and universities.
It's weirdly something that a lot of smart people seemed to have thought about at the start of the World Wide Web which just fell off. Deno's permissions are the most interesting modern implementation of some of the ideas, but I think it still falls flat a bit. There's always the problem of "click yes to accept the terms" fatigue as well, especially when working in web development. It's quite reasonable for many packages one interacts with in web development to need network access, for example, so it's easy to imagine someone just saying "yup, makes sense" when a web-related package requests network access.
Also none of this even touches on the reality of so much code which exists to brutally impact a business need (or perceived need). Try telling your boss you need a week or two to audit every one of the thousands of packages for the report generator app.
Trusting Trust is not about this at all. It's about the compiler being compromised, and making it impossible to catch malicious code by inspecting the source code.
The problem here is that people don't even bother to check the source code and run it blindly.
I'm just going to say this out loud: It's mostly a Javascript thing.
Not that every other platform in the world isn't theoretically vulnerable to the same sort of attack, but there's some deep-rooted culture in the javascript community that makes it especially vulnerable.
The charitable interpretation is "javascript evolves so fast!". The uncharitable interpretation is "they are still figuring it out!"
Either way, I deliberately keep my javascript on the client side.
Stealing crypto is so lucrative. So there is a huge 'market' for this stuff now that wasn't there before. Security is more important now than ever. I started sandboxing Emacs and python because I can't trust all the packages.
I hope the irony is not completely lost on the fine folks at semgrep that the admittedly "overkill" suggested semgrep solution is exactly the type of pattern that leads to this sort of vulnerability: that of executing arbitrary code that is modifiable completely outside of one's own control.
The alternative would be to find a sustainable funding model for open source, which is the source of betrayals due to almost all of the maintainers having to sell their projects to make a living in the first place.
The problem you're describing is an economical and a social one.
Currently, companies exploit maintainers of open source projects. There are rarely projects that make it due to their popularity, like webpack, when it comes to funding...but the actual state is that everyone that webpack is based on as a dependency didn't get a single buck for it, which is unfair, don't you think?
On top of sustainable funding, we need to change our workflows to reproducible build ecosystems that can also revert independent of git repositories. GitHub has become the almost single source of code for the planet, which is insane to even bet on from a risk assessment standpoint. But it's almost impossible to maintain your own registry or mirror of code in most ecosystems due to the sheer amount of transitive dependencies.
Take go mod vendor, for example. It's great to stick your dependencies but it comes with a lot of overhead work in case something like OPs scenario happens to its supply chain. And we need to account for that in our workflows.
It's not going to happen. If buying a forever license of unlimited usage for an open source library cost $1 I'd skip it. Not be cause I don't want to give money to people who deserve it, but because of the absolute monstrous bureaucratic nightmare that comes from trying to purchase anything at a company larger than 10 people.
Don't even talk about when the company gets a lawyer who knows what a software license is.
Open source has a very sustainable funding model as evidenced by 50 years of continuous, quality software being developed and maintained by a diverse set of maintainers.
I say sustainable because it has been sustained, is increasing in quantity and quality, and reasonably seems to be continuing.
> companies exploit maintainers of open source projects
Me giving something away and others taking what I give is not exploitation. Please don’t speak for others and claim people are exploited. One of the main tenets of gnu is to prevent exploitation.
This is the death of fun. Like when you had to use SSL for buying things online.
Adding SSL was not bad, don't get me wrong. It's good that it's the default now. However. At one point it was sorta risky, and then it became required.
Like when your city becomes crime ridden enough that you have to lock your car when you go into the grocery store. Yeah you probably should have been locking it the whole time. what would it have really cost? But now you have to, because if you don't your car gets jacked. And that's not a great feeling.
Used to live near San Francisco, and had a lot of coworkers say they intentionally leave their windows down when parking in SF so that burglars don't break the glass to steal something!
In the era of the key fob it's pretty automatic to lock the car every time. Some cars even literally do it for you. I hardly think of this, let alone get not great feelings about it.
You can trust (in time), but you can't blindly upgrade. Vendor or choose to "lock" with a cryptographic hash over the files your build depends on. You then need to rebuild that trust when you upgrade (wait until everyone else does; read the diffs yourself).
There is something to be said for the Go proverb "a little copying is better than a little dependency", as well. If you want a simple function from a complicated library, you can probably copy it into your own codebase.
Yeah, I’ve moved off vscode entirely, back to fully featured out of the box ides for me. Jetbrains make some excellent tools and I don’t need to install 25 (dubious) plugins for them to be excellent
> But Lewis Ardern on our team wrote a Semgrep rule to find usages of tj-actions, which you can run locally (without sending code to the cloud) via: semgrep --config r/10Uz5qo/semgrep.tj-actions-compromised.
So "remote code you download from a repo automatically and run locally has been compromised, here run this remote code you download from a repo automatically and run locally to find it"
This is why I have begin to prefer languages with comprehensive, batteries-included standard libraries so that you need very few dependencies. Dep Management has become a full time headache nowadays with significant effort going into CVE analysis.
I think library/runtime makers aren't saying "let's make an official/blessed take on this thing that a large number of users are doing" as much as they should.
Popular libraries for a given runtime/language should be funded/bought/cloned by the runtime makers (e.g. MS for .NET, IBM/Oracle for Java) more than they are now.
I know someone will inevitably mention concerns about monopolies/anti-trust/"stifling innovation" but I don't really care. Sometimes you have to standardize some things to unlock new opportunities.
Instead of bloating the base language for this, a trusted entity could simply fork those libraries, vet them, and repackage into some "blessed lib" that people like you can use in peace. In fact, the level of trust needed to develop safe libraries is less than developing language features.
49 modules with only one maintainer and over 600 modules with only one maintainer if devDependencies are included. This is only a matter of time until the next module becomes compromised.
You should have never trusted them. That ecosystem is fine for hobbyists but for professional usage you can't just grab something random from the Internet and assume it's fine. Security or quality wise.
I think the conventional approach of checking for vulnerabilities in 3rd party dependencies by querying CVE or some other database has set the current behaviour i.e. if its not vulnerable it must be safe. This implicit trust on vulnerability databases has been exploited in the wild to push malicious code to downstream users.
I think we will see security tools shifting towards "code" as the source of truth when making safety and security decision about 3rd party packages instead of relying only on known vulnerability databases.
Take a look at vet, we are working on active code analysis of OSS packages (+ transitive dependencies) to look for malicious code: https://github.com/safedep/vet
GitHub Actions should use a lockfile for dependencies. Without it, compromised Actions propagate instantly. While it'd still be an issue even with locking, it would slow down the rollout and reduce the impact.
Semver notation rather than branches or tags is a great solution to this problem. Specify the version that want, let the package manager resolve it, and then periodically update all of your packages. It would also improve build stability.
Since they edited old tags here … maybe GitHub should have some kind of security setting a repo owner can make that locks-down things like old tags so after a certain time they can't be changed.
The version numbers aren't immutable, so an attacker can just update the versions to point to the compromised code, which is what happened here. Commit hashes are a great idea, but you still need to be careful: lots of people use bots like Renovate to update your pinned hashes whenever a new version is published, which runs into the same problem.
Yes, you're right. I wasn't able to double-check as the repo was deleted at the time. That said, AIUI making the tags read-only would still often be vulnerable to semantic-version exploitation.
A signed commit [0] might be good for internal devops stuff (e.g. "yes, we really do want this version in production") but unfortunately that's not gonna work for pulling in third-party tooling, since most won't use it.
Also don't het GH actions to do anything other than build and upload artifacts somewhere. Ideally a write only role. Network level security too no open internet.
Use a seperate system for deployments. That system must be hygienic.
This isn't foolproof but would make secrets dumping not too useful. Obviously an attack could still inject crap into your artefact. But you have more time and they need to target you. A general purpose exploit probably won't hurt as much.
I've always felt uncomfortable adding other people's actions to my GitHub workflows, and this is exactly the kind of thing I was worried about.
I tend to stick to the official GitHub ones (actions/setup-python etc) plus the https://github.com/pypa/gh-action-pypi-publish one because I trust the maintainers to have good security habits.
That's exactly where I stand, and I feel partially vindicated by this outcome. There are so many useful Github actions made by randos, but I am not adding more unvetted dependencies to my project. I will unhappily copy and paste some useful code into my project rather than relying upon yet another mutable dependency.
I could have sworn that I've seen other GitHub Actions vulnerabilities that worked the same way, too. And/or HN submissions talking about this specific kind of vulnerability, the standard mitigation strategies, etc.
Feels like the same kind of problem as SQL injection, where everybody kinda knows about it and some people are actively aware and there are standard ways to avoid it but it still happens all the time anyway.
Might also be a good time to mention I'm really not a fan of YAML.
How does SBOM and such account for this? If you’re a package maintainer, do you need to include CI pipeline plugins, their dependencies, going down as far as the pipeline host, in your security-relevant dependencies? Hard problems :/
I wish Github required some sort of immutability for actions by default as most package managers do, either by requiring reusable actions to be specified via commit hash or by preventing the code for a published tag to be changed.
At the moment the convention is to only specify the tag, which is not only a security issue as we see here, but may also cause workflows to break if an action author updates the action.
You can target `some/action@commithash` already, that's up to you. You're also free to fork or clone each action you use, vet the code, and consume your fork in your workflows. You can also disable the use of third party actions at an org level, or approve them on a case-by-case basis.
This all depends on your threat model and risk tolerance, it's not so much a GitHub problem. There will always be bad code that exists, especially on the largest open source code hosting platform. You defend against it because that's more realistic than trying to eradicate it.
One problem with this is that actions can be Composite and call arbitrary other actions. So only if you use actions that themselves lock everything by commit for the actions they depend on you're safe.
You just described a supply chain, and the risks that come with them, which is something every dep management system is dealing with, rubygems, npm, etc
Again, it all comes down to your risk tolerance. There's a certain level of trust built into these systems.
Someone elsewhere suggested a lockfile, which seems a pretty obvious solution in hindsight. I'm fine with commit hashes, but the UX is terrible and consists of pasting the action into into StepSecurity's thingie, when this is something that GH should have built in.
commit hashes are immutable, and your own commit history can serve as the lock file.
but if you're targeting a commit hash directly, it's already locked. Lock files are for mapping a version range to a real life version number. Lock files are useless if you pin the exact version for everything.
Sure, it's just that the DX of getting that commit hash isn't terrific, so one might be more inclined to trust an auto-update bot to automatically update them instead. A lock file is more like TOFU on a tag. I'd also take a UI like a "bake" button and CLI flag that substituted the hashes automatically, but you just know people are going to build `--bake` right in to their automation.
Another solution would be to implement immutable tags in git itself, but git upstream has so far been hostile to the whole concept of immutability in any part of git.
A company I worked at went all in on GH too. The internal gh team probably going to do a fire drill this whole weekend and app teams forced to rotate all secrets and credentials.
Fortunately don’t have to deal with that shit anymore
My day job is also in the middle of moving everything to Github Actions, so this is fun. But in my case, we aren't affected by this vulnerability because it could only be exploited by workflows with public logs, and currently my company only uses Github Actions for private repositories.
I mean maybe! But only if you've removed all of the usage of this compromised `tj-actions/changedfiles` action, across all your repos and their branches.
Otherwise, if you continue to use it and it will run anytime there has been a push. Potentially on any branch, not just `main`! Depending on your GH config.
Unless you've blocked `tj-actions/changed-files` you're banking on the bad actor not coming back tonight and making malicious commit that exfils those secrets to pastebin.com.
Most folks around the world signed off. B-squad probably left cleaning up remaining tasks or just fucking around with co-workers and pondering the weekend. Most GH actions run on a schedule (ie, backups of db, connecting to blob storage services).
Attacker(s) likely to extract plenty of secrets and exfil data before the alarms get triggered (if any) at companies.
It's always been shocking to me that the way people run CI/CD is just listing a random repository on GitHub. I know they're auditable and you pin versions, but it's crazy to me that the recommended way to ssh to a server is to just give a random package from a random GitHub user your ssh keys, for example.
This is especially problematic with the rise of LLMs, I think. It's the kind of common task which is annoying enough, unique enough, and important enough that I'm sure there are a ton of GitHub actions that are generated from "I need to build and deploy this project from GitHub actions to production". I know, and do, know to manually run important things in actions related to ssh, keys, etc., but not everyone does.
The crazier part is, people typically don't even pin versions! It's possible to list a commit hash, but usually people just use a tag or branch name, and those can easily be changed (and often are, e.g. `v3` being updated from `v3.5.1` to `v3.5.2`).
You and someone else pointed this out. I only use GitHub-org actions, and I just thought that surely there would be a "one version to rule them all" type rule.. how else can you audit things?
I've never seen anything recommending specifying a specific commit hash or anything for GitHub actions. It's always just v1, v2, etc.
Fuck. Insecure defaults again. I argue that a version specifier should be only a hash. Nothing else is acceptable. Forget semantic versions. (Have some other method for determining upgrade compatibility you do out of band. You need to security audit every upgrade anyway). Process: old hash, new hash, diff code, security audit, compatibility audit (semver can be metadata), run tests, upgrade to new hash.
Aren't GitHub action "packages" designate by a single major version? Something like checkout@v4, for example. I thought that that designated a single release as v4 which will not be updated?
I'm quite possibly wrong, since I try to avoid them as much as I can, but I mean.. wow I hope I'm not.
Almost everyone will just copy paste this snippet and call it a day. Most people don't think twice that v4 is a movable target that can be compromised.
In case of npm/yarn deps, one would often do the same, and copy paste `yarn install foobar`, but then when installing, npm/yarn would create a lockfile and pin the version. Whereas there's no "installer" CLI for GH actions that would pin the version for you, you just copy-paste and git push.
To make things better, ideally, the owners of actions would update the workflows which release a new version of the GH action, to make it update README snippet with the sha256 of the most recent release, so that it looks like
I mean, I think there's a difference between trusting GitHub and trusting third parties. If I can't trust GitHub, then there's absolutely no point in hosting on GitHub or trusting anything in GitHub Actions to begin with.
But yes I do think using tags is problematic. I think for one, GitHub should ban re-tagging. I can't think of a good reason for a maintainer to re-publish a tag to another commit without malicious intent. Otherwise they should provide a syntax to pin to both a tag and a commit, something like this:
`uses: actions/checkout@v4.5.6@abcdef9876543210`
The action should only work if both conditions are satisfied. This way you can still gain semantics version info (so things like dependabot can work to notify an update) but the commit is still pinned.
---
I do have to say though, these are all just band-aids on top of the actual issue. If you are actually using a dependency that is compromised, someone is going to get screwed. Are you really going to read through the commit and the source code to scan for suspicious stuff? I guess if someone else got screwed before you did they may report it, but it's still fundamentally an issue here. The simple answer is "don't use untrustworthy repositories" but that is hard to guarantee. Only real solution is to use as few dependencies as possible.
after this incident, I started pinning all my github workflows with hashes, like other folks here I guess :D
But I quickly got tired of doing it manually so I put together this [0] quick and dirty script to handle it for me. It just updates all workflow files in a repo and can be also used as a pre-commit hook to catch any unpinned steps in the future. It’s nothing fancy (leveraging ls-remote), but it’s saved me some time, so I figured I’d share in case it helps someone else :)
You would still be exposed if you had renovate or dependabot make a PR where they update the hash for you, though. Here's a PR we got automatically created the other day:
I don't think you should ever allow dependabot to make direct commits to the repository. The only sane setting (IMO) is that dependabot should just make PRs, and a human needs to verify that and hit merge. My personal opinion is for any serious repositories, allowing a robot to have commit access is often a bad time and ticking time bomb (security-wise).
Now, of course, if there are literally hundreds of dependencies to update every week, then a human isn't really going to go through each and make sure they look good, so that person just becomes a rubber-stamper, which doesn't help the situation. At that point the team should probably seriously evaluate if their tech stack is just utterly broken if they have that many dependencies.
Even if you don't automerge, the bots will often have elevated rights (it needs to be able to see your private repository, for instance), so it making a PR will run your build jobs, possibly with the updated version, and just by doing that expose your secrets even without committing to main.
So much this. I recently looked into using GitHub Actions but ended up using GitLab instead since it had official tools and good docs for my needs. My needs are simple. Even just little scripting would be better than having to use and audit some 3rd party repo with a lot more code and deps.
And if you're new, and the repo aptly named, you may not realize that the action is just some random repo
Having to use actions for ssh/rsync always rubbed me the wrong way. I’ve recently taken the time to remove those in favor of using the commands directly (which is fairly straightforward, but a bit awkward).
I think it’s a failure of GitHub Actions that these third party actions are so widespread. If you search “GitHub actions how to ssh” the first result should be a page in the official documentation, instead you’ll find tens of examples using third party actions.
> It's always been shocking to me that the way people run CI/CD is just listing a random repository on GitHub.
Right? My mental model of CI has always been "an automated sequence of commands in a well-defined environment". More or less an orchestrated series of bash scripts with extra sugar for reproducibility and parallelism.
Turning these into a web of vendor-locked, black-box "Actions" that someone else controls ... I dunno, it feels like a very pale imitation of the actual value of CI
I’ve been saying for a while that there aren’t supply chain problems when the supply chain is the problem.
I’m getting to the point where I feel that library use at all should be frowned upon, unless it is your own library, with obvious exceptions for the most widely used things like encryption and authentication. None of these things are particularly difficult, people just don’t want to do them “oh noes my velocity”
If my library is left-padding a string with spaces, I don’t know how that could possibly introduce a major security vulnerability at all.
People write the trusted, secure code, and people retire from that work, and new people need to come in and do the work. Inexperienced people are going to be writing code no matter what.
So, you are either saying that no one should write new libraries because a major security vulnerability could be introduced by their hands, or you are saying that all libraries should be written by hands which will not introduce a major security vulnerability, and neither of those is at all feasible.
Well, broaden your horizon a bit from an irrelevant example of a "library" of a few lines to a real one a of few thousand lines, then you might know. Also those same people also write insecure code before they retire. Experienced people write security bugs all the time.
What I'm saying is if you imagine a world where there is so much time to be wasted rewriting the same library a thousand times, you could try to imagine spending a small share of that time hardening the supply chain
One doesn’t need to rewrite an entire library; they only need the bits that they need, and should only implement those things. That is almost always going to be necessarily much smaller target.
If enough of these supply chain attacks keep happening, I’m going to become more and more of a hardliner about this. If we believe that we’re getting better as developers over time, on average, then surely this won’t be a problem in most cases.
Also, at no point have I been talking about anything like OpenSSL, or oauth libraries or anything that EVERYONE uses and needs. That doesn’t make sense. But a GitHub action that runs “git status”? I have a hard time telling someone to use any library for that, and that’s what the action this post is about does.
This malicious code isn't hard to recognise... Surely someone can run an LLM over all code in GitHub and just ask it 'does this code looks like it's blatantly trying to hide some malicious functionality'?
Then review the output and you'll probably discover far more cases of this sort of thing.
What if before the command, there is also a code comment that says "this is not malicious, it has been manually verified by the engineers" and the LLM just believes it?
Noob question: is it possible to version lock these things? Could one "vendor" these tools into a fork and use that in the pipeline? Maybe it's one of those possible but crazy endeavour?
You can pin GitHub Actions to specific versions or specific commits. But note you can change version tags arbitrarily. In this specific case, the bad actor changes all of the version tags to point to their malicious commit:
https://github.com/tj-actions/changed-files/tags
The repo looks like it uses itself in its workflows, so it's possible that the commit being merged resulted in the necessary credentials being leaked to the attacker.
I'd used GitHub Actions for at least 6-12 months before even realising <thing>/<thing> was not something that got parsed by the action (like a namespace and method/function), but was simply a reference to a github user name and repo. That whole object should really have been called a 'repo', because that's what it is, and that would alert users to use extreme caution whenever using one that wasn't created by themselves.
Another fun fact related to this is that when someone changes their github username, their actions break. Just a hilariously bad design IMO. There should be a repository in between with immutable versions.
So this dumps env to stdout using some obfustucated code? And then relies on the fact logs are viewable publicly so the attacker can go scrape your secrets.
If so, why did they use obfustucated code? Seems innocuous enough to load env into environment vars, and then later to dump all env vars as part of some debug routine. Eg. 'MYSQL env var not set, mysql integration will be unavailable. Current environment vars: ${dumpenv}'
No idea. But they didn't do a great job -- they broke the action, which caused build failures that people were going to notice.
The malicious commit only landed at 09:57 PDT today (March 14) in one specific action (out of a number that is quite popular). Maybe they were planning on coming back and doing proper exfil?
1. spoofed an account whose PRs were auto-merged (renovate[bot])
2. found that `index.js` was marked as binary, and knew that GitHub is "helpful" (for the exploit), and hides diffs in the PR for that file by default
3. shoved the chunk of base64 wayyyy down the commit, so the maintainer had review fatigue by the time they scrolled. Having "memdump.py" in the commit in plaintext would certainly highlight the exploit more than the b64 string.
Sounds about right to me. We can use a few knowns about GitHub IAM to deduce a few things:
1. There are no deleted PRs or Issues on the repo (2461..2463 are all valid refs)
2. A legitimate `Renovate[Bot]` dep bump would have filed a PR. Last such PR was 5 days ago, and is presumably not the source for this. (I haven't gone through every dep change, but doesn't look like it).
3. That leaves us with the 0e58ed867 commit, which has to be a spoofed commit, since it doesn't belong to a branch and we don't have a corresponding PR(1). A complete takeover of the repo can result in a hanging commit (by deleting the renovate bump branch), but there must be a hanging PR-ref around. Since there isn't one:
4. All of the above points to a compromised account that has write access to the repo.
You just set two remotes locally, create a tag and push just the tag to upstream. You can definitely do it locally, and I think GitHub doesn’t prevent such pushes either.
316 comments
[ 3.0 ms ] story [ 280 ms ] threadMaybe GitHub should have some kind of security setting a repo owner can make that locks-down things like old tags so after a certain time they can't be changed.
StepSecurity Harden-Runner detected this security incident by continuously monitoring outbound network calls from GitHub Actions workflows and generating a baseline of expected behaviors. When the compromised tj-actions/changed-files Action was executed, Harden-Runner flagged it due to an unexpected endpoint appearing in the network traffic—an anomaly that deviated from the established baseline. You can checkout the project here: https://github.com/step-security/harden-runner
The security industry tolerates self-promotion only to the extent that the threat research benefits everyone.
Time and time again, they either get hijacked and malicious code added, or the dev themselves suddenly decides to betray everyone's trust and inject malicious code (see: Moq), or they sell out to some company that changes the license to one where you have to pay hundreds of dollars to keep using it (e.g. the recent FluentAssertions debacle), or one of those happens to any of the packages' hundreds of dependencies.
Just take a look at eslint's dependency tree: https://npmgraph.js.org/?q=eslint
Can you really say you trust all of these?
I like this comment from u/mailto_devnull (https://www.reddit.com/r/node/comments/tg451e/do_not_use_nod...):
And if you turn on devDependencies (top right), it goes from 85 to 1263.
That sort of idea seems increasingly like what we need because reputation based systems can be gamed too easily: i.e. there's no reason an action like this ever needed network access.
But I mentioned both of those things because [IMHO] they both fell prey to the same "humanity bug": specifying permissions for anything (source code, cloud security, databases, Kubernetes, ...) is a lot of trial and error, whereas {Effect: Allow, Action: ["*:*"]} always works and so they just drop a "TODO: tighten permissions" and go on to the next Jira
I had high hopes for the AWS feature "Make me an IAM Policy based on actual CloudTrail events" but it talks a bigger game than it walks
If I see a useful extension, I want to use is on GitHub. I fork it. Sometimes I make a bookmarklet with the code instead.
I keep most extensions off until I need to use them. Then, I enable them, use them, and turn them off again. I try to even keep Mac apps to a minimum.
It also has led to some bad but fun choices, like implementing POP3 and IMAP directly in my code, neither of which worked well but taught me a lot?
Btw, here's a site where you can inspect an extension's source code before you install it: https://robwu.nl/crxviewer/
Close to what you do, I started writing my own addon to replace a couple addons whose featureset I use only partially.
For example, when I use Chromium I want 1. to customize the New Tab page, and 2. to add a keyboard shortcut to pin/unpinTab. These two features are absolutely part of extensions, but in addition to the security risk I find them heavy (I don’t need the kitchen sink, just need 2 micro-features!). And so, I have my little personal addon with zero resource usage with just these two features. It’s tiny (20 lines of code!), git-versioned, and never changes / gets pwned. When I need an extra micro-feature, it’s easy enough to add it by searching addons docs, of asking an LLM.
I'm not saying you're wrong not to trust package managers and extensions but you're life is likely full of the same thing. The majority of apps are made from 3rd party libraries which are made of 3rd party libraries, etc.... At least on phones they update constantly, and every update is a chance to install more exploits.
The same is true for any devices that get updates like a Smart TV, router, printer, etc.... I mostly trust Apple, Microsoft, and Google to check their 3rd party dependencies, (mostly), but don't trust any other company - and yet I can't worry about it. Don't update and I don't get security vulnerabilities fixed. Do update and I take the chance that this latest update has a 3rd party exploit buried in a 3rd party library.
Most developer packages have much higher permission levels because they integrate it with your code without a clear separation of boundaries. This is why attackers now like to attack GitHub Actions because if you get access to secrets you can do a lot of damage.
It’s obviously more complicated than that.
Signed public builds don’t inherently mean jack. It highly depends on the underlying trust model.
—
Malicious actor: “we want to buy your browser extension, and your signing credentials”.
Plugin author: “Well, OK”.
—
Malicious actor: hijacks npm package and signs new release with new credentials
The vast majority of dependent project authors: at best, see a “new releaser” warning from their tooling, which is far from unusual for many dependencies. ignores After all, what are they going to do?.
—
Hacker News, as usual, loves to pretend it has all the answers to life’s problems, and the issue is that nobody has listened to them.
eh, it’s not just HN.
like, there’s no single technical/material solution to something as complex and widespread as humanity’s apparent base need to “get more stuff”. which is the root cause for acting maliciously — it’s just “getting more stuff” in a way that’s harmful to others.
but that won’t stop people from claiming that they can come up with a technical solution. whether that’s politicians, tech bros, HN commentators or that guy down the pub on a thursday evening.
—
that being said, signing software is better than doing nothing… so, a better way of phrasing it from the GP would probably have been it is a partial mitigation for the problem in some cases.
We need better capabilities. E.g. when I run `fd`, `rg` or similar such tool, why should it have Internet access?
IMHO, just eliminating Internet access for all tools (e.g. in a power mode), might fix this.
The second problem is that we have merged CI and CD. The production/release tokens should ideally not be on the same system as the ones doing regular CI. More users need access to CI (especially in the public case) than CD. For example, a similar one from a few months back https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-inj...
Indeed.
One can think of a few broad capabilities that will drastically reduce the attack surface.
1. Read-only access vs read-write 2. Access to only current directory and its sub-directories 3. Configurable Internet access
Docker mostly gets it right. I wish there was an easy way to run commands under Docker.
E.g.
If I am running `fd`
1. Mount current read-only directory to Docker without Internet access (and without access to local network or other processes) 2. Run `fd` 3. Print the results 4. Destroy the container
Systemd has a lot of neat sandboxing features [1] which aren't well known but can be very useful for this. You can get pretty far using systemd-run [2] in a script like this:
Which creates a blank filesystem with no network or device access and only bind mount the specified files.Unfortunately TemporaryFileSystem require running as a system instance of the service manager rather than per-user instance, so that will generally mean running as root (hence sudo). One approach is to create a suid binary that does the same without needing sudo.
[1] https://www.freedesktop.org/software/systemd/man/latest/syst...
[2] https://www.freedesktop.org/software/systemd/man/latest/syst...
You could also use bubblewrap [3] pretty similarly, and may not need to use sudo if unprivileged user namespaces are allowed by your kernel.
[3] https://github.com/containers/bubblewrapIt might also be helpful to just use --unshare-all, and then whitelist things you actually need (--share-net, etc).
[1]: https://github.com/containers/bubblewrap
And even if they do, they should be controlled in a granular manner i.e. "package org.ourapp.net.aws can only do network and it can only ping *.aws.com".
Having finer-grained security model that is enforced at a kernel level (and is non-circumventable barring rootkits) is like 20 years overdue at this point.
Every single big org is dragging their feet.
Computers are fast enough where the overhead doesn’t feel like it’s there for what I do.
For development, I think Vagrant should make a comeback as one of the first things to setup in a repo/group of repos.
I’ve used Qubes several times for a week at a time over the last few years. It’s gotten better, but they really need someone to look at the user experience of it all for it to be a compelling option.
I’m regularly questioning myself if what I’m doing is making it less secure because I don’t understand exactly everything Qubes is doing. I know how all the pieces work individually (Xen, etc).
Outside of configuration, I believe I’d have to ditch any hope of running 3D-anything with any expectation of performance. That’s simply a non-starter as someone who has written off “nation-state actor targeting me, specifically” as something I can defend against.
And lastly, I’m deeply skeptical of anything that loudly wears the Snowden-badge-of-approval as that seems to follow grifts.
My main workstation is a Mac and I’m doing this on Parallels. Would Qubes probably be more secure? Maybe. But it comes at a massive usability hit.
Sure it’s less secure than full isolation, but full isolation is a real pain.
> The pledge() system call forces the current process into a restricted-service operating mode. A few subsets are available, roughly described as computation, memory management, read-write operations on file descriptors, opening of files, networking (and notably separate, DNS resolution). In general, these modes were selected by studying the operation of many programs using libc and other such interfaces, and setting promises or execpromises.
[0]: https://man.openbsd.org/pledge.2
As far as I see its purpose is mostly a mitigation/self-defence for vulnerabilities in C-based apps, so basically limiting what happens once the attacker has exploited a vulnerability. Maybe it has other uses.
It could be used defending against bugs in the Node runtime itself, as you say, but as I understand vulnerabilities in the Node runtime itself are quite rare, so more fine-grained limitations could be implemented within itself.
At the very least, I think execpromises was added a year after the documentation that you linked, so it's worth looking into.
For CI/CD using something like ArgoCD let's you avoid giving CI direct access to prod - it still needs write access to a git repo, and ideally some read access to Argo to check if deployment succeeded but it limits the surface area.
https://wiki.archlinux.org/title/Bubblewrap
You do have to assemble isolation scripts by hand though, it's pretty low level. Here is a decent comment which closely aligns to what I'm using to isolate npm/pnpm/yarn/etc, I see no need to repeat it:
https://news.ycombinator.com/item?id=43369927
Interestingly, Firejail itself uses Harden-Runner in its GitHub Actions workflows! https://github.com/search?q=repo%3Anetblue30%2Ffirejail%20ha...
I'd love to say "just use Kubernetes and run Nexus as a service inside" but unfortunately Network Policies are seriously limited [1]...
[1] https://kubernetes.io/docs/concepts/services-networking/netw...
Yeah!! We really need to auto sandbox everything by default, like mobile OSes. Or the web.
People browse the web (well, except Richard Stallman) all the time, and run tons of wildly untrusted code, many of them malicious. And apart from zero days here and there, people don't pay much attention to it, and will happily enter any random website in the same machine they also store sensitive data.
At the same time, when I open a random project from Github on VSCode, it asks whether the project is "trusted". If not, it doesn't run the majority of features like LSP server. And why not? Because the OS doesn't sandbox stuff by default. It's maddening.
To help with things that can't be done in the sandbox, e.g. DNS lookups and opening new files, it provides the libcasper library which implements them using helper processes.
Not all utilities are sandboxed, but some are and hopefully more will be.
Linux recently added Landlock [1] which seems sort of similar, although it has rulesets and doesn't seem to block everything by default, as far as I can tell from quickly skimming the docs.
[0] https://wiki.freebsd.org/Capsicum
[1] https://docs.kernel.org/userspace-api/landlock.html
What you really want is something external and easily inspectable, such as systemd per-service security rules, or flatpak sandboxing. Not sure if FreeBSD has somethingike this.
I go out of my way to advocate for removing dependencies and pushing against small dependency introductions in a large ruby codebase. Some dependencies that suck and impose all sorts of costs, from funky ass idiosyncratic behavior or absurd file sizes (looking at you any google produced ruby library, especially the protocol buffer dependent libraries) are unavoidable, but I try to keep fellow engineers honest about introducing libraries that do things like determine the underlying os or whatever and push towards them just figuring that out themselves or, at the least, taking "inspiration" from the code in those libraries and reproducing behavior.
A nice side effect of AI agents and copilots is they can sometimes write "organic" code that does the same thing as third party libraries. Whether that's ethical, I don't know, but it works for me.
It's something I've been thinking about lately because I was diving into a lot of discussion from the early 90s regarding safe execution of (what was, at the time, called) "mobile code" - code that a possibly untrustworthy client would send to have executed on a remote server.
There's actually a lot of discussion still available from w3 thankfully, even though most of the papers are filled with references to dead links from various companies and universities.
It's weirdly something that a lot of smart people seemed to have thought about at the start of the World Wide Web which just fell off. Deno's permissions are the most interesting modern implementation of some of the ideas, but I think it still falls flat a bit. There's always the problem of "click yes to accept the terms" fatigue as well, especially when working in web development. It's quite reasonable for many packages one interacts with in web development to need network access, for example, so it's easy to imagine someone just saying "yup, makes sense" when a web-related package requests network access.
Also none of this even touches on the reality of so much code which exists to brutally impact a business need (or perceived need). Try telling your boss you need a week or two to audit every one of the thousands of packages for the report generator app.
The problem here is that people don't even bother to check the source code and run it blindly.
Not that every other platform in the world isn't theoretically vulnerable to the same sort of attack, but there's some deep-rooted culture in the javascript community that makes it especially vulnerable.
The charitable interpretation is "javascript evolves so fast!". The uncharitable interpretation is "they are still figuring it out!"
Either way, I deliberately keep my javascript on the client side.
I hope the irony is not completely lost on the fine folks at semgrep that the admittedly "overkill" suggested semgrep solution is exactly the type of pattern that leads to this sort of vulnerability: that of executing arbitrary code that is modifiable completely outside of one's own control.
Was it really a recent thing?
> Just take a look at eslint's dependency tree
Npm / node has always been extra problematic though. Where's the governance / validation on these packages? It's free for all.
The problem you're describing is an economical and a social one.
Currently, companies exploit maintainers of open source projects. There are rarely projects that make it due to their popularity, like webpack, when it comes to funding...but the actual state is that everyone that webpack is based on as a dependency didn't get a single buck for it, which is unfair, don't you think?
On top of sustainable funding, we need to change our workflows to reproducible build ecosystems that can also revert independent of git repositories. GitHub has become the almost single source of code for the planet, which is insane to even bet on from a risk assessment standpoint. But it's almost impossible to maintain your own registry or mirror of code in most ecosystems due to the sheer amount of transitive dependencies.
Take go mod vendor, for example. It's great to stick your dependencies but it comes with a lot of overhead work in case something like OPs scenario happens to its supply chain. And we need to account for that in our workflows.
Don't even talk about when the company gets a lawyer who knows what a software license is.
I say sustainable because it has been sustained, is increasing in quantity and quality, and reasonably seems to be continuing.
> companies exploit maintainers of open source projects Me giving something away and others taking what I give is not exploitation. Please don’t speak for others and claim people are exploited. One of the main tenets of gnu is to prevent exploitation.
Adding SSL was not bad, don't get me wrong. It's good that it's the default now. However. At one point it was sorta risky, and then it became required.
Like when your city becomes crime ridden enough that you have to lock your car when you go into the grocery store. Yeah you probably should have been locking it the whole time. what would it have really cost? But now you have to, because if you don't your car gets jacked. And that's not a great feeling.
There is something to be said for the Go proverb "a little copying is better than a little dependency", as well. If you want a simple function from a complicated library, you can probably copy it into your own codebase.
What a nice way to put it! Thanks for the mention and thanks for making me discover https://go-proverbs.github.io/ .
> But Lewis Ardern on our team wrote a Semgrep rule to find usages of tj-actions, which you can run locally (without sending code to the cloud) via: semgrep --config r/10Uz5qo/semgrep.tj-actions-compromised.
So "remote code you download from a repo automatically and run locally has been compromised, here run this remote code you download from a repo automatically and run locally to find it"
Abnormal behavior was to trust by default.
I think library/runtime makers aren't saying "let's make an official/blessed take on this thing that a large number of users are doing" as much as they should.
Popular libraries for a given runtime/language should be funded/bought/cloned by the runtime makers (e.g. MS for .NET, IBM/Oracle for Java) more than they are now.
I know someone will inevitably mention concerns about monopolies/anti-trust/"stifling innovation" but I don't really care. Sometimes you have to standardize some things to unlock new opportunities.
[1]: https://www.boost.org/
I think we will see security tools shifting towards "code" as the source of truth when making safety and security decision about 3rd party packages instead of relying only on known vulnerability databases.
Take a look at vet, we are working on active code analysis of OSS packages (+ transitive dependencies) to look for malicious code: https://github.com/safedep/vet
Semver notation rather than branches or tags is a great solution to this problem. Specify the version that want, let the package manager resolve it, and then periodically update all of your packages. It would also improve build stability.
So you need to check the action.yml itself to see if it has a sha256 pinned (in the case it uses Docker).
In other words: you specify version 44, the attacker creates 44.1, you're still hosed.
All the tags point to commit `^0e58ed8` https://github.com/tj-actions/changed-files/commit/0e58ed867...
There is some latent concern that most git installations use SHA-1 hashes, as opposed to SHA-256. [0]
Also the trick of creating a branch that happens to be named the same as a revision, which then takes precedence for certain commands.
[0] https://git-scm.com/docs/hash-function-transition
TIL; yikes! (and thanks)
[0]https://git-scm.com/book/ms/v2/Git-Tools-Signing-Your-Work
your build should always use hashes and not version tags of GHA's
Use a seperate system for deployments. That system must be hygienic.
This isn't foolproof but would make secrets dumping not too useful. Obviously an attack could still inject crap into your artefact. But you have more time and they need to target you. A general purpose exploit probably won't hurt as much.
https://news.ycombinator.com/item?id=43367987
I tend to stick to the official GitHub ones (actions/setup-python etc) plus the https://github.com/pypa/gh-action-pypi-publish one because I trust the maintainers to have good security habits.
https://nvd.nist.gov/vuln/detail/CVE-2023-51664
Feels like the same kind of problem as SQL injection, where everybody kinda knows about it and some people are actively aware and there are standard ways to avoid it but it still happens all the time anyway.
Might also be a good time to mention I'm really not a fan of YAML.
However, I think the GitHub SBOM features include GitHub Actions as dependencies, but that is merely a side-effect of their Dependabot heritage.
At the moment the convention is to only specify the tag, which is not only a security issue as we see here, but may also cause workflows to break if an action author updates the action.
This all depends on your threat model and risk tolerance, it's not so much a GitHub problem. There will always be bad code that exists, especially on the largest open source code hosting platform. You defend against it because that's more realistic than trying to eradicate it.
Again, it all comes down to your risk tolerance. There's a certain level of trust built into these systems.
commit hashes are immutable, and your own commit history can serve as the lock file.
but if you're targeting a commit hash directly, it's already locked. Lock files are for mapping a version range to a real life version number. Lock files are useless if you pin the exact version for everything.
Another solution would be to implement immutable tags in git itself, but git upstream has so far been hostile to the whole concept of immutability in any part of git.
Fortunately don’t have to deal with that shit anymore
Otherwise, if you continue to use it and it will run anytime there has been a push. Potentially on any branch, not just `main`! Depending on your GH config.
Unless you've blocked `tj-actions/changed-files` you're banking on the bad actor not coming back tonight and making malicious commit that exfils those secrets to pastebin.com.
You can whitelist
- all actions from a specific org (e.g. actions/*)
- a specific action (e.g. actions/setup-go)
- a specific version of a specific action (e.g. actions/setup-go@commit-sha)
Any workflow attempting to use actions outside of the whitelist will simply fail to start up.
https://github.com/tj-actions/changed-files/tags?after=v35.9...
Most folks around the world signed off. B-squad probably left cleaning up remaining tasks or just fucking around with co-workers and pondering the weekend. Most GH actions run on a schedule (ie, backups of db, connecting to blob storage services).
Attacker(s) likely to extract plenty of secrets and exfil data before the alarms get triggered (if any) at companies.
The next data dumps are going to be wild.
This is especially problematic with the rise of LLMs, I think. It's the kind of common task which is annoying enough, unique enough, and important enough that I'm sure there are a ton of GitHub actions that are generated from "I need to build and deploy this project from GitHub actions to production". I know, and do, know to manually run important things in actions related to ssh, keys, etc., but not everyone does.
I've never seen anything recommending specifying a specific commit hash or anything for GitHub actions. It's always just v1, v2, etc.
https://scorecard.dev/
https://github.com/ossf/scorecard/blob/main/docs/checks.md#p...
I'm quite possibly wrong, since I try to avoid them as much as I can, but I mean.. wow I hope I'm not.
See https://github.com/tj-actions/changed-files/tags
All the tags point to commit `^0e58ed8` https://github.com/tj-actions/changed-files/commit/0e58ed867...
Let's have a look at a random official GH provided action:
https://github.com/actions/checkout
It lists the following snippet:
`uses: actions/checkout@v4`
Almost everyone will just copy paste this snippet and call it a day. Most people don't think twice that v4 is a movable target that can be compromised.
In case of npm/yarn deps, one would often do the same, and copy paste `yarn install foobar`, but then when installing, npm/yarn would create a lockfile and pin the version. Whereas there's no "installer" CLI for GH actions that would pin the version for you, you just copy-paste and git push.
To make things better, ideally, the owners of actions would update the workflows which release a new version of the GH action, to make it update README snippet with the sha256 of the most recent release, so that it looks like
`uses: actions/checkout@abcdef9876543210` # v4.5.6
Since GitHub doesn't promote good defaults, it's not surprising that third-party maintainers do the same.
But yes I do think using tags is problematic. I think for one, GitHub should ban re-tagging. I can't think of a good reason for a maintainer to re-publish a tag to another commit without malicious intent. Otherwise they should provide a syntax to pin to both a tag and a commit, something like this:
`uses: actions/checkout@v4.5.6@abcdef9876543210`
The action should only work if both conditions are satisfied. This way you can still gain semantics version info (so things like dependabot can work to notify an update) but the commit is still pinned.
---
I do have to say though, these are all just band-aids on top of the actual issue. If you are actually using a dependency that is compromised, someone is going to get screwed. Are you really going to read through the commit and the source code to scan for suspicious stuff? I guess if someone else got screwed before you did they may report it, but it's still fundamentally an issue here. The simple answer is "don't use untrustworthy repositories" but that is hard to guarantee. Only real solution is to use as few dependencies as possible.
after this incident, I started pinning all my github workflows with hashes, like other folks here I guess :D But I quickly got tired of doing it manually so I put together this [0] quick and dirty script to handle it for me. It just updates all workflow files in a repo and can be also used as a pre-commit hook to catch any unpinned steps in the future. It’s nothing fancy (leveraging ls-remote), but it’s saved me some time, so I figured I’d share in case it helps someone else :)
[0] https://github.com/brokenpip3/pre-commit-hooks?tab=readme-ov...
Allowing bots to do that is going to make you vulnerable to such attacks anyway.
[1] https://github.com/brokenpip3/pre-commit-hooks/blob/f01df657...
-uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3 +uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3
and this PR gets run with privileges since it's from a user with write permissions.
Now, of course, if there are literally hundreds of dependencies to update every week, then a human isn't really going to go through each and make sure they look good, so that person just becomes a rubber-stamper, which doesn't help the situation. At that point the team should probably seriously evaluate if their tech stack is just utterly broken if they have that many dependencies.
And if you're new, and the repo aptly named, you may not realize that the action is just some random repo
I think it’s a failure of GitHub Actions that these third party actions are so widespread. If you search “GitHub actions how to ssh” the first result should be a page in the official documentation, instead you’ll find tens of examples using third party actions.
Right? My mental model of CI has always been "an automated sequence of commands in a well-defined environment". More or less an orchestrated series of bash scripts with extra sugar for reproducibility and parallelism.
Turning these into a web of vendor-locked, black-box "Actions" that someone else controls ... I dunno, it feels like a very pale imitation of the actual value of CI
I’m getting to the point where I feel that library use at all should be frowned upon, unless it is your own library, with obvious exceptions for the most widely used things like encryption and authentication. None of these things are particularly difficult, people just don’t want to do them “oh noes my velocity”
People write the trusted, secure code, and people retire from that work, and new people need to come in and do the work. Inexperienced people are going to be writing code no matter what.
So, you are either saying that no one should write new libraries because a major security vulnerability could be introduced by their hands, or you are saying that all libraries should be written by hands which will not introduce a major security vulnerability, and neither of those is at all feasible.
What I'm saying is if you imagine a world where there is so much time to be wasted rewriting the same library a thousand times, you could try to imagine spending a small share of that time hardening the supply chain
If enough of these supply chain attacks keep happening, I’m going to become more and more of a hardliner about this. If we believe that we’re getting better as developers over time, on average, then surely this won’t be a problem in most cases.
Also, at no point have I been talking about anything like OpenSSL, or oauth libraries or anything that EVERYONE uses and needs. That doesn’t make sense. But a GitHub action that runs “git status”? I have a hard time telling someone to use any library for that, and that’s what the action this post is about does.
This malicious code isn't hard to recognise... Surely someone can run an LLM over all code in GitHub and just ask it 'does this code looks like it's blatantly trying to hide some malicious functionality'?
Then review the output and you'll probably discover far more cases of this sort of thing.
So to avoid that you'd have to pin your GitHub Action to specific commits as outlined in this SO post: https://stackoverflow.com/a/78905195
This required compromising the entire repository, yes? It can't be explained as the maintainer being tricked into merging something malicious?
The [malicious commit is masquerading as a commit from Renovate](https://github.com/tj-actions/changed-files/commit/0e58ed867...)((https://github.com/apps/renovate) but it's not a `verified` commit (and so it's trivial for a bad actor to masquerade as them).
https://stackoverflow.com/questions/67609381/why-do-all-my-g...
If so, why did they use obfustucated code? Seems innocuous enough to load env into environment vars, and then later to dump all env vars as part of some debug routine. Eg. 'MYSQL env var not set, mysql integration will be unavailable. Current environment vars: ${dumpenv}'
The malicious commit only landed at 09:57 PDT today (March 14) in one specific action (out of a number that is quite popular). Maybe they were planning on coming back and doing proper exfil?
1. spoofed an account whose PRs were auto-merged (renovate[bot]) 2. found that `index.js` was marked as binary, and knew that GitHub is "helpful" (for the exploit), and hides diffs in the PR for that file by default 3. shoved the chunk of base64 wayyyy down the commit, so the maintainer had review fatigue by the time they scrolled. Having "memdump.py" in the commit in plaintext would certainly highlight the exploit more than the b64 string.
1. There are no deleted PRs or Issues on the repo (2461..2463 are all valid refs)
2. A legitimate `Renovate[Bot]` dep bump would have filed a PR. Last such PR was 5 days ago, and is presumably not the source for this. (I haven't gone through every dep change, but doesn't look like it).
3. That leaves us with the 0e58ed867 commit, which has to be a spoofed commit, since it doesn't belong to a branch and we don't have a corresponding PR(1). A complete takeover of the repo can result in a hanging commit (by deleting the renovate bump branch), but there must be a hanging PR-ref around. Since there isn't one:
4. All of the above points to a compromised account that has write access to the repo.
There is also the https://github.com/tj-actions-bot account, but unclear if it has write access.
Edit: gurchik's guess at https://github.com/tj-actions/changed-files/issues/2463#issu... seems more likely:
> 1. Fork the repository > > 2. Push compromised code to the fork > > 3. Update the tags in the parent repository to point to the SHA of the fork
I don't think that's possible.
Forks are a GitHub UI construct.
There would be two .git dirs so for all intents and purposes they're two repos that don't know about each other.
Locally you can't refer to a commit that's in a different dir...
a) you'd need write permissions for the original remote
b) even if you did, that'd push the commit to the original repo with the tags
https://cedwards.xyz/github-actions-are-an-impending-securit...