Totally agreed. However, when do we stop making this mistake? I think it's worth a callout when a large organization designs a dependency management system with such an obvious flaw on the "happy path".
A known collision brings a hash algorithm to be "not secure" for cryptographic use. It doesn't mean there is a reliable way to come up with an equivalent hash for different data.
That said, hashes like md5 are still plenty used in ETags for http 304s all the time despite being considered insecure for cryptographic use.
It's not secure for many applications. There are various practical collision attacks on it.
However, to rewrite history you need a preimage attack. In other words, you need to find a value X which hashes to the same hash value y which already exists h(X)=y
These are not currently practical. So SHA1 is certainly not recommended, but isn't completely broken yet.
There are collision attacks against SHA-1. But the currently known collision attacks aren't a big problem for git for a variety of reasons.
Git uses hashes that include more data than just the hash of the content of an object. Meaning the existing known collisions don't work and you'd have to develop collisions specifically to target git.
Additionally, the current known collision attacks require generating the two colliding contents as a pair. You can't take an existing document and then generate a collision for that document. Further, the technique is not particularly good for text documents because you need to include some data to force the collision. This is easily missed in binary documents but not the text documents most people are committing into git repos (granted you can put binary files into git and that would still work but the other reasons for this not being very useful still apply).
Git also has code to deal with possible collisions and will fall back to full comparison and will fail a commit that creates a collision. Finally, generating these document pairs leaves detectable patterns in the content. GitHub scans for these patterns as part of their process.
You hand-wave the difference between collision resistance and second-preimage resistance most away...
Just a recap: generating two files who's hashes collide takes 2^80 tries if SHA-1 wasn't broken, whereas finding a file with the same hash as a specific, externally-determined file takes 2^160 tries.
This also applies to most library, docker (as mentioned) and basically anything you use that's 3rd party. I suppose it varies on your level of paranoia, though honestly if you rely on a github action in your production flow you should fork the project. Never mind doing something malicious, they could just as easily delete the entire repo and now your critical feature you needed to deliver are blocked because the library you were relying one has disappeared.
Anything you use in a critical path you should control. If you're managing your own fork, nobody will inject bad code except yourself.
Just my 2 cents. Also, the things that github 3rd party actions are doing is usually not that complicated. I mean how many different ways can you publish a docker image or deploy an artifact to S3. Once it works, it either works or doesn't. It's not a programming library where optimizations are created or we fixed a security vulnerability (though I suppose that can come up ).
We definitely standardize runtime and compiler versions across runtime environments and build servers and have planned upgrade paths with rollback options.
Do I do this for hobby projects? No, of course not. If my toy projects break, meh. But for production? Where breakage actually costs money? Damn right I do.
It's not only breakage, though. You want a hermetic build system, so you can build software reproducibly and consistently. There are few things more "fun" than building "the same" code on two machines, and one executable has a bug while the second doesn't. (OK. Compiler bugs are even more fun)
While I understand the "fork everything you use" policy, github should make this process easier. Even though it is "free", it pollutes one's account/profile with hundreds of packages that are not really 'forks' with independent or alternate functionality.
Many automated vendor/package management tools will not automate pulling newer changes from upstream. If this was instead a simple a read-only version/tag, it is easy for those tools to provide upgrade/update path.
As a side effect this adds more organization icons to your Github profile, making it appear as if you're such an important OSS contributor that you're an official member of the Username Forks Alliance :)
That's the sort of tough-guy prepper-with-a-slide-rule speech that people (apparently) just can't get enough of. But does it mean anything?
There is no bright line delimitating what's on the "critical path" and what's not. Your servers? The electric grid? The datacenter? The backbone(s) it's connected to? The judicial system making contracts enforceable? NATO? Name any two that straddle the line, and anybody can come up with something in between them, because it's a continuum.
GitHub could completely disappear and it wouldn't cause mass societal chaos or the breakdown of civilization. Those other things you listed, if they disappeared, there's quite a good chance they would cause mass societal chaos or the breakdown of at least certain civilizations. If the judicial system suddenly disappears, I reckon we'll have other things to worry about besides how it affects our build-chains.
The lack of a bright line doesn't make the idea meaningless. It's just a judgement call.
It's not unreasonable to be more concerned about an open source library disappearing than say, postgresql or mysql disappearing. No macho posturing required.
But people who complain about this stuff usually don't frame it that way-- absolutist statements like from the GP: "Anything you use in a critical path you should control." are common. The message is typically "I realize something fundamental that you don't and I'm owning all my dependencies and you don't own any of yours," rarely "I made a different choice about dependency ownership and here are the reasons I think that's appropriate for you too."
You seemed to take a pretty holistic, system-wide view of "critical path" which is fine. Do we know if the other person meant that?
Let's say I'm working front-end web development. My "critical path" might include: js bundler, some kind of application framework, various libs, etc. The scope of my work (and authority/responsibility) informs what I can reasonably have under my control.
Welcome to a world were engineers aren't paid to copy code from Stack Overflow, but to make actual judgment calls.
What's the chance of the grid going down? What's the cost of it? Is the expected value significantly higher than a power generator? Congrats, you just bought a power generator.
Backbone down would cause you huge financial losses? Well, whaddaya say, let's build more backbone.[1]
Any redundancy that costs significantly less than the losses it mitigates will be built. The only reason we're not seeing a private NATO yet is that it costs more than pretty much any company makes. But 5,000 people security forces? Damn skippy those exist[2]
You can address image layers by sha256 hash.
Do a docker pull of the image and the log will tell you the hash. Then you can do docker pull myimage@sha256:abc123def456......
I usually use these in my deployments since they can't be modified, it will require a new deploy.
However, you'll sometimes want to update the commit hash after the action's maintainer releases new code. If you don't then you run the risk of running an old action that has vulnerable components or bugs.
The easiest workflow for that should be Dependabot's updater for GitHub actions:
https://dependabot.com/github-actions/
Has anyone tried that approach for GitHub actions?
You still need to review the action's code before you use it, and every time you update to a new commit hash. But this approach protects you from automatically running new malicious code pushed to master/re-tagged.
Sorry, that whole github actions concept gives me the screaming heeby-jeebies.
We're really not even pulling the dependency locally, we're just going to run someone else's code from their repo, on github's server, with our data and secrets?
On-premise CI/CD may still (and often does) use someone else's code, which can expose data and secrets in the same way as outlined. The only difference is that it's on your own infrastructure.
There's something to be said for on-premise solutions, but they won't save you from a malicious maintainer.
I think the main difference is actions is built around the idea of this ecosystem of "just include other people's code in your pipeline." With Team City or GitLab you're typically only running code you wrote in their respective configuration files.
That's super valuable because you get started faster - e.g. "oh here's how I can deploy a lambda" but brings up this security concern of running other people's code. As discussed in the article, this is something NPM solved after the left-pad debacle. And I am sure (I hope) that GitHub will either change the documentation or add features to prevent this.
Instead of an incorrectly applied tapping head meme I would have used the Big Brain meme to sarcastically suggest that using a package’s reported version on the marketplace was smarter than a git SHA
I guess the same holds for Azure DevOps plugins, over which users don’t even have control over the version used. So what’s preventing plugin owners from doing something harmful?
I think GitHub is making it extremely easy to make this mistake by making the Actions Marketplace so easily accessible when you're editing a workflow in their UI. This is horrible design. They should at least point out the real consequences of pulling in these other actions and letting you fork right there.
I mean, there is an action in the marketplace that runs git-crypt to decrypt your files. With this issue, no user should ever be using this action without forking it to their repo first.
It's to reduce the friction to lock-in. GitHub should Just Work™, regardless if that sacrifices security, because the smoother the experience the more processes you'll build around it, so the harder it'll be to migrate away from it.
My team and I work on Actions at GitHub and I just wanted to stop by and add a bit more context. We definitely understand the concerns the article brings up and it's actually why we recommend in our documentation using SHA references when consuming third-party Actions. We also introduced the concept of verified Actions that extend an Organization's verification to Actions in the marketplace. We know there's more to do here to help prevent any malicious Actions and we're planning to continue to iterate. Thanks so much for the feedback and keep it coming.
It sounds like you should add an abstraction layer that makes the default depending on the SHA. Your product should be secure by default and it clearly is not right now.
Likely, you can still do this and migrate people fairly painlessly to a secure version.
Edit: even if you select a tag or something, it should use the underlying reference instead of the tag. You can add an escape hatch if you REALLY want to target the tag (there are obviously workflows where that makes sense), but it should be off by default and warm users of the security risks.
I’m excited about Actions generally, so I’m glad to hear your team is aware of the challenges of building a reliable ecosystem of third party dependencies by reference to repos.
Still, recommending git SHAs has real ergonomic and maintainability issues, and while it’s more defensive it doesn’t prevent left-pad style broken builds due to disappearing content.
And at the same time, GitHub is investing heavily in package management infrastructure. I don’t say this lightly because designing good dependency management systems is much more subtle and difficult than people typically recognize, but from where I sit this appears to be fundamentally a dependency management concern. It seems a shame not to try to put GitHub Package Registry to good work here. Is that something potentially on the horizon?
This is an excellent point, and should solve the issue I put below about builds taking forever when pulling in a slow-building action.
If I'm using a marketplace action in my workflow, I want "foo/action@v1.0.0" to be treated like a binary package dependency, not a build from sourcecode.
In addition to the security aspect, it would be really great if you have the option to lock down or cache the Docker images used for actions. It was weird for me when I pulled in an action from the marketplace for a workflow, and now about 80% of my total build time is building of that action for each and every build I run (e.g. the Dockerfile is compiled for that action EVERY time I run my build). I should have the option to at least cache that built image for me after the first time I run a workflow.
What about hash collision?
You could write an obfuscator to keep appending useless code or comments to the payload to continually generate commit hashes until you get a collision with the same original hash.
A lot of work of course, and the hash is so f-ing huge it might take a few thousand compute-years, but you know, hackers find a way.
If nothing else, use the full 40-char hash to ensure that your attacker has to find the same atom twice across 10 moons.
Looks like you can now run actions on your own servers instead https://help.github.com/en/actions/automating-your-workflow-... which might help with things quite a bit considering that you shouldn't have to expose any secrets if you are running them on your own machines
One thing I'm missing from Actions is a way to manually trigger an action from the UI.
Currently the closest equivalents are making a request via the API (but then you need a token with it, and all the security concerns it opens) or sending a special message using an empty commit (but I haven't found a simple way to do that from inside VSCode, unless I missed it somewhere).
64 comments
[ 0.76 ms ] story [ 107 ms ] threadWith Gitlab.com:
https://docs.gitlab.com/ee/user/packages/container_registry/
https://docs.gitlab.com/ee/ci/docker/README.html
I thought SHA-1 wasn't considered secure?
Here is Linus on SHA-1 in git: https://www.theregister.co.uk/2017/02/26/git_fscked_by_sha1_...
That said, hashes like md5 are still plenty used in ETags for http 304s all the time despite being considered insecure for cryptographic use.
However, to rewrite history you need a preimage attack. In other words, you need to find a value X which hashes to the same hash value y which already exists h(X)=y
These are not currently practical. So SHA1 is certainly not recommended, but isn't completely broken yet.
Git uses hashes that include more data than just the hash of the content of an object. Meaning the existing known collisions don't work and you'd have to develop collisions specifically to target git.
Additionally, the current known collision attacks require generating the two colliding contents as a pair. You can't take an existing document and then generate a collision for that document. Further, the technique is not particularly good for text documents because you need to include some data to force the collision. This is easily missed in binary documents but not the text documents most people are committing into git repos (granted you can put binary files into git and that would still work but the other reasons for this not being very useful still apply).
Git also has code to deal with possible collisions and will fall back to full comparison and will fail a commit that creates a collision. Finally, generating these document pairs leaves detectable patterns in the content. GitHub scans for these patterns as part of their process.
https://github.blog/2017-03-20-sha-1-collision-detection-on-...
Just a recap: generating two files who's hashes collide takes 2^80 tries if SHA-1 wasn't broken, whereas finding a file with the same hash as a specific, externally-determined file takes 2^160 tries.
Anything you use in a critical path you should control. If you're managing your own fork, nobody will inject bad code except yourself.
Just my 2 cents. Also, the things that github 3rd party actions are doing is usually not that complicated. I mean how many different ways can you publish a docker image or deploy an artifact to S3. Once it works, it either works or doesn't. It's not a programming library where optimizations are created or we fixed a security vulnerability (though I suppose that can come up ).
Sorry minor rant. food for thought.
Do I do this for hobby projects? No, of course not. If my toy projects break, meh. But for production? Where breakage actually costs money? Damn right I do.
It's not only breakage, though. You want a hermetic build system, so you can build software reproducibly and consistently. There are few things more "fun" than building "the same" code on two machines, and one executable has a bug while the second doesn't. (OK. Compiler bugs are even more fun)
Many automated vendor/package management tools will not automate pulling newer changes from upstream. If this was instead a simple a read-only version/tag, it is easy for those tools to provide upgrade/update path.
There is no bright line delimitating what's on the "critical path" and what's not. Your servers? The electric grid? The datacenter? The backbone(s) it's connected to? The judicial system making contracts enforceable? NATO? Name any two that straddle the line, and anybody can come up with something in between them, because it's a continuum.
It's not unreasonable to be more concerned about an open source library disappearing than say, postgresql or mysql disappearing. No macho posturing required.
Let's say I'm working front-end web development. My "critical path" might include: js bundler, some kind of application framework, various libs, etc. The scope of my work (and authority/responsibility) informs what I can reasonably have under my control.
What's the chance of the grid going down? What's the cost of it? Is the expected value significantly higher than a power generator? Congrats, you just bought a power generator.
Backbone down would cause you huge financial losses? Well, whaddaya say, let's build more backbone.[1]
Any redundancy that costs significantly less than the losses it mitigates will be built. The only reason we're not seeing a private NATO yet is that it costs more than pretty much any company makes. But 5,000 people security forces? Damn skippy those exist[2]
[1] https://engineering.fb.com/data-center-engineering/building-... [2] https://www.businessinsider.com/facebook-off-duty-police-off...
https://en.wikipedia.org/wiki/Not_invented_here
?
> It's not a programming library where optimizations are created or we fixed a security vulnerability (though I suppose that can come up ).
But it's a service provided by a paid-for entity which has support which you pay for. If it doesn't work, you let support know....
However, you'll sometimes want to update the commit hash after the action's maintainer releases new code. If you don't then you run the risk of running an old action that has vulnerable components or bugs.
The easiest workflow for that should be Dependabot's updater for GitHub actions: https://dependabot.com/github-actions/ Has anyone tried that approach for GitHub actions?
You still need to review the action's code before you use it, and every time you update to a new commit hash. But this approach protects you from automatically running new malicious code pushed to master/re-tagged.
We're really not even pulling the dependency locally, we're just going to run someone else's code from their repo, on github's server, with our data and secrets?
What could possibly go wrong with that? /s
It's a relevant distinction on this context.
There's something to be said for on-premise solutions, but they won't save you from a malicious maintainer.
I am thinking specifically about: * Gitlab's CI * CircleCI * TeamCity
etc....
That's super valuable because you get started faster - e.g. "oh here's how I can deploy a lambda" but brings up this security concern of running other people's code. As discussed in the article, this is something NPM solved after the left-pad debacle. And I am sure (I hope) that GitHub will either change the documentation or add features to prevent this.
I mean, there is an action in the marketplace that runs git-crypt to decrypt your files. With this issue, no user should ever be using this action without forking it to their repo first.
Likely, you can still do this and migrate people fairly painlessly to a secure version.
Edit: even if you select a tag or something, it should use the underlying reference instead of the tag. You can add an escape hatch if you REALLY want to target the tag (there are obviously workflows where that makes sense), but it should be off by default and warm users of the security risks.
Still, recommending git SHAs has real ergonomic and maintainability issues, and while it’s more defensive it doesn’t prevent left-pad style broken builds due to disappearing content.
And at the same time, GitHub is investing heavily in package management infrastructure. I don’t say this lightly because designing good dependency management systems is much more subtle and difficult than people typically recognize, but from where I sit this appears to be fundamentally a dependency management concern. It seems a shame not to try to put GitHub Package Registry to good work here. Is that something potentially on the horizon?
If I'm using a marketplace action in my workflow, I want "foo/action@v1.0.0" to be treated like a binary package dependency, not a build from sourcecode.
Currently the closest equivalents are making a request via the API (but then you need a token with it, and all the security concerns it opens) or sending a special message using an empty commit (but I haven't found a simple way to do that from inside VSCode, unless I missed it somewhere).