What’s more, GitHub has basically stopped maintaining their own actions, pushing people to sketchy forks to do basic things. Their entire ecosystem is basically held up with duct tape and gets very little investment.
I never used any actions and never understood why would I need to. I just wrote bash script to build my project and that's about it. This modern tendency to add dependencies for trivial things baffles me. You don't need "action" to do `git clone`.
They barely maintain Azure pipeline tasks / actions as well.
We had a critical outage because they deprecated Windows 2019 agents a month earlier than scheduled. MS support had the gall to both blame us for not migrating sooner, and refuse to escalate for 36 hours!
What? No they didn't. They extended the deprecation timeline for Windows 2019 agents from the original EOL date of 30 June 2025 to 31 December 2025; with a well-published brownout period from 2 December to 9 December in addition to the original brownout period from 3 June to 24 June.
The initial banners and warning emails about it went out well ahead of the original EOL timeline; and again as the extended EOL drew close.
If you were caught off guard by the brownout period, it's your devops team that's to blame, not Microsoft; and Microsoft was absolutely right to blame you for not migrating sooner. They gave you an extra 6 months to do it because you should have had all this done back in the first half of the year.
(If you want to blame Microsoft for anything here, blame them for not having a comprehensive tool to identify all your windows-2019 pipelines and instead just relying on "just go look at the latest pipeline runs page and hope everything's run recently enough to be on that".)
Pleased this is being discussed somewhere as it’s something that has troubled me for a while.
There are so many third party actions where the docs or example reference the master branch. A quick malicious push and they can presumably exfiltrate data from a ton of repositories
(Even an explicit tag is vulnerable because it can just be moved still, but master branch feels like not even trying)
> The researchers identified four fundamental security properties that CI/CD systems need: admittance control, execution control, code control, and access to secrets.
Why do CI/CD systems need access to secrets? I would argue need access to APIs and they need privileges to perform specific API calls. But there is absolutely nothing about calling an API that fundamentally requires that the caller know a secret.
I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
“Good CI systems shouldn’t support secrets, at most there should be [the most complicated secret support ever]”
Let’s just call it secret support.
I agree with your suggestion that capabilities-based APIs are better, but CI/CD needs to meet customers where they’re at currently, not where they should be. Most customers need secrets.
While good in theory, in practice secrets are used to validate those privileges have been assigned. Even in schemes like metadata servers, you still use a secret.
Pedantically I'd say maybe it's more fair to say they shouldn't have access to long lived secrets and should only use short lived values.
The "I" stands for Integration so it's inevitable CI needs to talk to multiple things--at the very least a git repo which most cases requires a secret to pull.
> I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
This all seems right, but the reality is that people will put secrets into CI/CD, and so the platform should provide an at least passably secure mechanism for them.
(A key example being open source: people want to publish from CI, and they’re not going to set up additional infrastructure when the point of using third-party CI is to avoid that setup.)
Fine. Then let people set up a little WASM app that gets access to the secret and has an API callable by the workflow. And make that app be configured as part of the secret’s configuration, not as a file in the repository.
You're missing that the D in CI/CD means deployment; be that packaging on pushing tags and publishing to a registry, or building images, or packaging github releases.
Yep. I'm switching our workflows to instead use regular utilities running inside a Docker container.
This works well for _most_ things. There are some issues with doing docker-in-docker for volume mapping, but they're mostly trivial. We're using taskfiles to run tasks, so I can just rely on it for that. It also has a built-in support for nice output grouping ( https://taskfile.dev/docs/reference/schema#output ) that Github actions can parse.
Pros:
1. Ability to run things in parallel.
2. Ability to run things _locally_ in a completely identical environment.
3. It's actually faster!
4. No vendor lock-in. Offramp to github runners and eventually local runners?
Cons:
It often takes quite a while to understand how actions work when you want to run them in your own environment. For example, how do you get credentials to access the Github Actions cache and then pass them to Docker? Most of documentation just tells: "Use this Github Action and stop worrying your pretty little head about it".
Normally I’d say stop kicking the dead horse, but GHA deserves all the complaints it gets and then some. It’s the embodiment of everything that’s bad in ‘less is more’.
My biggest concern with it is that it’s somehow the de facto industry standard. You could do so much better with relatively small investments, but MS went full IE6 with it… and now there’s a whole generation of young engineers who don’t know how short their end of the stick actually is since they never get to compare it to anything.
It's funny that absolutely everything about GHA fucking sucks, and everyone agrees about this. BUT, the fact that it's free compute, and it's "right there"... means it's very very difficult to say no to!
Personally I've just retired a laptop and I'm planning to turn it into a little home server. I think I'm gonna try spinning up Woodpecker on there, I'm curious to see what a CI system people don't hate is like to live with!
TFA mentions this option and then goes on at some length to explain that this doesn't help for transitive dependencies, which is how these attacks usually work.
If I write actions/setup-python@v1, I'm expecting the action to run with the v1 tag of that repository. If I rerun it, I expect it to run with the v1 tag of that repository...which I'm aware may not be the same if the tag was updated.
If I instead use actions/setup-python@27b31702a0e7fc50959f5ad993c78deac1bdfc29 then I'm expecting the action to run with that specific commit. And if I run it again it will run with the same commit.
So, whether you choose the tag or the commit depends on whether you trust the repository or not, and if you want automatic updates. The option is there...isn't it?
> which I'm aware may not be the same if the tag was updated.
That's the mistake that breaks the following. People don't usually expect that it's an arbitrary modifiable reference, but instead that it's the same version they've picked when they created the file (ie a tag is just a human friendly name for a commit)
I checked out the linked GitHub repo https://github.com/ecosyste-ms/package-manager-resolvers and it appears to be just a README.md that collects summaries of different package managers? How do I know these weren't just LLM-generated?
This is making me feel quietly vindicated in pushing back on migrating our Jenkins/Ansible setup to GHA simply because corporate wanted the new shiny thing. Fortunately the "this will be a lot of work, i.e. cost" argument won.
Mind you, CI does always involve a surprising amount of maintenance. Update churn is real. And Macs still are very much more fiddly to treat as "cattle" machines.
CI is interesting because what really happens is you run a script on someone elses computer but with fairly weird constraints.
I get it's use, especially in large companies and I also get the culture leading up to it being widely used but I can't help but chuckle a bit about the problems we cause for ourselves in this industry.
While I hate defending GHA, the docs do include this:
- Using the commit SHA of a released action version is the safest for stability and security.
- If the action publishes major version tags, you should expect to receive critical fixes and security patches while still retaining compatibility. Note that this behavior is at the discretion of the action's author.
So you can basically implement your own lock file, although it doesn't work for transitive deps unless those are specified by SHA as well, which is out of your control. And there is an inherent trade-off in terms of having to keep abreast if critical security fixes and updating your hashes, which might count as a charitable explanation for why using hashes is less prevalent.
On the other hand, this issue has been known to GitHub since shortly after Actions’ release[0]. They added some cya verbiage to their docs, but they never followed up by making version pinning meaningful.
Sure you can implement it yourself for direct dependencies and decide to only use direct dependencies that also use commit sha pinning, but most users don’t even realize it’s a problem to begin with. The users who know often don’t bother to use shas anyway.
Or GitHub could spend a little engineer time on a feasible lock file solution.
I say this as somebody who actually likes GitHub Actions and maintains a couple of somewhat well-used actions in my free time. I use sha pinning in my composite actions and encourage users to do the same when using them, but when I look at public repos using my actions it’s probably 90% using @v1, 9% @v1.2 and 1% using commit shas.
[0] Actions was the first Microsoft-led project at GitHub — from before the acquisition was even announced. It was a sign of things to come that something as basic as this was either not understood or swept under the rug to hit a deadline.
Using an SHA is an anti-pattern for me. Because by using one, you kind of modeled "I am getting this fixed/static thing"; when in reality, it is very far from that. I got bitten by it twice that I learned that you either have a lock file or you don't.
I agree 100% with what I think is the key phrase, viz. "the results can change without any modification to your code".
I maintain an R package that is quite stable and is widely used. But every month or so, the GHA on one of the R testing machines will report an error. The messages being quite opaque, I typically spend a half hour trying to see if my code is doing something wrong. And then I simply make a calendar item to recheck it each day for a while. Sure enough, the problems always go away after a few days.
After having to deal with shell rc files causing chaos and unpredictability, I can appreciate your top level code fundamental not being random code but something that is parsable and not subject to the halting problem.
We are currently using GitHub Actions for all our CI tasks and I hate it. Yes, the marketplace is nice and there are a lot of utility actions which make life easier, but they all come with the issues the post highlights. Additionally, testing Actions locally is a nightmare. I know that act exists but for us it wasn't working most of the time. Also the whole environment management is kinda odd to me and the fact, that when using an environment (which then allows to access secrets set in that environment) it always creates a new deployment is just annoying [1]
I guess the best solution is to just write custom scripts in whatever language one prefers and just call those from the CI runner. Probably missing out on some fancy user interfaces but at least we'd no longer be completely locked into GHA...
> Some teams vendor actions into their own repos. zizmor is excellent at scanning workflows and finding security issues. But these are workarounds for a system that lacks the basics.
Harsh given GitHub makes it very easy to setup attestations for Artifact (like build & sbom) provenances.
That said, Zizmor (static analyser for GitHub Actions) with Step Security's Harden Runner (a runtime analyser) [0] pair nicely, even if the latter is a bit of an involved setup.
Pip has been a flag bearer for Python packaging standards for some time now, so that alternatives can implement standards rather than copy behavior. So first a lock file standard had to be agreed upon which finally happened this year: https://peps.python.org/pep-0751/
Now it's a matter of a maintainer, who are currently all volunteers donating their spare time, to fully implement support. Progress is happening but it is a little slow because of this.
It is concerning that GitHub hosts the majority of open-source software, while actively locking its users into a platform that is based on closed source for eerything except Git itself.
This issue with Actions shows how maintaining proprietary software inevitably ends up rather low on the priority list. Adding new features is much more marketable, just like for any other software product. Enshittification ensues.
When you have a multi-platform image the actual per-platforms are usually not tagged. No point.
But that doesn't mean that they are untagged.
So on GitHub Actions when you upload a multi-platform image the per-platform show up in the untagged list. And you can delete them, breaking the multi-platform image, as now it points to blobs that don't exist anymore.
I committed the project I maintain to GitHub Actions when Actions first came out, and I'm really starting to regret it.
The main problem, which this article touches, is that GHA adds a whole new dimension of dependency treadmill. You now have a new set of upstreams that you have to keep up to date along with your actual deployment upstreams.
Has anyone built a “deep fork” tool that lets you make a private fork of all dependencies, then modifies their transitive dependencies to point to private forks, and so on? Ideally in a way where updates can be pulled in manually? Seems feasible.
> Has anyone built a “deep fork” tool that lets you make a private fork of all dependencies, then modifies their transitive dependencies to point to private forks, and so on? Ideally in a way where updates can be pulled in manually? Seems feasible.
If you do, please submit a "show HN." I'd love to use it.
A lot of the actions people tend to use are just unnecessary. They're simple wrappers around real tools. In those cases, use mise-en-place. It's a single action that installs all relevant tools (and keeps your local dev env in check), and it supports lock files.
63 comments
[ 2.6 ms ] story [ 97.4 ms ] threadThat isn't gonna get better anytime soon.
"GitHub Will Prioritize Migrating to Azure Over Feature Development" [1]
[1] https://thenewstack.io/github-will-prioritize-migrating-to-a...
We had a critical outage because they deprecated Windows 2019 agents a month earlier than scheduled. MS support had the gall to both blame us for not migrating sooner, and refuse to escalate for 36 hours!
The initial banners and warning emails about it went out well ahead of the original EOL timeline; and again as the extended EOL drew close.
If you were caught off guard by the brownout period, it's your devops team that's to blame, not Microsoft; and Microsoft was absolutely right to blame you for not migrating sooner. They gave you an extra 6 months to do it because you should have had all this done back in the first half of the year.
(If you want to blame Microsoft for anything here, blame them for not having a comprehensive tool to identify all your windows-2019 pipelines and instead just relying on "just go look at the latest pipeline runs page and hope everything's run recently enough to be on that".)
There are so many third party actions where the docs or example reference the master branch. A quick malicious push and they can presumably exfiltrate data from a ton of repositories
(Even an explicit tag is vulnerable because it can just be moved still, but master branch feels like not even trying)
Why do CI/CD systems need access to secrets? I would argue need access to APIs and they need privileges to perform specific API calls. But there is absolutely nothing about calling an API that fundamentally requires that the caller know a secret.
I would argue that a good CI/CD system should not support secrets as a first-class object at all. Instead steps may have privileges assigned. At most there should be an adapter, secure enclave style, that may hold a secret and give CI/CD steps the ability to do something with that secret, to be used for APIs that don’t support OIDC or some other mechanism to avoid secrets entirely.
I don't really understand what you mean by "secure enclave style"? How would that be different?
Let’s just call it secret support.
I agree with your suggestion that capabilities-based APIs are better, but CI/CD needs to meet customers where they’re at currently, not where they should be. Most customers need secrets.
Pedantically I'd say maybe it's more fair to say they shouldn't have access to long lived secrets and should only use short lived values.
The "I" stands for Integration so it's inevitable CI needs to talk to multiple things--at the very least a git repo which most cases requires a secret to pull.
This all seems right, but the reality is that people will put secrets into CI/CD, and so the platform should provide an at least passably secure mechanism for them.
(A key example being open source: people want to publish from CI, and they’re not going to set up additional infrastructure when the point of using third-party CI is to avoid that setup.)
Because you need to be able to sign/notarize with private keys and deploy to cloud environments. Both of these require secrets known to the runner.
This works well for _most_ things. There are some issues with doing docker-in-docker for volume mapping, but they're mostly trivial. We're using taskfiles to run tasks, so I can just rely on it for that. It also has a built-in support for nice output grouping ( https://taskfile.dev/docs/reference/schema#output ) that Github actions can parse.
Pros:
1. Ability to run things in parallel.
2. Ability to run things _locally_ in a completely identical environment.
3. It's actually faster!
4. No vendor lock-in. Offramp to github runners and eventually local runners?
Cons:
It often takes quite a while to understand how actions work when you want to run them in your own environment. For example, how do you get credentials to access the Github Actions cache and then pass them to Docker? Most of documentation just tells: "Use this Github Action and stop worrying your pretty little head about it".
My biggest concern with it is that it’s somehow the de facto industry standard. You could do so much better with relatively small investments, but MS went full IE6 with it… and now there’s a whole generation of young engineers who don’t know how short their end of the stick actually is since they never get to compare it to anything.
Personally I've just retired a laptop and I'm planning to turn it into a little home server. I think I'm gonna try spinning up Woodpecker on there, I'm curious to see what a CI system people don't hate is like to live with!
I'm from a generation who had to use VSS for a few years. The sticks are pretty long these days, even the ones you get from github.
> actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744
If I write actions/setup-python@v1, I'm expecting the action to run with the v1 tag of that repository. If I rerun it, I expect it to run with the v1 tag of that repository...which I'm aware may not be the same if the tag was updated.
If I instead use actions/setup-python@27b31702a0e7fc50959f5ad993c78deac1bdfc29 then I'm expecting the action to run with that specific commit. And if I run it again it will run with the same commit.
So, whether you choose the tag or the commit depends on whether you trust the repository or not, and if you want automatic updates. The option is there...isn't it?
That's the mistake that breaks the following. People don't usually expect that it's an arbitrary modifiable reference, but instead that it's the same version they've picked when they created the file (ie a tag is just a human friendly name for a commit)
Mind you, CI does always involve a surprising amount of maintenance. Update churn is real. And Macs still are very much more fiddly to treat as "cattle" machines.
Current job is using blacksmith to save on costs, but the reality of it is that this caching layer only adds costs in some of our projects
I get it's use, especially in large companies and I also get the culture leading up to it being widely used but I can't help but chuckle a bit about the problems we cause for ourselves in this industry.
- Using the commit SHA of a released action version is the safest for stability and security.
- If the action publishes major version tags, you should expect to receive critical fixes and security patches while still retaining compatibility. Note that this behavior is at the discretion of the action's author.
So you can basically implement your own lock file, although it doesn't work for transitive deps unless those are specified by SHA as well, which is out of your control. And there is an inherent trade-off in terms of having to keep abreast if critical security fixes and updating your hashes, which might count as a charitable explanation for why using hashes is less prevalent.
So in other words the strategy in the docs doesn't actually address the issue
Sure you can implement it yourself for direct dependencies and decide to only use direct dependencies that also use commit sha pinning, but most users don’t even realize it’s a problem to begin with. The users who know often don’t bother to use shas anyway.
Or GitHub could spend a little engineer time on a feasible lock file solution.
I say this as somebody who actually likes GitHub Actions and maintains a couple of somewhat well-used actions in my free time. I use sha pinning in my composite actions and encourage users to do the same when using them, but when I look at public repos using my actions it’s probably 90% using @v1, 9% @v1.2 and 1% using commit shas.
[0] Actions was the first Microsoft-led project at GitHub — from before the acquisition was even announced. It was a sign of things to come that something as basic as this was either not understood or swept under the rug to hit a deadline.
I maintain an R package that is quite stable and is widely used. But every month or so, the GHA on one of the R testing machines will report an error. The messages being quite opaque, I typically spend a half hour trying to see if my code is doing something wrong. And then I simply make a calendar item to recheck it each day for a while. Sure enough, the problems always go away after a few days.
This might be specific to R, though.
Why not just build the workflows themselves as docker images? I guess running other docker images in the workflow would then become a problem.
I guess the best solution is to just write custom scripts in whatever language one prefers and just call those from the CI runner. Probably missing out on some fancy user interfaces but at least we'd no longer be completely locked into GHA...
[1] https://github.com/orgs/community/discussions/36919
Harsh given GitHub makes it very easy to setup attestations for Artifact (like build & sbom) provenances.
That said, Zizmor (static analyser for GitHub Actions) with Step Security's Harden Runner (a runtime analyser) [0] pair nicely, even if the latter is a bit of an involved setup.
[0] https://github.com/step-security/harden-runner
> The fix is a lockfile.
Hopefully, SLSA drafts in Hermetic build process as a requirement: https://slsa.dev/spec/v1.2/future-directions
I have a little launcher for that which helps: https://github.com/7mind/mudyla
I'm pretty sure it contains the exact line of it being "deeply confused about being a package manager".
[1]: https://www.youtube.com/watch?v=9qljpi5jiMQ
Well... not Pip!
Pip has been a flag bearer for Python packaging standards for some time now, so that alternatives can implement standards rather than copy behavior. So first a lock file standard had to be agreed upon which finally happened this year: https://peps.python.org/pep-0751/
Now it's a matter of a maintainer, who are currently all volunteers donating their spare time, to fully implement support. Progress is happening but it is a little slow because of this.
https://github.com/cachix/cloud.devenv.sh
https://broderic.blog/post/moving-away-from-netlify/
For those who can still escape the lock-in, this is probably a good occasion to point to Forgejo, an open-source alternative that also has CI actions: https://forgejo.org/2023-02-27-forgejo-actions/ It is used by Codeberg: https://codeberg.org/
When you have a multi-platform image the actual per-platforms are usually not tagged. No point.
But that doesn't mean that they are untagged.
So on GitHub Actions when you upload a multi-platform image the per-platform show up in the untagged list. And you can delete them, breaking the multi-platform image, as now it points to blobs that don't exist anymore.
The main problem, which this article touches, is that GHA adds a whole new dimension of dependency treadmill. You now have a new set of upstreams that you have to keep up to date along with your actual deployment upstreams.
If you do, please submit a "show HN." I'd love to use it.