Definitely not a skill issue if you read the details in their post - https://www.feldera.com/blog/the-pain-that-is-github-actions - they're clearly very experienced with using GitHub Actions and have run into legitimate challenges due to the complexity of what they're using it for.
GitHub Actions may cover this stuff in the documentation but it's still difficult to use. That means the product could be designed better (and the documentation could be easier to follow.)
Could we not usually say most software could be documented better. I do not think GitHub Actions ranks near the bottom in terms of documentation and user experience overall. I do understand your point though.
Should have been zero permissions by default. The current model is a mess of global settings, workflow permissions, and job tokens that nobody understands.
Azure DevOps is nearly identical, but with slightly different zoo of issues that are less well documented in public sources.
It also has the problem of not having a local dev runner for actions. The "inner loop" is atrociously slow and involves spamming your colleagues with "build failed" about a thousand times, whether you like it or not.
IMHO, a future DevOps runner system must be an open-source, local-first. Anything else is madness.
Right now we're in the "mainframe era" of DevOps, where we edit text files in baroque formats with virtually no tooling assistance, "submit" that to a proprietary batch system on a remote server that puts it into a queue... then come back after our coffee to read through the log printout.
I should buy a dot matrix printer to really immerse myself into the paradigm.
They are so identical, there's code in the GitHub runner to search and replace "Azure DevOps" with "GitHub Actions" in log output on the fly.
The entire code is a wonderful mess. We found that when we early-adopted ephemeral runners, that the control flow is full of races and the status code you get at the end is indicative of exactly nothing. So even if the backend is just having a hickup picking up a job with an obscure Azure error code, you better just throw that entire VM away, because you can't know if that runner will ever recover or has already done things to break the next run.
The current version of GHA is "Azure DevOps v3".
IIRC, it came after Microsoft purchased GitHub and I think it was part of a plan to discontinue/kill Azure DevOps altogether in favor of GitHub. I don't think they have feature parity yet, specially on the issues and permissioning parts.
Although, I never saw a public announcement of this discontinuation, ADO is kind of abandoned AFAICT and even their landing page hints to use GitHub Enterprise instead [1].
tldr but: don't use GitHub Actions. Its a mess, the availability is often atrocious, and the UI around it is _still_ as clunky as when they first rolled it out many years ago.
GitHub Actions is like Microsoft Teams. Nobody who knows better wants to use it, but it's slightly better than what most did before (email/jenkins/nothing) and came with the thing you're already using. At least your boss thinks it's better. And it's such a good deal!
A 1000% yes, because it means the default experice most devs have of CI is using ephemeral runners which is a massive win for security and build rot.
Every company I've worked at with stateful runners was a security incident begging to happen, not to mention builds that would do different things depending on what runner host you got placed on (devs manually installing different versions of things on hosts, etc)
Does anyone use GHA - full stop? Their stuff seems "me too" in most cases, they are definitely a follower, like microsoft Bing search ... google AI ...
Not sure why this was downvoted/flagged, I do use Drone CI myself currently and it's quite pleasant: https://www.drone.io/
There's also the Woodpecker CI fork, which has a very similar user experience: https://woodpecker-ci.org/
When combined with Docker images, it's quite pleasant to use - you define what environment you want for the CI/CD steps, what configuration/secrets you need and define the steps (which can also just be a collection of scripts that you can run locally if need be), that's it.
Standalone, so you can integrate it with Gogs, Gitea or similar solutions for source control and perhaps a bit simpler than GitLab CI (which I also think is lovely, though maintaining on-prem GitLab isn't quite a nice experience all the time, not that you have to do that).
To expand on this, CircleCI lets you easily ssh into a run so you can debug it as if you were debugging it on your own machine. I cannot tell you how many times this has saved me countless hours because you don't need to wait for your changes to build and then subsequently fail repeatedly, drastically shortening the iteration cycle.
After using Gitlab CI for years and setting up some pretty complex scenarios, when I switched over to GitHub I found the UX to be pretty rough. Seems very opaque and I find the documentation to be at best hard to navigate.
Maybe it was just the pain of switching but that was my initial impression.
I am really really hoping that someone (not me, I've already tried and failed) could slim it down into a single-purpose, self-contained, community maintainable tool ...
These are all real pains, author definitely has done a lot of work in Github Actions; respect. I'm sure these notes will save a lot of people a lot of frustration in the future, since Github Actions isn't going away --- it's too damn convenient.
I wonder why they chose to move back to Github Actions rather than evaluate something like Buildkite? At least they didn't choose Cloud Build.
Yep, I've run into every one of these issues in my time working with the CI. It's still leagues beyond the old Azure DevOps pipelines or god forbid, Jenkins.
I think incremental progress in the CI front is chugging along nicely, and I really haven't seen any breathtaking improvements from other solutions I've tried, like CircleCI.
Used to use GH actions quite a bit. At my current company we set up RWX Mint (rwx.com/mint) and haven't looked back. (disclaimer: used to work at rwx but no longer affiliated)
So Github was really the perfect acquisation for the Microsoft portfolio. Applications with a big market share that are technically inferior to the competition.
// Luckily still a gitlab user, but recently forced to Microsoft Teams and office.
my condolences to you and your team for that switch; it's my 2nd used-and-disliked thing (right next to atlassian) - oh well
but one cool feature i found with ms teams that zoom did not have (some years ago - no clue now) is turning off incoming video so you dont have to be constantly distracted in meetings
edit:
oh yeah, re github actions and the user that said:
> Glad I’m not the only one
me too, me too; gh actions seem frustrating (from a user hardly using gh actions, and more gitlab things - even though gitlab seems pretty wonky at times, too)
I prefer teams just for the fact that by default everyone can mute everyone else in the call. It just gives me peace of mind that if I ever leave my mic on by mistake, someone in the call would have my back and just mute me.
Actions have special integration with GitHub (e.g. they can annotate the pull request review UI) using an API. If you forgo that integration, then you can absolutely use GitHub Actions like "a container you run scripts in." This is the advice that is usually given in every thread about GitHub Actions.
If you want to make a CI performant, you'll need to use some of its features like caches, parallel workers, etc. And GHA usability really fall short there.
The only reason I put up with it is that it's free for open source projects and integrated in GitHub, so it took over Travis-ci a few years ago.
Devil's advocate: They could make the github CLI capable of doing all of those things (if it's not already), and then the only thing the container needs is a token.
> For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
There are lots of problems.
Actions try to abstract the script away and give you a consistent experience and, must crucially, allow sharing. Because gitlab has no real way to share actions or workflows (I can do yaml include, but come on that sucks even harder than actions) you are constantly reinventing the wheel.
That's ok if all you do is " build folder" but if you need caching, reporting of issues, code coverage etc. Pp it gets real ugly really fast.
Example: yesterday I tried services, i.e. starting up some DB and backend containers to run integration tests against. Unfortunately, you cannot expand dynamic variables (set by previous containers) but are limited to already set bars. So back to docker compose...and the gitlab pipelines are chock full of such weird limitations
I haven't looked too much into how sharing workflows works, but isn't the use of shared GitHub workflows (from outside your org) a little dangerous? I get it, we use other people's code all the time. Some we trust more (ISO of a Linux OS with SHA) and others we trust a little less even if it comes from a verified source with GPG, because we know that supply chain attacks can happen.
Every time someone introduced a new way to use someone else's shared magic I feel nervous about using it. Like GitHub Actions. Perhaps it's time for me to dig into them a bit more and try to understand if/how they're safe to use. But I seem to remember just a few days ago someone mentioning a GitHub action getting hijacked?
Definitely a mixed bag. Lots of community derived actions which yes, potentially have some bad supply chain questions. I tend to try and avoid these as much as possible. Lots of established vendors also have their own actions shared though, so you don't have to reinvent the wheel when interacting with their platforms/services/products.
For instance, AWS has a lot of actions they maintain to assist with common CI/CD needs with AWS services.
I will be stunned if this doesn't become a more popular attack vector over the next few years. Lots of valuable stuff sits in github, and they're a nearly-wide-open hole to access it.
You can apply dynamic env to other jobs by exporting an env file as a dotenv artifact. So first job creates a dotenv file and export it as artifact. Second depends on the first so it can consume the artifact. https://docs.gitlab.com/ci/yaml/artifacts_reports/#artifacts...
Same. I'd been using Gitlab for a few years when Actions came out. Looked at it and thought, wow that's weird, but gave it the benefit of the doubt as it's just different, surely it would make sense eventually. Well no, it doesn't make sense, and seeing all the shocked Pikachu at the action compromise the other day was amusing.
> I have some GitHub actions for some side projects and it just seems so much more confusing to setup for some reason.
Because the docs are crap perhaps? I prefer it, having used both professionally (and Jenkins, Circle, Travis), but I do think the docs are really bad. Even just the nesting of pages once you have them open, where is the bit with the actual bloody syntax reference, functions, context, etc.
A few years back I wanted to throw in the towel and write a more minimal GHA-compatible agent. I couldn't even find where in the code they were calling out to GitHub APIs (one goal was to have that first party progress UI experience). I don't know where I heard this, so big hearsay warning, but apparently nobody at GitHub can figure it out either.
> A few days ago, someone compromised a popular GitHub Action. The response? "Just pin your dependencies to a hash." Except as comments also pointed out, almost no one does.
I used GitHub actions when building a fin services app, so I absolutely used the hash to specify Action dependencies.
I agree that this should be the default, or even the required, way to pull in Action dependencies, but saying "almost no one does" is a pretty lame excuse when talking about your own risk. What other people do has no bearing on your options here.
Pin to hashes when pulling in Actions - it's much, much safer
I think the HN community at large had a bit of a learning experience a couple of days ago.
"Defaults matter" is a common phrase, but equally true is: "the pattern everyone recommends including example documentation matters".
It is fair to criticise the usage of GH Actions, just like it's fair to criticise common usage patterns of MySQL that eat your data - even if smarter individuals (who learn from deep understanding, or from being burned) can effectively make correct decisions, since the population of users are so affected and have to learn the hard way or be educated.
I wholeheartedly agree, and perhaps it was just how I was interpreting the author's statement in the article. If it's saying that the "default" way of using GitHub Actions is dangerous and leads to subtle security footguns, I completely agree. But if you know the proper way to use and secure Actions, saying "everyone else does it a bad way" is irrelevant to your security posture.
But there is no transitive locking like package manager lockfiles. So if I depend on good/foo@hash, they depend on bad/hacked@v1 and V1 gets moved to malicious version I get screwed.
This is for composite actions. For JS actions what if they don't lock dependencies but pull whatever newest package at action setup time? Same issue.
Would have to transitively fork everything and pin it myself, and then keep it updated.
Pinning dependencies is trading one problem for another.
Yes, your builds will work as expected for a stretch of time, but that period will come to an end, eventually.
Then one day you will be forced to update those pinned dependencies and you might find yourself having to upgrade through several major versions, with breaking changes and knock-on effects to the rest of your pipelines.
Allowing rolling updates to dependencies helps keep these maintenance tasks small and manageable across the lifetime of the software.
Not pinning dependencies is an existential risk to the business. Yes it’s a tradeoff, you must assign a probability of any dependency being hijacked in your timeframe yourself, but it is not zero.
You don’t have to update them manually. Renovate supports pinned GitHub Actions dependencies [1]. Unfortunately, I don’t use Dependabot so can’t say whether it does the same.
Just make sure you don’t leak secrets to your PRs. Also I usually review changes in updated actions before merging them. It doesn’t take that much time, so far I’ve been perfectly fine with doing that.
Dependabot does support pinned hashes, even adds the comment after them with the tag. Dependabot fatigue is a thing though, and blindly mashing "merge" doesn't do much for your security, but at least there's some delay between a compromise and your workflow being updated to include it.
That isn't even the biggest problem. That breaks, and breakage gets fixed. Other than some slight internal delays there is little harm done. (You have a backup emergency deploy process that doesn't depend on GitHub anyways right?)
The real problem is security vulnerabilities in these pinned dependencies. You end up making a choice between:
1. Pin and risk a malicious update.
2. Don't pin and have your dependencies get out of date and grow known security vulnerabilities.
Interesting. I‘m also moving our CI to GitHub actions after years of using Jenkins with custom pipelines written in groovy etc. I checked out GitHub actions every now and then to feel if a move finally makes sense. I started with simple builds then tested adding our Jenkins macOS agents as self hosted runners. Just yesterday I wrote two actions to build and test a new .net project. I was able to run the whole thing with „act“ locally before running it on GitHub proper. I also played around and created a custom action in typescript (kicked off from the available predefined templates) to see how much work maintaining that means. All in all I‘m super happy and see no bigger issues. But here are some things that might be a reason:
I split CI in build system logic which should and need to run locally and just stuff that GitHub needs to execute. At best that means describing what runs in parallel, and making specific connections. Any complicated logic needs to be abstracted away behind a a setup that is itself testable. I handle it the same for our build system components. We use gradle a lot and of a few custom plugins which encapsulate specific build / automations. It’s like dividing your problem into many smaller pieces which are tested and developed in isolation.
Next to json I also used travisCI and appveyor for projects. And they all had the same (commit and pray) setup that ai hate. I wish if „act“ was a tool directly maintained by the GitHub folks though.
Whenever I get mad at GitHub Actions, I refer to it by it's true name: VisualSourceSafe Actions. Because that's what it is, and it shows. If you check out their Action Runner's source code[1], you'll find the VSS prefix all over, showing it's lineage.
I know they've fixed VSS ages ago, but for many years it was buggy af and would catastrophically lose data on automerges that it confidently made and was wrong.
I had a coworker who called it Visual Sorta-Safe which is just about the best parody name I've ever heard in my entire career.
One thing I found useful was writing a runner for giteas actions CI which is similar to GHA. When you dig down and ask "what is ACTUALLY happening to run this job" then a lot of things such as the docker entrypoint not being modifiable make perfect sense.
My team uses GitLab and most other teams are on Azure dev ops. They keep trying to get us to switch telling us how amazing pipelines are. Glad to know we are not missing anything.
> like binding a macro to the MS Word save button to make it conditional
You have no idea how much I'd love that feature. Inasmuch as "save" is still a thing anyway. I don't miss explicit saves in IDEA, I see commit as the "real" save operation now, and I don't mind being able to hook that in an IDE-independent way.
I think the UX of git hooks has been sub-par for sure, but tools like the confusingly named pre-commit are helping there.
Because if you haven’t auto-formatted, lined, etc. then it’s a very easy way to do that so you don’t waste time watching CI fail for something stupid like trailing comma placement.
I don’t want to think about formatting, I just want everything to be consistent. A pre commit hook can run those tools for me, and if any changes occurred, it can add them to the commit.
There's a long set of steps to making a tool mandatory in a development environment, but the final step should always, always be, "And you will find yourself on a PIP if you refuse to use the mandatory tools."
If people want to die on a hill that is demonstrably causing problems for all of their coworkers then let em.
In my opinion neither hooks nor CI should ever make changes to code automatically. When I commit changes, I want to see exactly what I commit, and not have some system change it at the last minute.
Instead, have tooling to do that before committing (vscode format-on-save, or manually run a task), then have a pre-commit hook just do a sanity-check on that. It only needs to check modified files, so usually very fast.
Then, have an additional check on CI to verify formatting on all files. That should rarely be triggered, but helps to catch cases where the hooks were not run, for example from external contributes. That also makes it completely fine for this CI step to take a couple of minutes - you don't need that feedback immediately.
The tension in any system is how many ways the build can fail other than the most obvious one. So I generally only encourage things in the pre-commit hook like no weird punctuation (I'm looking at you, Microsoft), no empty commit messages, and maybe require a ticket number (or try to guess one out of the branch name).
Though it would be sort of interesting or maybe just amusing if you made something like ssh-agent but for 'git commit' and your test runner. Only allow commits when all files are older than your last green test run.
Usually if you’re using it, it’s because you’re forced to.
In my experience, the best strategy is to minimize your use of it — call out to binaries or shell scripts and minimize your dependence on any of the GHA world. Makes it easier to test locally too.
This is what I do. I've written 90% of the logic into a Go binary and GitHub Actions just calls out to it at certain steps. It basically just leaves GHA doing the only thing it's decent at...providing a local UI for pipelines. The best part is you get unit tests, can dogfood the tool in its own pipeline, and can run stuff locally (by just having the CLI nearby).
Makes migrations easier too; better to let gitHub or gitlab etc to just be the platform to host source code and trigger events which you decide how to deal with. Your CI itself should be another source controlled repo that provides the features for the application code's thin CI layer to invoke and use. That allows you to be able to run your CI locally in a pretty realistic manner too.
I have done something similar with Jenkins and groovy CI library used by Jenkins pipeline. But it wasn't super simple since a lot of it assumed Jenkins. I wonder if there is a more cleaner open source option that doesn't assume any underlying platform.
We recently had a developer -- while trying to debug container builds for a version upgrade for a PR on their local branch -- accidentally trigger a deployment of their local branch's docker container to production (!) while messing around with Github action workflow files in their pull request (not main).
Outside of locking down edit access to the .github workflow yml files I'm not sure how vulnerabilities like this can be prevented.
Yeah it's a difficult problem, templates help, then put the templates into another repo that is managed by a specific person and imported into others. Not sure how that work in a monorepo, I expect those controls wouldn't.
The problem is it's still possible to work around those controls unless you create some YAML monstrosity that stops people from making the mistake in the first place.
I'm interested in learning more about this. How would we go about adding a secret only available to runners on the main branch? Is there a configuration option on Github to create a secret only available to runners on main?
Presumably anything configured via a .github workflow wouldn't assure safety, as those files can be edited to trigger unexpected actions like deploys on working branches. Our Github Action workflow yml file had a check to only deploy for changes to the main branch. The deploy got triggered because that check got removed from the workflow file in a commit on a working branch.
GHA is full of such obure behaviours. One I recently discovered is that one action can not trigger another:
If one action pushes a tag to the repo, `on:tag` does not trigger. The workaround apparently is to make the first action push the tag using a custom SSH key, which magically has the ability to trigger `on:tag`.
> When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN, with the exception of `workflow_dispatch` and `repository_dispatch`, will not create a new workflow run.
It has bitten me in the rear before too. I use this pattern a lot when I publish a new version, which tags a piece of code and then marks assets as part of that version (for provenance reasons I cannot rebuild code).
That actually seemed reasonable when I hit it, because you can easily accidentally have an action triggered on commit that makes a new commit, ending up in an infinite loop.
The workaround is to use a token tied to you instead of GitHub Actions, so you get charged (or run out of quota).
> The workaround is to use a token tied to you instead of GitHub Actions, so you get charged (or run out of quota).
You get charged no matter what, a personal access token doesn’t change anything.
If they are concerned about infinite loops then put a limit on how many workflows can be triggered but another workflow. Each time a workflow chains off another pass along some meta data of “runsDeep” and stop when that hits X, which can be configured.
No, requiring a PAT to kick off a workflow from a workflow is gross and makes zero sense. I don’t want every tag associated with my user, I want it to be generic, the repo itself should be attributed. The only way to solve this is to create (and pay for) another GH user that you create PAT tokens under. A bunch of overhead, cost, and complexity for no good reason.
Genuine question: what's the GitLab equivalent of GitHub Actions?
I'm using GitHub Actions to easily reuse some predefined job setup (like installing a certain Python version on Linux, macOS, Windows runners). For these tyoe of tasks, I find GitHub actions very useful and convenient. If you want to reuse predefined jobs, written by someone else, with GitLab CI/CD, what can I use?
include:component is usually what you want now, you can version your components (Semver), add a nice readme and it is somewhat integrated in the gitlab UI. Not sure about the other include: ones, but you can also define inputs for component and use them at arbitrary places like template variables.
Since the integration is done statically, it means gitlab can provide you a view of the pipeline script _after_ all components were included, but without actually running it.
We are using this and it is so nice to set up. I have a lot of gripes with other gitlab features (e.g. environments, esp. protected ones and their package registry) but this is one they nailed so far.
Doesn't include:component still require all your shell script to be written inside YAML? or is there a way to move the logic to a, for instance, .sh file and call it from YAML?
I realize this may be splitting hairs, but pedantically there's nothing in GitLab CI's model that requires shell; it is, as best I can tell, 100% docker image based. The most common setup is to use "script:" (or its "before_script:" and "after_script:" friends) but if you wanted to write your pipeline job in brainfuck, you could have your job be { image: example.com/brainfuckery:1, script: "" } and no shell required[1]
There is nothing.
Oh sure there is include, but that's like most gitlab features: it marks a nice shiny checkbox in some management presentation. But usefulness in the real world is limited.
But hey let's do secops oh no AI instead!
I'd say write a Python CLI (or any language you're comfortable with) which does all the actions (setup, deploy), download it and use it in the CI (or install on the runner images if you control them). That way you can use same workflow (command) in local development and CI.
There is a gitlab CI feature `include`, but you pretty much have to write shell scripts inside YAML, losing on whole developer experience (shellcheck etc..). I would recommend this way only if you can't factor your code into a CLI in proper language.
> A few days ago, someone compromised a popular GitHub Action. The response? "Just pin your dependencies to a hash." Except as comments also pointed out, almost no one does.
I'm surprised nobody has mentioned dependabot yet. It automates this, keeping action dependencies pinned by hash automatically whilst also bringing in stable upgrades.
Wasn’t part of the problem though that renovate was automatically upgrading people to the compromised hash? Or is that just the fault of people configuring it to be too aggressive with upgrades?
Well but that’s the problem. You cannot fully automate this. You have to manually check the diff of each dependency and only accept the dependabot PR if the changes are safe.
The only automation that I know of is cargo vet. Although it doesn’t work for GitHub Actions, the idea sounds useful. Basically, vet allows people who trust each other to vet updates. So one person verifies the diff and then approves the changes. Next, everyone who trusts this person can update the dependency automatically since it has been “vetted”.
Dependabot is only approximately as good as your tests. If you have holes in your testing that you can drive a bus through, you're gonna have a bad time.
We also, to your point, need more labels than @latest. Most of the time I want to wait a few days before taking latest, and if there have been more updates since that version, I probably don't want to touch anything for a little bit.
Common reason for 2 releases in 2 days: version 1 has a terrible bug in it that version 2 tries to fix. But we won't be certain about that one either until it's been a few more days with no patch for the patch for the patch.
GitHub Actions started off great as they were quickly iterating, but it very much seems that GitHub has taken its eye of the ball and the improvements have all but halted.
Sad to see Earthly halting development and Dagger jumping on the AI train :(. Hopefully we'll get a proper alternative.
On a related note, if you're considering https://www.blacksmith.sh/, you really should consider https://depot.dev/. We evaluated both but went with Depot because the team is insanely smart and they've solved some pretty neat challenges. One of the cooler features is that their caching works with the default actions/cache action. There's absolutely no need to switch out popular third party actions in favor of patched ones.
Sigh, this is awful. Earthly is/was not perfect, but is basically the most capable build tool I've ever used. Fingers crossed there's enough enthusiasm in the community to fork it (I'd be organizing it myself if I had any experience with Go at all)
We switched to Depot last week. Our Rust builds went down from 20+ minutes to 4-8 minutes. The easy setup and their docker builds with fast caching are really good.
What makes Depot so fast is that they use NVMe drives for local caching and they guarantee that the cache will always be available for the same builders. So you don't suffer from the cold-start problem or having to load your cache from slow object storage.
Thanks! We already use self-hosted runners on physical machines with NVMe drives that we assembled ourselves. I was wondering if there's something else you're doing for the caching.
Founder of Depot here. For image builds, we’ve done quite a bit of optimization to BuildKit for our image builders to make certain aspects of the builds fast like load, cache invalidations, etc.
We also do native multi-platform builds behind one build command. So you can call depot build —platform Linux/amd64,linux/arm64 and we will build on native Intel and ARM CPUs and skip all the emulation stuff. All of that adds up to really fast image builds.
Presumably the issue is that GH underpriced Actions such that it's not worth improving because driving more usage won't drive revenue, and that then forced prices down for everyone else because everyone fixed on the Actions pricing.
> Sad to see Earthly halting development and Dagger jumping on the AI train :(. Hopefully we'll get a proper alternative.
Hi, Dagger CEO here. We're advertising a new use case for Dagger (running AI agents) while continuing to support the original use case (running complex builds and tests). Dagger has always been a general purpose engine, and our community has always used it for more than just CI. It's still the exact same engine, CLI, SDKs and observability stack. It's not like we're discontinuing a product, to the contrary: we're getting more workloads on the platform, which benefits all our users.
Great to know. I think the fear is that so many companies are prioritizing AI workloads for the valuation bump rather than delivering actual meaningful value.
I completely understand that fear. I see lots of other tech companies making that mistake, throwing away a perfectly good product and market out of pure "FOMO". I really, really don't want us to be one of those companies.
I think what we're doing is different: we built a product that was always meant to be general purpose; encouraged our community to experiment with alternative use cases; and are now doubling down on a new use case, for the same product. We are still worried about the perception of a FOMO-driven AI pivot (and the reactions on this thread confirm that we still have work to do there); but we're confident that the product really is capable of supporting both.
Thank you for the thoughtful comments, I appreciate it.
583 comments
[ 3.1 ms ] story [ 323 ms ] threadCould we not usually say most software could be documented better. I do not think GitHub Actions ranks near the bottom in terms of documentation and user experience overall. I do understand your point though.
It also has the problem of not having a local dev runner for actions. The "inner loop" is atrociously slow and involves spamming your colleagues with "build failed" about a thousand times, whether you like it or not.
IMHO, a future DevOps runner system must be an open-source, local-first. Anything else is madness.
Right now we're in the "mainframe era" of DevOps, where we edit text files in baroque formats with virtually no tooling assistance, "submit" that to a proprietary batch system on a remote server that puts it into a queue... then come back after our coffee to read through the log printout.
I should buy a dot matrix printer to really immerse myself into the paradigm.
The entire code is a wonderful mess. We found that when we early-adopted ephemeral runners, that the control flow is full of races and the status code you get at the end is indicative of exactly nothing. So even if the backend is just having a hickup picking up a job with an obscure Azure error code, you better just throw that entire VM away, because you can't know if that runner will ever recover or has already done things to break the next run.
Although, I never saw a public announcement of this discontinuation, ADO is kind of abandoned AFAICT and even their landing page hints to use GitHub Enterprise instead [1].
[1] https://azure.microsoft.com/en-us/products/devops
There are better solutions out there.
I was doing things more than 20 years ago in Hudson that GHA can't do now.
A 1000% yes, because it means the default experice most devs have of CI is using ephemeral runners which is a massive win for security and build rot.
Every company I've worked at with stateful runners was a security incident begging to happen, not to mention builds that would do different things depending on what runner host you got placed on (devs manually installing different versions of things on hosts, etc)
And what are those?
There's also the Woodpecker CI fork, which has a very similar user experience: https://woodpecker-ci.org/
When combined with Docker images, it's quite pleasant to use - you define what environment you want for the CI/CD steps, what configuration/secrets you need and define the steps (which can also just be a collection of scripts that you can run locally if need be), that's it.
Standalone, so you can integrate it with Gogs, Gitea or similar solutions for source control and perhaps a bit simpler than GitLab CI (which I also think is lovely, though maintaining on-prem GitLab isn't quite a nice experience all the time, not that you have to do that).
Anda a lot more flexibility in terms of what can you do.
Maybe it was just the pain of switching but that was my initial impression.
[1] https://github.com/earthly/earthly/commit/6d7f6786ad9fa4392f... [2] https://github.com/earthly/earthly/commit/89d31fc014a8980a50...
I am really really hoping that someone (not me, I've already tried and failed) could slim it down into a single-purpose, self-contained, community maintainable tool ...
I wonder why they chose to move back to Github Actions rather than evaluate something like Buildkite? At least they didn't choose Cloud Build.
I think incremental progress in the CI front is chugging along nicely, and I really haven't seen any breathtaking improvements from other solutions I've tried, like CircleCI.
Recently switched to a company using Github, and assumed I'd be blown away by their offering because of their size.
Well, I was, but not in the way I'd hoped. They're absolutely awful in comparison, and I'm beyond confused how it got to that state.
If I were running a company and had to choose between the two, I'd pick Gitlab every time just because of Github actions.
I have some GitHub actions for some side projects and it just seems so much more confusing to setup for some reason.
// Luckily still a gitlab user, but recently forced to Microsoft Teams and office.
my condolences to you and your team for that switch; it's my 2nd used-and-disliked thing (right next to atlassian) - oh well
but one cool feature i found with ms teams that zoom did not have (some years ago - no clue now) is turning off incoming video so you dont have to be constantly distracted in meetings
edit: oh yeah, re github actions and the user that said: > Glad I’m not the only one
me too, me too; gh actions seem frustrating (from a user hardly using gh actions, and more gitlab things - even though gitlab seems pretty wonky at times, too)
If you want to make a CI performant, you'll need to use some of its features like caches, parallel workers, etc. And GHA usability really fall short there.
The only reason I put up with it is that it's free for open source projects and integrated in GitHub, so it took over Travis-ci a few years ago.
> For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
When the cli didnt have support for what I needed
one of them beeing echoing text to a file. to me, your comparison makes no sense.
Every time someone introduced a new way to use someone else's shared magic I feel nervous about using it. Like GitHub Actions. Perhaps it's time for me to dig into them a bit more and try to understand if/how they're safe to use. But I seem to remember just a few days ago someone mentioning a GitHub action getting hijacked?
For instance, AWS has a lot of actions they maintain to assist with common CI/CD needs with AWS services.
https://github.com/aws-actions
I will be stunned if this doesn't become a more popular attack vector over the next few years. Lots of valuable stuff sits in github, and they're a nearly-wide-open hole to access it.
Because the docs are crap perhaps? I prefer it, having used both professionally (and Jenkins, Circle, Travis), but I do think the docs are really bad. Even just the nesting of pages once you have them open, where is the bit with the actual bloody syntax reference, functions, context, etc.
A few years back I wanted to throw in the towel and write a more minimal GHA-compatible agent. I couldn't even find where in the code they were calling out to GitHub APIs (one goal was to have that first party progress UI experience). I don't know where I heard this, so big hearsay warning, but apparently nobody at GitHub can figure it out either.
I used GitHub actions when building a fin services app, so I absolutely used the hash to specify Action dependencies.
I agree that this should be the default, or even the required, way to pull in Action dependencies, but saying "almost no one does" is a pretty lame excuse when talking about your own risk. What other people do has no bearing on your options here.
Pin to hashes when pulling in Actions - it's much, much safer
"Defaults matter" is a common phrase, but equally true is: "the pattern everyone recommends including example documentation matters".
It is fair to criticise the usage of GH Actions, just like it's fair to criticise common usage patterns of MySQL that eat your data - even if smarter individuals (who learn from deep understanding, or from being burned) can effectively make correct decisions, since the population of users are so affected and have to learn the hard way or be educated.
This is for composite actions. For JS actions what if they don't lock dependencies but pull whatever newest package at action setup time? Same issue.
Would have to transitively fork everything and pin it myself, and then keep it updated.
Yes, your builds will work as expected for a stretch of time, but that period will come to an end, eventually.
Then one day you will be forced to update those pinned dependencies and you might find yourself having to upgrade through several major versions, with breaking changes and knock-on effects to the rest of your pipelines.
Allowing rolling updates to dependencies helps keep these maintenance tasks small and manageable across the lifetime of the software.
Though, yes, I prefer pinning dependencies for my personal projects. I don't see why things should break when I explicitly keep them the same.
Just make sure you don’t leak secrets to your PRs. Also I usually review changes in updated actions before merging them. It doesn’t take that much time, so far I’ve been perfectly fine with doing that.
[1]: https://docs.renovatebot.com/modules/manager/github-actions/...
The real problem is security vulnerabilities in these pinned dependencies. You end up making a choice between:
1. Pin and risk a malicious update.
2. Don't pin and have your dependencies get out of date and grow known security vulnerabilities.
Next to json I also used travisCI and appveyor for projects. And they all had the same (commit and pray) setup that ai hate. I wish if „act“ was a tool directly maintained by the GitHub folks though.
https://github.com/nektos/act
[1] https://github.com/actions/runner/blob/6654f6b3ded8463331fb0...
I had a coworker who called it Visual Sorta-Safe which is just about the best parody name I've ever heard in my entire career.
https://news.ycombinator.com/item?id=18983586
https://rewiring.bearblog.dev/blog/?q=azure
PS: I am not the author of any of these posts.
You have no idea how much I'd love that feature. Inasmuch as "save" is still a thing anyway. I don't miss explicit saves in IDEA, I see commit as the "real" save operation now, and I don't mind being able to hook that in an IDE-independent way.
I think the UX of git hooks has been sub-par for sure, but tools like the confusingly named pre-commit are helping there.
I don’t want to think about formatting, I just want everything to be consistent. A pre commit hook can run those tools for me, and if any changes occurred, it can add them to the commit.
If people want to die on a hill that is demonstrably causing problems for all of their coworkers then let em.
Example config: https://github.com/anttiharju/vmatch/blob/9e64b0636601c236a5...
Instead, have tooling to do that before committing (vscode format-on-save, or manually run a task), then have a pre-commit hook just do a sanity-check on that. It only needs to check modified files, so usually very fast.
Then, have an additional check on CI to verify formatting on all files. That should rarely be triggered, but helps to catch cases where the hooks were not run, for example from external contributes. That also makes it completely fine for this CI step to take a couple of minutes - you don't need that feedback immediately.
Though it would be sort of interesting or maybe just amusing if you made something like ssh-agent but for 'git commit' and your test runner. Only allow commits when all files are older than your last green test run.
In my experience, the best strategy is to minimize your use of it — call out to binaries or shell scripts and minimize your dependence on any of the GHA world. Makes it easier to test locally too.
I have done something similar with Jenkins and groovy CI library used by Jenkins pipeline. But it wasn't super simple since a lot of it assumed Jenkins. I wonder if there is a more cleaner open source option that doesn't assume any underlying platform.
Like teams.
Outside of locking down edit access to the .github workflow yml files I'm not sure how vulnerabilities like this can be prevented.
The problem is it's still possible to work around those controls unless you create some YAML monstrosity that stops people from making the mistake in the first place.
Presumably anything configured via a .github workflow wouldn't assure safety, as those files can be edited to trigger unexpected actions like deploys on working branches. Our Github Action workflow yml file had a check to only deploy for changes to the main branch. The deploy got triggered because that check got removed from the workflow file in a commit on a working branch.
But for actually good security CI and CD should be different tools.
You create an environment, restrict it to the main branch, add your secret to it and then tie your deploy workflow to it.
If someone runs that workflow against another branch it will run but it won’t be able to access those secrets.
[0] https://docs.github.com/en/actions/managing-workflow-runs-an...
If one action pushes a tag to the repo, `on:tag` does not trigger. The workaround apparently is to make the first action push the tag using a custom SSH key, which magically has the ability to trigger `on:tag`.
> When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN, with the exception of `workflow_dispatch` and `repository_dispatch`, will not create a new workflow run.
It has bitten me in the rear before too. I use this pattern a lot when I publish a new version, which tags a piece of code and then marks assets as part of that version (for provenance reasons I cannot rebuild code).
The workaround is to use a token tied to you instead of GitHub Actions, so you get charged (or run out of quota).
You get charged no matter what, a personal access token doesn’t change anything.
If they are concerned about infinite loops then put a limit on how many workflows can be triggered but another workflow. Each time a workflow chains off another pass along some meta data of “runsDeep” and stop when that hits X, which can be configured.
No, requiring a PAT to kick off a workflow from a workflow is gross and makes zero sense. I don’t want every tag associated with my user, I want it to be generic, the repo itself should be attributed. The only way to solve this is to create (and pay for) another GH user that you create PAT tokens under. A bunch of overhead, cost, and complexity for no good reason.
I'm using GitHub Actions to easily reuse some predefined job setup (like installing a certain Python version on Linux, macOS, Windows runners). For these tyoe of tasks, I find GitHub actions very useful and convenient. If you want to reuse predefined jobs, written by someone else, with GitLab CI/CD, what can I use?
https://docs.gitlab.com/ci/yaml/#includeremote
Since the integration is done statically, it means gitlab can provide you a view of the pipeline script _after_ all components were included, but without actually running it.
We are using this and it is so nice to set up. I have a lot of gripes with other gitlab features (e.g. environments, esp. protected ones and their package registry) but this is one they nailed so far.
1: although TIL that the "script:" field itself is actually required in GLCI https://docs.gitlab.com/ci/yaml/#script
There is a gitlab CI feature `include`, but you pretty much have to write shell scripts inside YAML, losing on whole developer experience (shellcheck etc..). I would recommend this way only if you can't factor your code into a CLI in proper language.
I'm surprised nobody has mentioned dependabot yet. It automates this, keeping action dependencies pinned by hash automatically whilst also bringing in stable upgrades.
The only automation that I know of is cargo vet. Although it doesn’t work for GitHub Actions, the idea sounds useful. Basically, vet allows people who trust each other to vet updates. So one person verifies the diff and then approves the changes. Next, everyone who trusts this person can update the dependency automatically since it has been “vetted”.
[1]: https://github.com/mozilla/cargo-vet
We also, to your point, need more labels than @latest. Most of the time I want to wait a few days before taking latest, and if there have been more updates since that version, I probably don't want to touch anything for a little bit.
Common reason for 2 releases in 2 days: version 1 has a terrible bug in it that version 2 tries to fix. But we won't be certain about that one either until it's been a few more days with no patch for the patch for the patch.
It's really upsetting how little attention Actions is getting these days (<https://github.com/orgs/community/discussions/categories/act...> tells the story -- the most popular issues have gone completely unanswered).
Sad to see Earthly halting development and Dagger jumping on the AI train :(. Hopefully we'll get a proper alternative.
On a related note, if you're considering https://www.blacksmith.sh/, you really should consider https://depot.dev/. We evaluated both but went with Depot because the team is insanely smart and they've solved some pretty neat challenges. One of the cooler features is that their caching works with the default actions/cache action. There's absolutely no need to switch out popular third party actions in favor of patched ones.
What happened there?
What makes Depot so fast is that they use NVMe drives for local caching and they guarantee that the cache will always be available for the same builders. So you don't suffer from the cold-start problem or having to load your cache from slow object storage.
We also do native multi-platform builds behind one build command. So you can call depot build —platform Linux/amd64,linux/arm64 and we will build on native Intel and ARM CPUs and skip all the emulation stuff. All of that adds up to really fast image builds.
Hopefully that’s helpful!
You wouldn't really have to change anything on your dockerfile to leverage this and see significant speed up.
The docs are here: https://docs.warpbuild.com/docker-builders#usage
Hi, Dagger CEO here. We're advertising a new use case for Dagger (running AI agents) while continuing to support the original use case (running complex builds and tests). Dagger has always been a general purpose engine, and our community has always used it for more than just CI. It's still the exact same engine, CLI, SDKs and observability stack. It's not like we're discontinuing a product, to the contrary: we're getting more workloads on the platform, which benefits all our users.
I think what we're doing is different: we built a product that was always meant to be general purpose; encouraged our community to experiment with alternative use cases; and are now doubling down on a new use case, for the same product. We are still worried about the perception of a FOMO-driven AI pivot (and the reactions on this thread confirm that we still have work to do there); but we're confident that the product really is capable of supporting both.
Thank you for the thoughtful comments, I appreciate it.
Example:
https://github.com/actions/runner/pull/2477#issuecomment-244...