GitHub Actions Are a Problem (felix-knorr.net)

What I always guessed: if you don't want to bite the bullet of trusting SRE scripters (with an s) to the role of building the CD that actually makes sense for your use case, why do you expect some random company will have more of your interest at heart? At least with Joe's python deployment scripts, Jane can do a code review. So much of cloud CI/CD lingo translates to "believe in magic, and also our bigness-based authority" when actually proper software deploys itself properly without making a fuss or glueing your fingers with sticky Yaml...

I've been migrating some Workflows to dagger.io. So far it's amazing.

This are the reasons I don't use GitHub actions. I don't want to add another hard dependency into my development environment and testing actions looks like a very slow feedback loop, very time consuming.

If you want the ability to run as much of your GitHub Actions locally as possible (which you should! Testing via pushing commits is painful) you can arrange your actions so the majority of the logic lives in shell scripts (or Python scripts or whatever) that are called by the workflow.

Then you can run those scripts locally and iterate in them before you try them on GitHub Actions.

I agree: Actions would be more useful if I could run the entire stack locally (an official implementation, not a community-maintained clone). But it's not a big enough inconvenience for me to care very much.

    > gh actions run

My strategy is just to keep the yaml to a minimum and call python scripts within the action to do anything complicated. The GHA yaml is just used for defining when the action runs, and on what architectures, etc. It's worked well for me.

GHA is "good enough" (GitLab CI was/is much better IMO), and it'll likely never go away because of how dominant GitHub is.

Also, a simpler solution that takes a bit more discipline is writing all your CI tasks as scripts/makefile|justfile targets/etc and using containers to run them in GHA.

IIRC containers are a little slower than running scripts in the VM directly (more than it must necessarily be), and also cross-platform builds might be slightly more complicated...

It's scary how closed-source our development platforms are.

That's why building a fully-self-contained VCS, one that includes CI, is on my roadmap, but I fear the open part of our industry will fail before I finish.

I love GitHub Actions for basic CI/CD use cases, but I agree with the criticisms that testing locally is a major pain point and coding as a YAML DSL is a minor pain point.

> Defining a language to run scripts in parallel with some constraints is actually quite easy if you just pass the script to an existing interpreter and let the language mainly deal with the constraints around order and parallelism (and code reuse)

Isn't this Make?

Github Actions gives me literally free server time across an extremely wide range of OS'es that I dont have to worry about at all, including Windows and OSX, which I therefore dont have to deal with in any way, buy any license keys, none of that. It's nothing short of miraculous for us as it's how Python projects can have binary wheel files for dozens of OSes and Python versions: https://pypi.org/project/SQLAlchemy/#files . This task remained impossible for years (to be clear: because I don't have a server farm, or Windows licenses, or however you'd run OSX on a headless server, or any kind of resources to fund / manage dozens of images and keep them running, or any of that) until GH Actions made it possible.

Now is this all part of Microsoft's evil plan ? It probably is! But unless someone else wants to give me a free server farm that includes Windows / OSX up and running without me paying anything / writing containers / storing images / etc. I dont see this aspect of Github actions losing any popularity.

Local actions can be defined with a dockerfile. I haven’t used this approach extensively, but it seems like a simple way to use actions while also removing significant dependence on the platform.

Why not just use act? It was made for this exact set of problems?

https://github.com/nektos/act

Yes Github actions is (are?) a mess lacking conceptual integrity and widely misused. But we do the following with success:

1. Jobs should always (just) execute a script or program. This allows running outside the CI system.

2. To test/debug CI jobs, use act.

3. To test/debug more complex scenarios spin up a Gitea instance (which provides a Github clone wrapping act).

  [checks/lint]     Failure - Main actions/setup-go@v3
  [checks/lint] failed to create exec: Error response from daemon: Container b9059f831d3a1549c9902cabc7e5258231b2a7291b2692eaa33f976794059738 is not running

The over-complication of software deployments is just mind-blowing. I hope the end of cheap money reigns in the insane bloat that is DevOps today. It doesn’t have to be this way, but there continues to be too few rational adults in the room.

The runner is open source, but once you read the code you'd wish it wasn't.

https://github.com/actions/runner

For anyone in the audience who didn't know: GitHub Actions is based on Visual Studio Team Foundation Server's CI, and later Azure DevOps. And nobody, including the current maintainers, seem to know exactly how it all works (or how it doesn't).[1] The random exit codes is just the cherry on top!

[1]: https://github.com/actions/runner/issues/510

This was my first startup idea, and the first one we executed on: an open-source CI that runs on your machine exactly like it does in the cloud and doesn’t use YAML as the main configuration format.

Others have tried the same, including Cicada, which was built by the same folks as Fig.

Here are the failure modes for this idea:

- CI succeeds most of the time. It’s only in the minority of cases that running CI locally is useful, as you don’t need to debug that often.

- The only case where CI fails more often is when building pipelines. If you’re building pipelines then GitHub actions already has bazillions of actions and recipes available. Their “whole product” is a huge advantage (shout-out to readers of Crossing the Chasm).

- The friction to adopt GHA is much smaller as it’s one click away on GitHub and all your other stuff is already there.

- Making an MVP of CI is really difficult because it requires lots of features. You can’t even start if you don’t have things like permissions, outputs for different workflows, and a significant amount of infra for orchestration.

- Even when you have an MVP the switching costs of CI are ridiculously high, and it’s a critical piece of infrastructure, so it’s difficult to get people to move over if you’re only offering marginal gains.

- GitHub is owned by Microsoft, which owns Azure. Their compute will be way cheaper 99% of the time. Your product advantage must be significant enough to justify higher costs. In any case, their margin will be ridiculously higher.

It’s funny how many people get to the same conclusion, yet this is a really difficult problem to solve.

TL;DR: the author idea is great and many people thought about it before. Executing such an idea is extremely difficult though.

We have built an open-source generic workflow engine to run arbitrary scripts (https://windmill.dev) with a vscode extension to build the yaml using a low-code builder and each individual script in their dedicated python/ts files so you get your full editor assistants https://youtu.be/aSOF6AzyDr8?t=116

One of the area we are expanding next is a github app so you get exactly the same UX as github actions but running windmill workflows on your windmill workers.

I actually really like GitHub actions, I read the article and while I get some of the concerns, others I don't understand. In any case the author's situation doesn't apply to me, and I wanted to share something I really liked about GHA.

So, I recently figured out a way to host a remote browser on them by using an Ngrok tunnel. It's really cool to see BrowserBox running from inside a GitHub action container. I literally couldn't believe it actually worked when I first figured it out!

I was so excited. It started as just this tech prototype in my mind (could this be possible? Probably not but I Feel like it could be). And to see it actually achieved so cool! :)

It has made CI integration testing SO much easier and more repeatable. I love that it can just run it up on Ubuntu and I can verify.

Anyhow, I thought this was so cool, and such a useful way for people to either just get started with BrowserBox trying it out, or even run a quick little VPN-like/proxy browser from another region. I've even logged into HN from it on the sly. I liked this whoel concept so much, that I even wrote an action that integrates with issues to make the process as easy as possible for people.

Basically you can just clone or fork the repo: https://github.com/BrowserBox/BrowserBox and then open an issue and pick the template that is like "Make VPN". The login link will get published in the repo. The link is not private (unless you make your fork or template private) and there's a bit of setup with your ngrok API key (free is OK) but the issue conversation automatically guides you through all that.

I thought this was so cool (free server time, actually working app), that I even created another version that uses MS Edge under the hood instead of Chrome in the original, just to show how easy it is: https://github.com/MSEdgeLord/EdgeLord

Just a niggle is that the other services we normally have (secure doc viewer, audio, remote devtools) do not work as ngrok only maps 1 port. I could use an ngrok config file I think to fix that but somehow, easy as that is, I have not gotten around to it! Another niggle is I noticed the auto-tab opening used in the GHA demo seems a little funky lately, and you may need to manually reload or resize them to un-wonkify it. Probably a little regression!

Anyway! :)

> Hosting a git repo is hardly more than providing a file system and SSH access. The actual mechanism they use to keep you on their platform is the CI-pipelines

Then why was GitHub so popular for the 10+ years it had no built in CI system?

A big problem with making an abstraction over CI providers is that they are, on the surface, very little more than wrappers around steps made up of small scripts, but that abstraction leaks like hell. So I think you would wind up needing to go quite high-level to make something that is useful here, but even if you do do this, the variance in runtime environment will make it very difficult to make sure things are relatively portable, and attempts to work around this will lead to problems (e.g. you could just wrap everything in Docker, but fetching large images on each CI invocation isn't really a good use of resources.)

Let's say you have a project using CMake with vcpkg. Vcpkg has support for the GitHub Actions Cache API, so it can integrate very well into GHA and provide good performance for caching dependencies so they do not need to be built on each invocation. Doesn't seem too unreasonable, you could have a vcpkg+CMake plugin of some kind that tries to do the best thing for each CI provider. But, what version of vcpkg do you use? what version of CMake do you use? Vcpkg is essentially just a rolling-release Git repo, there's no specific version to use. But it would save a lot of time to make use of the vcpkg installation that already exists on GitHub Actions images. And obviously, a tool like CMake depends super heavily on the environment around it. A lot of people use the Ninja provider for better performance... but if you use it on Windows, you need to be careful: CMake with the Ninja provider on Windows can not automatically pick up the Visual Studio installation, unlike the default Windows provider (which is, well, Visual Studio.) So instead it might pick up MinGW, which is often unexpected, and does not always work correctly. Does Microsoft have an answer for how to deal with this? Not really. You might think to use the official setup-msbuild step, which has been mistakenly recommended by Microsoft, but it only sets up enough environment to do .NET development.

So actually your CI abstraction needs plugins that deal with:

- Every CI provider

- Using every operating system

- For each build step

And, also, it needs to be updated fairly frequently, especially if you want to keep up to date on e.g. macOS versions.

Personally, I think this is an unwinnable situation, though I'd love to be proven wrong. I think it would take something fairly dramatic to actually solve this problem most of the way, though. Something that might need a lot of collaboration between stakeholders, possibly the CI providers themselves, and probably even build/packaging tools at some level.

One thing I thought was interesting was the approach of using Nix[1]. I've tried rudimentary implementations of Nix with GitHub Actions though, and it does definitely have some issues still. The most obvious issue is that it doesn't really support Windows, followed by the fact that binaries made using Nix are generally not going to be very easy to distribute (unless you're building an OCI or AppImage or something like that.) But for just running tests, it's enticing at least, and it can do a whole lot of caching in theory (for dependencies; you'll still need to handle caching of intermediate files for local tests/builds in your build system underneath Nix.)

The more I think about this problem, the more I wonder which angle is the best one to come at it from. Is this ultimately a problem about software SDKs in disguise?

[1]: https://determinate.systems/posts/nix-github-actions

I've always been opposing push-to-deploy, unless you actually have per-branch, ephemeral test environments, and take the required care to set up your data fixtures - which is a major technical challenge on any non-trivial project.

The trick is to keep the deployment code simple and stupid, while making the process transparent and dependable for the team.

For production environments, rather than thinking in terms of pushing a commit, you want to reframe the problem as promoting a version from a staging environment - preferably subject to approval from QA and/or project owner. ChatOps is great for that, and it doesn't have to be painfully complicated, or full of hacky Hubot code - you can get huge ROI from a simple script[1] that posts a message at the start&end of a deployment, which includes a version string (autogenerated from a git tag[2]), environment, maybe your login/host name ("$(id -un)@$(hostname)"). CI can build the artifacts, then your job is just "./deploy prod v3.14", which is also easily delegated.

[1]:

    def slack_post(data):
        req = urllib.request.Request(
            "https://hooks.slack.com/services/xxx/xxx/xxx",
            headers={"content-type": "application/json"},
            data=json.dumps(data).encode("utf8"),
            method="POST",
        )
        res = urllib.request.urlopen(req)
        assert res.getcode() in range(200, 300)
        return res

[2]:

    #!/bin/sh
    set -eu
    slugify() {
        sed -E -e 's/[^A-Za-z0-9]/-/g' -e 's/-+/-/g' -e 's/(^-|-$)//g'
    }
    git_tag=$(git describe --tags --dirty --always)
    git_branch=$(git rev-parse --abbrev-ref HEAD | slugify)
    case "${git_branch}" in
        master|HEAD) echo "${git_tag}"; ;;
        *) echo "${git_tag}-${git_branch}"; ;;
    esac

Yaml shouldn’t be used for anything except configuration files. It’s just lazy for services to use it as a kind of DSL

These kinds of systems should use an actual programming language or their own DSL

I’d like to see better centralized support generally from CI systems. Six months of steady migration and a custom DSL seems alarmingly common.

CI Wishlist:

* testable outside of CI

* required steps based on repo characteristics (ex: security scans)

* central step and workflow definitions

* controlled rollouts/rollbacks to simplify migrations

Crazy thing about this and is frustratingly obvious, as I become a little older in IT, is that this is just a workflow problem.

CI build pipelines aren't even really in the class of workflow problems that are even difficult to implement... workflow engines start you down a rabbit hole towards different models of computation that eventually ends up with a turing machine anyway, but I digress.

It's just really kind of ridiculous that the same basic problem gets re-implemented literally thousands and thousands of times across different platforms and use cases. The fact that the current implementation is based on some Frankenstein of former visual studio code and other stuff really, isn't that surprising.

Of course! Standardizing workflows leads precisely to the XkCD referenced in the article, in standards produced by the workflow vendors of of course been horrendous and focused on preserving or enhancing market share, and we're done in the era of XML.

The issue of course is that workflows are an "Enterprise" feature and solution.

Perhaps someday in the halcyon future someone will actually address a useful cross platform workflow model I could conserve algorithms, processes, techniques, and maybe even plugins. Because right now the workflow engine is balkanized across every single CI platform.

I would respectfully suggest that the author is misusing CI. If you have trouble running your tests locally, you have a problem. If you have trouble deploying from your local code, you have a problem. All of those capabilities should exist as simple scripts in your project already. Once you have that done, the CI yaml is a simple glue layer that defines an order of operations, e.g.:

1. Run static tests

2. If those pass, run unit/integration tests

3. If those pass, deploy

If you find yourself screaming about YAML, you're leaning too heavily on it and need to refactor your project's scripts.

Maybe a good question to ask would be "if I had to switch to another CI system today, how hard would it be?" If the answer is "hard", perhaps you're leaning too heavily on it and need to refactor your project's scripts.

  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v2
  - name: Build and push mwdb-core image
    uses: docker/build-push-action@v4
    with:
      file: ./deploy/docker/Dockerfile
      tags: |
        certpl/mwdb:${{ github.sha }}
        certpl/mwdb:master
      cache-from: |
        type=registry,ref=certpl/mwdb:buildcache
      outputs: type=docker,dest=./mwdb-image
  - name: Upload mwdb-core image
    uses: actions/upload-artifact@v3
    with:
      name: mwdb-image
      path: mwdb-image