> I find the most frustrating part of using CI to be to wait for a CI run to finish on a server and then try to deduce from the run log what went wrong. I’ve alleviated this by writing an extension to rad to run CI locally: rad-ci.
Agreed. I've got a crazy idea that I think might help...
Most tests have a step where you collect some data, and another step where you make assertions about that data. Normally, that data only ever lived in a variable, so it is not kept around for later analysis. All you get when you're viewing a failed test is logs with either exception or a failed assertion. It's not enough to tell a full story, and I think this contributes to the frustration you're talking about.
I've been playing with the idea that all of the data generation should happen first (since it's the slow part), it then gets added to a commit (overwriting data from the previous CI run) and then all of the assertions should run afterwards (this is typically very fast).
So when CI fails, you can pull the updated branch and either:
- rerun the assertions without bothering to regenerate the data (faster, and useful if the fix is changing an assertion)
- diff the new data against data from the previous run (often instructive about the nature of the breakage)
- regenerate the data and diff it against whatever caused CI to fail (useful for knowing that your change will indeed make CI happy once more)
Most tools are uncomfortable with using git to transfer state from the failed CI run to your local machine so you can just rerun the relevant parts locally, so there's some hackery involved, but when it works out it feels like a superpower.
nix-ci.com is built with this as one of the two central features.
The other is that it figures out what to do by itself; you don't have to write any YAML.
The whole point we started using automation servers as an integration point was to avoid the "it works on my machine" drama. (Have watched at least 5 seasons of it - they were all painful!).
+1 on running the test harness locally though (where feasible) before triggering the CI server.
I've been using brisk to run my CI from my local machine (so it runs in the cloud from my local terminal). The workflow is just a drop in replacement for local running. They've recently changed their backend and it seems to be working pretty smoothly. It works very well with AI agents too that are running in the terminal - they can run my tests for me if they make a change and it doesn't kill my machine.
I've poked at this a few times, and I think it breaks down for CI that needs/wants to run integration tests against other services. Eg it wants to spin up a Postgres server to actually execute queries against.
Managing those lifetimes is annoying, especially when it needs to work on desktops too. On the server side, you can do things like spin up a VM that CI runs in, use Docker in the VM to make dependencies in containers, and then delete the whole VM.
That's a lot of tooling to do locally though, and even then it's local but has so many abstractions that it might as well be running in the cloud.
Yeah, and I will never understand why developers accept anything less. GitHub CI is really bad for this. GitLab is a lot better as you can just run the exact thing through Docker locally. I like tools like tox too that automate your whole test matrix and do it locally just as well.
There's a difference between small scale CI and large scale CI.
Small scale: a project is almost small enough to run the build and tests locally, but you still want to have a consistent environment and avoid "works on my machine" problems.
Large scale: a project is so large that you need to leverage remote, distributed computing to run everything with a reasonable feedback loop, ideally under 10 minutes.
The opposite ends of the spectrum warrant different solutions. For small scale, actually being able to run the whole CI stack locally is ideal. For large scale, it's not feasible.
> A CI system that’s a joy to use, that sounds like a fantasy. What would it even be like? What would make using a CI system joyful to you?
I spent the past few years building RWX[1] to make a CI system joyful to use for large scale projects.
- Local CLI to read the workflow definitions locally and then run remotely. That way can you test changes to workflow definitions without having to commit and push.
- Remote breakpoints to pause execution at any point and connect via ssh, which is necessary when running on remote infrastructure.
- Automatic content-based caching with sandboxed executions, so that you can skip all of the duplicative steps that large scale CI otherwise would. Sandboxing ensures that the cache never produces false positives.
- Graph-based task definitions, rather than the 1 job : 1 VM model. This results in automatic and maximum parallelization, with no redundancy in setup for each job.
- The graph based model also provides an improved retry experience, and more flexibility in resource allocation. For example, one task in the DAG can crank up the CPU and memory without having to run more resources for downstream tasks (steps in other platforms).
We've made dozens of other improvements to the UX for projects with large build and test workflows. Big engineering teams love the experience.
Op, Radicle had a very glitchy style home page before it went more 8-bit. Do you have an archive of that anywhere? I’d like to use it as reference for a style period in design!
12 comments
[ 3.2 ms ] story [ 26.3 ms ] threadlocally running CI should be more common
Most tests have a step where you collect some data, and another step where you make assertions about that data. Normally, that data only ever lived in a variable, so it is not kept around for later analysis. All you get when you're viewing a failed test is logs with either exception or a failed assertion. It's not enough to tell a full story, and I think this contributes to the frustration you're talking about.
I've been playing with the idea that all of the data generation should happen first (since it's the slow part), it then gets added to a commit (overwriting data from the previous CI run) and then all of the assertions should run afterwards (this is typically very fast).
So when CI fails, you can pull the updated branch and either:
- rerun the assertions without bothering to regenerate the data (faster, and useful if the fix is changing an assertion)
- diff the new data against data from the previous run (often instructive about the nature of the breakage)
- regenerate the data and diff it against whatever caused CI to fail (useful for knowing that your change will indeed make CI happy once more)
Most tools are uncomfortable with using git to transfer state from the failed CI run to your local machine so you can just rerun the relevant parts locally, so there's some hackery involved, but when it works out it feels like a superpower.
And yet, that's technically not CI.
The whole point we started using automation servers as an integration point was to avoid the "it works on my machine" drama. (Have watched at least 5 seasons of it - they were all painful!).
+1 on running the test harness locally though (where feasible) before triggering the CI server.
Managing those lifetimes is annoying, especially when it needs to work on desktops too. On the server side, you can do things like spin up a VM that CI runs in, use Docker in the VM to make dependencies in containers, and then delete the whole VM.
That's a lot of tooling to do locally though, and even then it's local but has so many abstractions that it might as well be running in the cloud.
Small scale: a project is almost small enough to run the build and tests locally, but you still want to have a consistent environment and avoid "works on my machine" problems.
Large scale: a project is so large that you need to leverage remote, distributed computing to run everything with a reasonable feedback loop, ideally under 10 minutes.
The opposite ends of the spectrum warrant different solutions. For small scale, actually being able to run the whole CI stack locally is ideal. For large scale, it's not feasible.
> A CI system that’s a joy to use, that sounds like a fantasy. What would it even be like? What would make using a CI system joyful to you?
I spent the past few years building RWX[1] to make a CI system joyful to use for large scale projects.
- Local CLI to read the workflow definitions locally and then run remotely. That way can you test changes to workflow definitions without having to commit and push.
- Remote breakpoints to pause execution at any point and connect via ssh, which is necessary when running on remote infrastructure.
- Automatic content-based caching with sandboxed executions, so that you can skip all of the duplicative steps that large scale CI otherwise would. Sandboxing ensures that the cache never produces false positives.
- Graph-based task definitions, rather than the 1 job : 1 VM model. This results in automatic and maximum parallelization, with no redundancy in setup for each job.
- The graph based model also provides an improved retry experience, and more flexibility in resource allocation. For example, one task in the DAG can crank up the CPU and memory without having to run more resources for downstream tasks (steps in other platforms).
We've made dozens of other improvements to the UX for projects with large build and test workflows. Big engineering teams love the experience.
[1] https://rwx.com