Ask HN: Is “Distributed CI” Possible?
Is it possible to have distributed CI in the following sense:
- The “CI” is a program that checks if the code is okay (like runs a test suite)
- Anyone can run it and get the result
- A central server/node can receive the output and SHA1 (that the code was run on) and quickly check if the CI passed
- It takes some effort to cheat the procedure—it doesn't have to be bullet proof (it's meant to be used “among friends”)
- No proof-of-work or something else that burns up electricity for questionable gain
It seems that a simple plaintext script CI is not sufficient since whatever node has run the script can just change the script: `return true`. But if you can either encrypt the program or obfuscate it enough (for a decompiler) then you might be able to implement something like this:
<do tests>
if they pass: calculate the output of the CI based on the stable
success-output of the tests + a secret + the SHA1
28 comments
[ 4.1 ms ] story [ 252 ms ] threadBut am glad if it exists.
There are some theoretic foundations in this line of work: https://en.wikipedia.org/wiki/Verifiable_computing
You either trust the client to do the requested work, or you don't. If you can't trust the client to use the appropriate CI script version, then you can't trust anything the client responds with.
An example of this is how hacked DVD/BluRay drives work, where they fib to the driver about what byte is in which memory address.
You can issue the work to multiple clients, and treat it like a node failure. If you have a quorum of executions then you can trust that result. However, that's similar to the disallowed proof-of-work requirement.
No matter what, the client has to trust that the server isn't out to get them.
From what I understand, requiring developers to assert they run the tests and not run them again from a clean checkout before pushing to prod is likely a compliance problem (SOX, PCI, etc).
ofc this only works on smallish teams that push / pull very regularly and have good communication around who is deploying when.
it used to be a shout over a desk partition.
we go faster than you and break far far less than you
https://nixos.wiki/wiki/Hydra
https://nixos.wiki/wiki/Binary_Cache
[1] https://github.com/buildfoundation/mainframer
In this setup the nix daemon on the master can delegate build tasks to other build machines, let's call them agents. These agents have a feature set, arch set and a speed factor to steer task distribution. The feature sets define if, e.g., a machine can compile builds in a sandbox VM. The arch set is for cross compilation support.
You could schedule code checks as nix checks, e.g., linting and validarion could be done by exporting the output as a derivation.
Basically what you're describing is reproducible builds. The crypto stuff won't work because homomorphic encryption doesn't work, and neither do trusted compute platforms.
If it's for work, look into cryptographic signing (probably PKI) and as long as the signing keys are only issued to trusted agents, you can treat any result (plus Git hash) with a valid signature as a valid result.
Also any obfuscation can be removed and any compiled code can be decompiled.
Maybe zero knowledge proofs?
There's also problem of incentive. Why would anyone want to run your builds? So you can run theirs? In that case why you won't run yours and their theirs and save the overhead?
In retrospect, it's remarkable that it's been running as long as it has with little major incident.
This is already possibly and being done by some, you of course need to trust the node, because that could swap out tool on you, but there are various ways to push the attestation further down the stack, into the hardware and CPU. You'll also want to think about how your dependencies fit into the trust model as well, lots of scary stuff can happen there (i.e. things like dependency confusion)
I'm particularly fond of Dagger, as they are doing a lot of the heavy lifting on this front, and their tool makes local -> CI a joy. You can run Dagger in any larger CI system, they focus on replacing your Dockerfiles and other build scripts as one of the next-gen solutions build on BuildKit
Compare to this actual working solution: I run gitlab runners on spot instances (or at home for personal projects) The runners check in with the server, get a job, process it and send the results. A CI job can have multiple parts. A server can be handling multiple client requests simultaneously.
CI is a resource intensive process so you want beefy machines but they shouldn’t run when not needed (at night if you’re in one time zone, when unloaded if you’re distributed). Enter autoscailing groups and spot instances. 4x the speed for 10% of the cost of always on and “on demand” priced instances. Or for personal projects, speedy when needed idle when not.
Compare this solution to what you’re working on. You’ll see you’re doing a lot of extra work for no additional value.
It turned out all right, but I dropped it for a few reasons. Overall, it was not worth the effort and complexity. First, making it fast and safe was a PITA compared to just...running a few commands in a normal CI environment. Second, for manifest-style hashes, you have to either exclude the .git folder (which means no diffing/affected commands) or try to clean it up somehow (which obviously can be done, but if you do that in a Docker layer it screws up your caching, etc.). Third, it generated an ungodly amount of local cached data. There are a few other reasons I dropped it, but they're generally related to safety/optimization/caching/git.
1. You update a copy of a repo which defines all inputs in a repeatable way (all inputs are either part of the repo or their hashes are verified during build). Nix does this.
2. You upload the build outputs to a publicly available cache server, like Cachix.
3. You send the changes to someone else for inclusion in their repo.
4. They verify that your code doesn't do anything that looks like sandbox breaking, and that at least all the original tests are still in place. Or they have their own way to keep the run sandboxed and to test coverage.
5. They run the code (pointing at the cache server), which should reuse the cached results and finish almost immediately.
Running CI is typically done internally with the stakeholders of the code all aligned on a company's path, aka shared buy-in towards a successful outcome, so why would anyone want to cheat their CI test runs? If its a public or open source project, anyone can download and run tests and look at results.
But it requires some work to move all your build an test targets to bazel.
But for you case big problem, that you need to first write 100% specifications and 100% tests, and only then your CI could make automatic checks, and this is not typical for open source unfortunately.