HN Explorer

Ask HN: Avoid/Prove CI works in a PR without running it?

3 points by jiehong ↗ HN
When developing in a repository, it's common to have jobs like Github's Actions triggered to check everything in the PR (build, tests, coverage, etc.).

Teams often try to speed up that process. Yet, the developer usually run some of that locally, and the job runs those steps again when the PR is created.

I am wondering if there could be a way to _prove_ that you've run a series of step locally so that the CI job can bypass those steps?

I thought that maybe a cryptographic string could be generated only by running a build, commit this, and this string could be verified without having to run the build again (easy to check, hard to generate).

(maybe something like a private key + commit hash + signature generated only by running the build, and the server checks the signature validity. Or maybe something from the blockchain?)

13 comments

[ 3.0 ms ] story [ 40.8 ms ] thread
re-thc
I rather not trust any local runs even if they're valid. It's likely not the same infrastructure, configuration etc. The CI run could catch things otherwise.

Lots of local environments also cache certain things or have things preconfigured.

Yes, it's best to speed up the process rather than skipping it.

Unless we're in the C/C++/Rust/etc space, the whole process should be relatively quick. Otherwise something else is wrong.

jiehong
The point of provable trust is also to take I to account the environment IMO.

Aka, zero trust computing.

re-thc
> The point of provable trust is also to take I to account the environment IMO.

To ask everyone to stick a Graviton in their home and run Amazon Linux to do work?

I rather the PR take a bit longer to run...

viraptor
I think you're looking at the wrong side of the issue. If you're working in a team, why would the developers ever be so hostile that you need cryptography to validate this information? Why isn't running "set($commit:tested, true)" on some shared database enough?

On the other hand, you're missing half the benefits of the CI by not ensuring the tests run in a common, predictable environment. The tests may pass for the developer but not elsewhere by accident.

jiehong
Questioning why something is not enough isn’t the point of my question. The environment meeting requirements could be part of the certification process.

Just like if you mine crypto currencies, it works on any computer, and the result can be checked but not forged.

viraptor
You can't do that. You know the test, you know the expected answers. You can trivially forge the test being run. There's no way you can get around it for deterministic tests. You could write only nondeterministic tests and require a unique answer every time... but that gets rid of any other time gains you got from the system.
viraptor
Actually no, even nondeterministic tests won't work, because you can just generate the answers assuming the changed source, but submit it for the change you actually wanted. Unless you DRM the whole machine successfully (good luck), there's no way to do it.
jiehong
Fair enough.

I guess if the testing framework of the language used would take into account the hash of the current file and imported sources and generate a signature based on that, it could detect on which content it ran, and certify this, somehow.

But I guess it's your DRM idea: the testing lib should not be tampered with in the first place.

viraptor
That's an area of TOCTOU issues. You can either hash the file and then replace it before running, or replace the code which was supposed to do the hashing with a constant value you want, or just run in a VM, freeze it, and replace the hash in memory at the time you want. Unless you perfectly control everything: drive, system, application - you can't validate anything.

That is unless you want to run in a signed trusted computing enclave, but those have at least one security failure a year in practice.

re-thc
> The environment meeting requirements could be part of the certification process.

Have you seen some of the stacks used these days? 90% of the time it isn't even replicable locally. And even if it did it'd break 1/2 the time.

- There are endless SaaS products that won't give an environment per developer and the shared environment is different to production

- There are so many cloud-based constructs like SES, S3, SQS, etc that replicating it all locally turns into a giant mess

- You are forced to run a local kubernetes cluster taking a long time to start and forever to debug

The list goes on. The better place to start is that people need to stop with this outsource mentality, simplify things and go back to the basics.

verdverm
You'll probably find https://www.sigstore.dev/ interesting

You cannot trust developer machine consistency or integrity, like never trusting user input in an app. Best to have a controlled build environment. CI is cheap all things considered, much cheaper than a supply chain exploit

I've been using Dagger more, and there is the potential to share the cache between developer machines and CI, which in theory would address your "remove duplication of work" idea, while also supporting Sigstore features.

jiehong
Thanks for the link to sigstore!

I am just wondering if distributed CI is possible, just like distributed crypto currency mining/defi is.

We have distributed source control, so it would be great to have distributed CI based on “zero trust computing”.

verdverm
Check out Dagger, their shared (or distributed if you squint) cache is sort of what you are getting at. It works based on containers (with hashes) and their inputs (also hashes of filesystems & config), and will only run steps that are not already computed. To get the "distributed" aspect, connect builders (local & cloud) to the same engine.

Most CI is already "distributed" in the sense that we typically have sub-jobs and multiple workers. We already use git, which is the ledger. Adding the permissionless layer would be too expensive and a distraction from getting real work done.

On the trust side, people don't want to rely on computers they don't control for their software supply chain. We already have enough to worry about with external dependencies we pull in.