tl;dr: Nix will soon be deciding whether to rebuild packages based on the dependency contents rather than the build inputs, which should result in much fewer rebuilds.
This is excellent news, and hopefully the savings will end up being used to run more aggressive tests and builds of lower priority packages, to have fewer breakages.
My biggest problem using Nix has been minor changes, like the address of upstream sources, rippling down through dependencies. Hopefully this will improve with content addressing, and make it easier to fetching from places like IPFS. As an example:
- I wrote a piece of software as part of a research project, which I kept in git and built using Nix.
- That tool was generally useful, so I didn't want to pollute its repo with my particular experiment scripts/results, etc. so I kept those in a separate repo which pinned a particular commit of that tool.
- The writeups/papers based on these experiments had a lot of churn (as usual), so I kept those in a third git repo, which pinned a particular commit of the experimental results. These writeups did some analysis, generated graphs, etc. which was also managed by Nix.
This all worked great, everything was reproducible, identified by unforgable git commit IDs, etc. Then Microsoft bought GitHub and a bunch of people/projects deleted their repos (e.g. moving to GitLab, or wherever). That broke the tool's build, making the whole chain from tool -> experiment -> paper unreproducible. It was easy enough to fix by swapping the URLs in the .nix files, but that changed the git commit ID; hence the existing experimental data was tied to a commit which didn't work.
Content-addressing would:
- Allow monkey-patching a fix, by letting me declare that both commits produce the same tool (despite their sources containing slightly different .nix files), and hence this change doesn't alter the experimental results.
- Allow such issues to be prevented, by making it easier to fetch dependencies from content-addressed storage like IPFS, which reduces the reliance on third-party servers keeping URLs alive; e.g. I could keep hosting the desired sources myself. This doesn't actually need content-addressed Nix builds, but that would make things easier (e.g. reducing the distinction between "source" and "build product").
For this specific example I'm confused why the output paths changed. Remote data fetches are already content addressed. That is why you need to provide the hash when you fetch them.
This is still a problem in general though as small unimportant changes can cause rebuilds.
Each repo defines its own Nix files; changing them (e.g. to swap github to gitlab) required new git commits; using those new git commits as our source changes the hash, since it's based on the inputs rather than the content. If it were content-based, the tool's hash wouldn't change (since the change to the .nix files in the new source commit wouldn't affect the contents of the resulting binaries).
The git hashes are stored in the .nix files, which are part of the repo contents. This gives us a Merkle tree that pins all of the versions of all of the dependencies. Changing these git hashes hence requires changing the content of the repos.
For example, we start with:
tool-repo commit T1 with default.nix:
myDependency = fetchGit {
url = "https://....github.com/myDependency...";
rev = "...123...";
}
experiment-repo commit E1 with default.nix:
tool = fetchGit {
url = "http://...tool-repo...";
rev = "...T1...";
}
experiment-repo also contains results in a file like data/machine1/T1.json
writeup-repo commit W1 with default.nix:
data = fetchGit {
url = "http://...experiment-repo...";
rev = "...E1...";
}
When myDependency disappears from github, commit T1 becomes unbuildable; and hence the experimental results data/machine1/T1.json can't be reproduced. To fix this we need to change to the new gitlab URL:
This needs to propagate to experiment-repo. We need to decide how to handle the existing data:
- We could put a note to users that we pinkie-promise that T1.json is reproducible by commit T2 (despite having no hashes/etc. to attest to this)
- We could rename T1.json to T2.json, which is dishonest since that's not where the data came from.
- We could re-run the experiment, which takes several days, doubles the amount of data stored in the repo and gives slight differences w.r.t. timing (since the experiments are mostly about the performance/scaling behaviour of the algorithms used by the tool).
Using commit T2 instead of T1 will give T1's `tool` derivation a different hash to T2's `tool` derivation. Even though the git checkouts are fixed-output, the contents of T1 and T2 are different (one contains default.nix with a github URL, the other contains default.nix with a gitlab URL). Since we're building from different sources, the resulting tool will get a different input-based hash (if we were using content-based hashes, that wouldn't change).
We then update the writeup-repo so that anyone trying to recreate our paper's results will not find the broken commit:
writeup-repo commit W2 with default.nix:
data = fetchGit {
url = "http://...experiment-repo...";
rev = "...E2...";
}
This will also change the input-based hash of the paper, although that's not too bad since rendering the paper (+ graphs, etc.) is very quick compared to the days required to run the experiments.
Nix has been at the forefront of the reproducible build movement from the start. Large parts of nixpkgs build reproducible, due to the insane amount of isolation the builds are run in. Nevertheless, some build systems and compilers are still nondeterministic and therefore no package management system can make software builds reproducible with those tools.
Seems like they only should depend on the parts of dependency that actually get incorporated in the dependent.
For instance:
- Changes to C source files of a shared library should not require a rebuild of dependents (assuming you don't LTO with shared library bitcode)
- Changes to an executable in a dependency that you run as a child process during usage but not at build time should not require a rebuild of dependents
> Changes to C source files of a shared library should not require a rebuild of dependents
Yes it should, as dependents may be looking for the existence of a given function being exported from the library at build time (through dlopen/dlsym) and enable / disable features accordingly
This issue can be solved by extracting the "interface" and using that as the input. So for a shared library the interface may be all exported symbols. If that changes in any way you should rebuild the dependencies (to be cautious).
I think the ideal way to do this is to make a "dummy" library with the same interface and expose that to the build of the other program. That way you know it doesn't depend on anything else.
Of course there are complexities:
- What if the library is called in the build. Maybe the shim can proxy back to the original? But now you risk it depending on something. For tests proxying may make sense for build maybe not.
- For Nix you still want to produce a new output that is hardcoded to the new library version. You can skip the compile, but you still need to update the references.
> - Changes to C source files of a shared library should not require a rebuild of dependents (assuming you don't LTO with shared library bitcode)
Not so. Nix does dynamic linking right, too, which means you dynamically link to a unique content (or build recipe) addressable name. Concrete example:
15 comments
[ 3.5 ms ] story [ 46.8 ms ] threadThis is excellent news, and hopefully the savings will end up being used to run more aggressive tests and builds of lower priority packages, to have fewer breakages.
- I wrote a piece of software as part of a research project, which I kept in git and built using Nix.
- That tool was generally useful, so I didn't want to pollute its repo with my particular experiment scripts/results, etc. so I kept those in a separate repo which pinned a particular commit of that tool.
- The writeups/papers based on these experiments had a lot of churn (as usual), so I kept those in a third git repo, which pinned a particular commit of the experimental results. These writeups did some analysis, generated graphs, etc. which was also managed by Nix.
This all worked great, everything was reproducible, identified by unforgable git commit IDs, etc. Then Microsoft bought GitHub and a bunch of people/projects deleted their repos (e.g. moving to GitLab, or wherever). That broke the tool's build, making the whole chain from tool -> experiment -> paper unreproducible. It was easy enough to fix by swapping the URLs in the .nix files, but that changed the git commit ID; hence the existing experimental data was tied to a commit which didn't work.
Content-addressing would:
- Allow monkey-patching a fix, by letting me declare that both commits produce the same tool (despite their sources containing slightly different .nix files), and hence this change doesn't alter the experimental results.
- Allow such issues to be prevented, by making it easier to fetch dependencies from content-addressed storage like IPFS, which reduces the reliance on third-party servers keeping URLs alive; e.g. I could keep hosting the desired sources myself. This doesn't actually need content-addressed Nix builds, but that would make things easier (e.g. reducing the distinction between "source" and "build product").
This is still a problem in general though as small unimportant changes can cause rebuilds.
For example, we start with:
When myDependency disappears from github, commit T1 becomes unbuildable; and hence the experimental results data/machine1/T1.json can't be reproduced. To fix this we need to change to the new gitlab URL: This needs to propagate to experiment-repo. We need to decide how to handle the existing data:- We could put a note to users that we pinkie-promise that T1.json is reproducible by commit T2 (despite having no hashes/etc. to attest to this)
- We could rename T1.json to T2.json, which is dishonest since that's not where the data came from.
- We could re-run the experiment, which takes several days, doubles the amount of data stored in the repo and gives slight differences w.r.t. timing (since the experiments are mostly about the performance/scaling behaviour of the algorithms used by the tool).
In any case, we then update the .nix files:
Using commit T2 instead of T1 will give T1's `tool` derivation a different hash to T2's `tool` derivation. Even though the git checkouts are fixed-output, the contents of T1 and T2 are different (one contains default.nix with a github URL, the other contains default.nix with a gitlab URL). Since we're building from different sources, the resulting tool will get a different input-based hash (if we were using content-based hashes, that wouldn't change).We then update the writeup-repo so that anyone trying to recreate our paper's results will not find the broken commit:
This will also change the input-based hash of the paper, although that's not too bad since rendering the paper (+ graphs, etc.) is very quick compared to the days required to run the experiments.For instance:
- Changes to C source files of a shared library should not require a rebuild of dependents (assuming you don't LTO with shared library bitcode)
- Changes to an executable in a dependency that you run as a child process during usage but not at build time should not require a rebuild of dependents
Yes it should, as dependents may be looking for the existence of a given function being exported from the library at build time (through dlopen/dlsym) and enable / disable features accordingly
I think the ideal way to do this is to make a "dummy" library with the same interface and expose that to the build of the other program. That way you know it doesn't depend on anything else.
Of course there are complexities:
- What if the library is called in the build. Maybe the shim can proxy back to the original? But now you risk it depending on something. For tests proxying may make sense for build maybe not.
- For Nix you still want to produce a new output that is hardcoded to the new library version. You can skip the compile, but you still need to update the references.
Not so. Nix does dynamic linking right, too, which means you dynamically link to a unique content (or build recipe) addressable name. Concrete example:
The 90illc73xfs933d06daq6d41njs8yh66 bit identifies an unique version of glibc.This way it's actually reproducible and you don't have DLL-hell clashes.
[1] https://www.tweag.io/blog/2018-03-15-bazel-nix/