Show HN: Nix Snapshotter – Native understanding of Nix packages for containerd (github.com)

Hi flurie, Nixery exposes an API (in the form of an Docker registry) to dynamically build Nix-based images, but still tar & compresses Nix packages into layer tarballs. Since the image spec has a limit of 128 layers (due to overlayfs), a heuristic is used to put popular packages together. However, in practice there is still a large amount of duplication between images that share the same packages due to this heuristic based strategy. You also may deploy the same packages outside of containers but have to duplicate that same data (in a slightly different format) on a Docker Registry.

With `pkgs.nix-snapshotter.buildImage`, containerd natively understands Nix packages, so everything is pulled at package granularity without the layer limit. This means all container content is either already in your host nix store or fetched from your Nix binary cache. Think delta image pulls, which isn't possible with regular images since one bit difference will change the layer hash and thus duplicate.

That said, nothing stops Nixery from building nix-snapshotter images. Then we'll have an Docker Registry that dynamically builds native Nix images.

To answer that, I think you’d first need to define what “putting my Nix config in e.g. a Dockerfile” means.

Do you mean using a Dockerfile to run a Nix package build, embedding the resulting files (and transitive dependencies) in the image file? If so, this objectively superior in several ways: 1) only one copy of any package exists on disk: the one in the Nix store, instead of multiple images having independent copies. 2) Related to this de-duplication, it will be faster to go from pristine host (virtual machine, dev machine, whatever) to having containers up and running, as you no longer have bulky images to transfer over the network. 3) I haven’t looked at the finer details, but I’m pretty sure this means that the host can build the required packages once, meaning better caching — instead of multiple Dockerfiles rebuilding the same packages, the host builds once and all dependent containers get use the host’s Nix store, again without any copying of files involved.

Edit:

From hinshun’s comment above, point #3 is incorrect — either the host’s Nix store would need a copy already present, or the package would need to be available in a remote store/cache. Even with that correction, you still avoid redundant package builds — either the package is available remotely/locally, or the image fetch fails — so things are still only ever built once :)

It can leverage de-duplication in your host nix store (as a sibling service to your container runtime), as opposed to having a containerized nix store each fetching its necessary packages.

If you have a nix environment and want to just kubernetes for orchestration, this is exactly what you want!

That's fair, adding another service is definitely a downside. If you are running nix-built images though, the underlying data still needs to land somewhere on disk. Nix-snapshotter lets you store less data and download less data due to its package granularity.

We already have rootless containerd and nix-snapshotter. And I believe there is work underway to run rootless nix (and rootless k3s too). You may be able to run the whole thing unprivileged one day.

Even at the cost of the k8s nodes having to be "special", this is still fabulously useful running Nix-native workloads on stuff like GitLab or Jenkins workers.

Thank you! We appreciate it.

If you are comparing non-Nix image build performance, Nix is comparable to BuildKit in that it’s able to parallelize a build graph and cache intermediary outputs. Nix doesn’t require a Dockerfile to build, and constructs the layers directly like in the article.

In terms of image size, since it’s dependencies are explicit (all the way down to glibc and lower), the Nix image is comparable to a scratch image with a single statically compiled binary.

Comparing to the existing Nix built images, we are moving image build performance from O(n) (n is number of Nix packages) to ~O(1) because the build process becomes just constructing a JSON referring to already built Nix packages.

I don't think this is true, I think it only requires the host system to have nix installed.

We will be running it on a non-NixOS Kubernetes cluster, all you need is a nix daemon running as a systemd service.

I have tested with EKS, and you should be able to use nix-snapshotter with GKE too. May be able to put together some docs for that later.

In the meantime, see this for running a different snapshotter on EKS: https://blog.realvarez.com/using-estargz-to-reduce-container...

I'm not sure exactly how the communication of components works there, but in k8s can this be deployed using daemonset rather host service?

I am still sceptical. What is the difference between "native understanding of nix packages" versus DIY putting your project derivation into dockerTools.buildImage?

I dont see a killer selling point, just some middle ware with extra containerization features which also could be pulled into nixpkgs.

But i have to admit, in my perception bubble VC gets more desperate and to them, the search metrices for nix must be shining like a christmas tree. I am certainly biased negatively.

Yes! Our talk wasn’t accepted but Robbie and I, and a few others from my team will be there. Come say Hi!