Funny that the article only obliquely references the compression issues. The OCI users that I have seen are using gzip due to inertia, while zstd layers have been supported for a while and are radically faster.
Uhhh what? Isn’t the whole point of Bazel that it’s a monorepo with all dependencies so you don’t need effing docker just to build or run a bloody computer program?
It drives me absolute batshit insane that modern systems are incapable of either building or running computer programs without docker. Everyone should profoundly embarrassed and ashamed by this.
I’m a charlatan VR and gamedev that primarily uses Windows. But my deeply unpopular opinion is that windows is a significantly better dev environment and runtime environment because it doesn’t require all this Docker garbage. I swear that building and running programs does not actually have to be that complicated!! Linux userspace got pretty much everything related to dependencies and packages very very very wrong.
I am greatly pleased and amused that the most reliable API for gaming in Linux is Win32 via Proton. That should be a clear signal that Linux userspace has gone off the rails.
I'm struggling with the caching right now. I'm trying to switch from the Github actions to just running stuff in containers, and it works. Except for caching.
Buildkit from Docker is just a pure bullshit design. Instead of the elegant layer-based system, there's now two daemons that fling around TAR files. And for no real reason that I can discern. But the worst thing is that the caching is just plain broken.
I went down this rabbit hole before, you have to ignore all the recommended approaches. The real solution is to have a build server with a global Docker install and a script to prune cache when the disk usage goes above a certain percentage. Cache is local and instant. Pushing and pulling cache images is an insane solution.
Buildkit can be very efficient at caching, but you need to design your image build around it. Once any step encounters a cache miss, all remaining steps will too.
I'd also avoid loading the result back into the docker daemon unless you really need it there. Buildkit can output directly to a registry, or an OCI Layout, each of which will maintain the image digest and support multi-platform images (admittedly, those problems go away with the containerd storage changes happening, but it's still an additional export/import that can be skipped).
All that said, I think caching is often the wrong goal. Personally, I want reproducible builds, and those should bypass any cache to verify each step always has the same output. Also, when saving the cache, every build caches every step, even if they aren't used in future builds. As a result, for my own projects, the net result of adding a cache could be slower builds.
Instead of catching the image build steps, I think where we should be spending a lot more effort is in creating local proxies of upstream dependencies, removing the network overhead of pulling dependencies on every build. Compute intensive build steps would still be slow, but a significant number of image builds could be sped up with a proxy at the CI server level without tuning builds individually.
My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top.
I'm not too surprised that out of the box docker images exhibit more of this. While it's good they're fixing it, it feels like maybe some of the core concepts cause pretty systematic issues anytime you try to do anything beyond the basic feature set...
Seconded. I tried hard to use Bazel in a polyglot repo because I really wanted just one builder.
Unfortunately, the amount of work you need to just maintain the build across language and bazel version upgrades is incredibly high. Let alone adding new build steps, or going even slightly off the well-trodded path.
I feel like Bazel would need at least 5 more full-time engineers to eventually turn it into an actually usable build tool outside Big Tech. Right now many critical open source Bazel rules get a random PR every now and then from people who don't actually (have time to) care about the open source community.
My go-to now is to use mise + just to glue together build artifacts from every language's standard build tools. It's not great but at least I get to spend time on programming instead of fixing the build.
Is load not hermetic? Ideally you should be mirroring all layers you use as inputs to your OCI builds and pin SHA256 versions. Your caching will also have issues if you don’t pin versions. Push should also be idempotent, but not hermetic.
> The current recommendation is rules_oci, which takes the opposite approach: use only off‑the‑shelf tools...
I'm the author of one of those off the shelf tools, and the rules_oci decision here always struck me as a bit unusual. OCI is a relatively easy spec with a number of libraries that implement it. Instead of creating a custom build command that leveraged those libraries to be an efficient build tool, they found commands that could be leveraged even if image building wasn't their design.
It looks like rules_img is taking that other path with their own build command based on the go-containerregistry library. I wish them all the best with their effort.
That said, if all you need to do is add a layer to an existing base, there are tools like crane [0] and regctl [1] that do that today.
The reason other build tools typically pull the base image first is to support "RUN" build steps that execute a command inside of a container and store the filesystem changes in a new layer. If that functionality is ever added to rules_img, I expect it to have the same performance as other build tools.
The underlying problem is that most container images are not cache efficient. Compressed tarballs arent and that’s what most of container images are. And Bazel relies heavily on caching to stay fast.
Most of the hyper scaler actually do not store container images as tarballs at scale. They usually flatten the layers and either cache the entire file system merkle tree, or breaking it down to even smaller blocks to cache them efficiently. See Alibaba Firefly Nydus, AWS Firecracker, etc… There is also various different forms of snapshotters that can lazily materialize the layers like estargz, soci, nix, etc… but none of them are widely adopted.
I think this is really close to the way nix2container works (https://github.com/nlewo/nix2container). nix2container generates metadata at build time and streams the required data at runtime.
At build time, it generates a JSON file describing the image metadata and the layers data location.
At runtime, it consumes this JSON file to stream layer data and image configuration to a destination. This is implemented by adding a new transport to Skopeo. Thanks to this implementation, nix2container doesn't need to handle all various destrination since this is managed by Skopeo itself.
15 comments
[ 3.1 ms ] story [ 38.2 ms ] threadIt drives me absolute batshit insane that modern systems are incapable of either building or running computer programs without docker. Everyone should profoundly embarrassed and ashamed by this.
I’m a charlatan VR and gamedev that primarily uses Windows. But my deeply unpopular opinion is that windows is a significantly better dev environment and runtime environment because it doesn’t require all this Docker garbage. I swear that building and running programs does not actually have to be that complicated!! Linux userspace got pretty much everything related to dependencies and packages very very very wrong.
I am greatly pleased and amused that the most reliable API for gaming in Linux is Win32 via Proton. That should be a clear signal that Linux userspace has gone off the rails.
Buildkit from Docker is just a pure bullshit design. Instead of the elegant layer-based system, there's now two daemons that fling around TAR files. And for no real reason that I can discern. But the worst thing is that the caching is just plain broken.
I'd also avoid loading the result back into the docker daemon unless you really need it there. Buildkit can output directly to a registry, or an OCI Layout, each of which will maintain the image digest and support multi-platform images (admittedly, those problems go away with the containerd storage changes happening, but it's still an additional export/import that can be skipped).
All that said, I think caching is often the wrong goal. Personally, I want reproducible builds, and those should bypass any cache to verify each step always has the same output. Also, when saving the cache, every build caches every step, even if they aren't used in future builds. As a result, for my own projects, the net result of adding a cache could be slower builds.
Instead of catching the image build steps, I think where we should be spending a lot more effort is in creating local proxies of upstream dependencies, removing the network overhead of pulling dependencies on every build. Compute intensive build steps would still be slow, but a significant number of image builds could be sped up with a proxy at the CI server level without tuning builds individually.
I'm not too surprised that out of the box docker images exhibit more of this. While it's good they're fixing it, it feels like maybe some of the core concepts cause pretty systematic issues anytime you try to do anything beyond the basic feature set...
Genuine question - also find Bazel frustrating at times.
Unfortunately, the amount of work you need to just maintain the build across language and bazel version upgrades is incredibly high. Let alone adding new build steps, or going even slightly off the well-trodded path.
I feel like Bazel would need at least 5 more full-time engineers to eventually turn it into an actually usable build tool outside Big Tech. Right now many critical open source Bazel rules get a random PR every now and then from people who don't actually (have time to) care about the open source community.
My go-to now is to use mise + just to glue together build artifacts from every language's standard build tools. It's not great but at least I get to spend time on programming instead of fixing the build.
Container layers are so large that moving them around is heavy.
So defer that part for the non-hermetic push/load parts of the process, while retaining heremticity/reproducibility.
You can sort of think of it like the IO monad in Haskell…defer it all until the impure end.
I'm the author of one of those off the shelf tools, and the rules_oci decision here always struck me as a bit unusual. OCI is a relatively easy spec with a number of libraries that implement it. Instead of creating a custom build command that leveraged those libraries to be an efficient build tool, they found commands that could be leveraged even if image building wasn't their design.
It looks like rules_img is taking that other path with their own build command based on the go-containerregistry library. I wish them all the best with their effort.
That said, if all you need to do is add a layer to an existing base, there are tools like crane [0] and regctl [1] that do that today.
The reason other build tools typically pull the base image first is to support "RUN" build steps that execute a command inside of a container and store the filesystem changes in a new layer. If that functionality is ever added to rules_img, I expect it to have the same performance as other build tools.
[0]: https://github.com/google/go-containerregistry/blob/main/cmd...
[1]: https://regclient.org/cli/regctl/image/mod/
Ok, wait, why?
Most of the hyper scaler actually do not store container images as tarballs at scale. They usually flatten the layers and either cache the entire file system merkle tree, or breaking it down to even smaller blocks to cache them efficiently. See Alibaba Firefly Nydus, AWS Firecracker, etc… There is also various different forms of snapshotters that can lazily materialize the layers like estargz, soci, nix, etc… but none of them are widely adopted.
At build time, it generates a JSON file describing the image metadata and the layers data location. At runtime, it consumes this JSON file to stream layer data and image configuration to a destination. This is implemented by adding a new transport to Skopeo. Thanks to this implementation, nix2container doesn't need to handle all various destrination since this is managed by Skopeo itself.
Recently, we introduced a mechanism to also produce such kind of JSON file for the base image (see https://github.com/nlewo/nix2container?tab=readme-ov-file#ni...).
I'm pretty sure the added (not usptreamed yet) transport could be useful in some other close contexts, such as Bazel or Guix.
I'm the nix2container author and i will be glad to discuss with you if you think this Skopeo transport could be useful for you!
(btw, your blog post is pretty well written!)