bootc and OSTree are both very neat, but the leading edge of immutable Linux distros (GNOME OS, KDE Linux) is currently converging on a different proposal by systemd developers that's standardized by the UAPI Group (https://uapi-group.org/specifications/). It fixes quite a few of the complexities with OSTree (updates are handled by `systemd-sysupdate`/`updatectl` and are just files served via HTTP) and is quite a bit easier to extend with things like an immutable version of the Nvidia drivers or codecs thanks to system extensions handled by `systemd-sysext` (which in turn are just simple squashfs files overlayed over `/usr`) and configuration via `systemd-confext`. `mkosi`, also by systemd, is quickly becoming _the_ way to build custom images too, and is somewhat tied to these new standards.
system extensions are just as usable on bootc based systems but bootc based systems allow for much simpler composition, you dont have to know about all the intricacies of linux filesystem tree to install a set of packges for your own image as you can base uppon what someone else did easily using industry standard semantics and use actual package managers. And on top of all that it uses a far better distribution system, the oci's distribution spec
It is very odd to me to watch OStree-based distros starting to take off and win recruits.
The only reason Red Hat needed to invent this very complex mechanism was because RH does not officially have a COW-snapshot capable filesystem in its enterprise distro.
A filesystem with snapshots makes software installation transactional. You take a snapshot, install some software, and if it doesn't work right, you can revert to the snapshot. (With very slightly more flexible snapshots, you can limit the snapshot to just some part of the directory tree, but this is not essential; it merely permits more flexibility.)
In other words, you are a long way toward what in database language is called ACID:
Atomicity, consistency, isolation, durability. It makes your software inastallation transactional: an update either happens completely (A), you can check it is valid (C) and works (I), or it can be totally reverted, and the system restored to the earlier state (D).
That's a good thing. It means you can safely automate software deployment knowing that if it goes wrong you have an Undo mechanism. Databases got this 50+ years ago; in the 21st century it's making its way to FOSS OSes.
Do this in the filesystem and it's easy. SUSE's implementation is so simple, it's basically a bunch of shell scripts, and it can be turned on and off. You can run an immutable OS, reboot for updates, and if you need, disable it, go in and fix the system, and then turn it back on again.
This is because SUSE leans very heavily on Btrfs and that is the critical weakness -- Btrfs is only half finished and is not robust.
But RH removed Btrfs from RHEL and Btrfs was the only GPL COW filesystem, so core infrastructure in the distro means no COW on RH. Oracle Linux has Btrfs -- the FS was developed at Oracle, after all -- and so does Alma.
(Yes I know, Fedora put it back, but the key thing is, it only uses Btrfs only for compression so that Flatpak looks less horrendously inefficient. Fedora doesn't use snapshots.)
With no COW FS, RH had to invent a way to do transactional updates without filesystem support. Result, OStree. Git, but for binaries.
And yes, everyone developing FOSS uses Git, but almost nobody understands Git:
You know that if there's an Xkcd about it, it must be true.
Embedding something you don't understand in your OS design is a VERY BAD PLAN.
With OStree your FS is a virtual one, it's not real, it's synthesized on the fly from a local repository. The real FS is hidden and can't be hand-edited or anything. It generates the OS filesystem tree on the fly, you see. OS-tree.
Use it just for GUI apps, that's Flatpak.
Use it for the whole OS, that's OStree. It is so mind-shreddingly complicated that you can't do package management any more, you can't touch the underlying FS. So you need a whole new set of layers on top: virtual directories on top of the main virtual directory, and some bits with extra pseudo-filesystems layered on top of that to make some bits read-write.
It's like the scene in the Wasp Factory where under the skull plate it's just writhing maggots. I recall in horror and revulsion when I see it.
So it's deeply bizarre to read blog posts praising all the cool stuff you can do with it.
Yawn. You again. Hater!1!! ;-> Mine is still running, after all these (two) years abusing it. Even with installing(not really, much more complicated) some Windoze into a subvolume of it by means of WinBtrfs. I can either boot that Windows natively, or from the same install into a VM, by means of what once were called bootmanager profiles. Automagically chosen. Without having to reserve space for NTFS, VFAT(excluding UEFI/boot), or anything else. Its just a fucking folder, like any other.
> But RH removed Btrfs from RHEL and Btrfs was the only GPL COW filesystem
There is NILFS(2) which is GPL. Lacks some features, but that could have been worked around, or implemented?
> I recall in horror and revulsion when I see it.
You have total recall of recoiling in horror? What a trip... ;-)
That aside, it's all cargo-cult anyway, because of this thing called FHS. Which seems crazy, because in a world of open-source, which can be compiled almost any way one could wish for, why not just use slashpkg, Gobolinux or similar stuff?
“Once is happenstance. Twice is coincidence. Three times is enemy action.”
I saw Btrfs die so many times, I know what to suspect.
The official internal guidance was: give it all your disk space. Make the volumes so big that they will never fill up.
Well, if I could trust it, I might, but I know I can't trust it, so I want my data safe somewhere else. And the company wouldn't provide me with 2 or 3 disk drives so `/home` could live on another physical volume.
And once you've watched your primary work OS self-destruct again, well, if you're me, you reinstall with dual-boot so you have something else to fall back on if it happens again. As it did happen again. Repeatedly.
> You have total recall of recoiling in horror? What a trip... ;-)
OK, my bad. Brain/finger error.
> why not just use slashpkg, Gobolinux or similar stuff?
You cannot replicate the full filesystem among multiple hosts with their own drives though, can you?
With OSTree my team can deploy a commit over an existing ones to a field of 60k embedded Linux devices. Similarly bootc is a great alternative for deploying images for dedicated or virtualized servers.
How do you tag / mark a commit a snapahot of all files in an OS image that's independent from each device's local state and then move them from one commit to another atomically? Basically a `git pull` on a branch but binary and atomic.
CoW is strictly local. You can make a snapshot of local state sure and even atomically change it. But that's not we want.
Apart from NixOS and GuixSD there's also Arkane Linux (<https://www.arkanelinux.org/>), which seems kind of interesting, but I don't like various decisions it makes.
Personally, I've really enjoyed using bootc for both my personal laptop and my NAS.
I really like the NAS use case because I can build the ZFS kmods for that specific version of Fedora CoreOS in CI/CD. If there's any compatibility failure, then my NAS doesn't get an update and I get the CI/CD failed email. No downtime because of some kernel incompatibility.
For the laptop though, I feel like there's a better way that I haven't found. Some way not to require CI/CD, to build the next image and switch to it all locally. I haven't gone down that path yet, but it looks kinda like that Option 2 the author described. Maybe it's really just that easy.
I like the idea behind ostree and bootc, but I feel that OCI (with tarlayers) is not a good fit. `repack` makes an absolute hash of things, and since the layers are logically packaged, they will have to be composed somehow, and then ostree becomes only slightly more useful than coreos's A/B usr.
OCI roughly assumes that layers will be laid out in some logical way, and that a given host will see opportunities to reuse across different instances, but with bootc, there will only ever be one instance.
OCI also assumes that individual layers are small enough that it is always worth pulling and unpacking a layer instead of some kind of authentication delta, which is great for a k8s cluster in a center, but not great for devices out on the edge, where you might want this kind of pseudo-immutable system even more.
I really want some standardized way for a manifest in OCI to say that "this content is also available in other format X here".
20 comments
[ 2.5 ms ] story [ 46.2 ms ] threadNext up, backups stored as layers in the same OCI registries.
I am not, however, sure ostree is going to be the final image format. Last time I looked work was in progress to replace that.
It works quite well for our edge devices. It’s tightly integrated with Toradex hardware, but not limited to it.
It may seems litte a niche, but it has strong potential for long‑term supported edge products. Any additional experiences to share?
The only reason Red Hat needed to invent this very complex mechanism was because RH does not officially have a COW-snapshot capable filesystem in its enterprise distro.
A filesystem with snapshots makes software installation transactional. You take a snapshot, install some software, and if it doesn't work right, you can revert to the snapshot. (With very slightly more flexible snapshots, you can limit the snapshot to just some part of the directory tree, but this is not essential; it merely permits more flexibility.)
In other words, you are a long way toward what in database language is called ACID:
https://en.wikipedia.org/wiki/ACID
Atomicity, consistency, isolation, durability. It makes your software inastallation transactional: an update either happens completely (A), you can check it is valid (C) and works (I), or it can be totally reverted, and the system restored to the earlier state (D).
That's a good thing. It means you can safely automate software deployment knowing that if it goes wrong you have an Undo mechanism. Databases got this 50+ years ago; in the 21st century it's making its way to FOSS OSes.
Do this in the filesystem and it's easy. SUSE's implementation is so simple, it's basically a bunch of shell scripts, and it can be turned on and off. You can run an immutable OS, reboot for updates, and if you need, disable it, go in and fix the system, and then turn it back on again.
This is because SUSE leans very heavily on Btrfs and that is the critical weakness -- Btrfs is only half finished and is not robust.
But RH removed Btrfs from RHEL and Btrfs was the only GPL COW filesystem, so core infrastructure in the distro means no COW on RH. Oracle Linux has Btrfs -- the FS was developed at Oracle, after all -- and so does Alma.
(Yes I know, Fedora put it back, but the key thing is, it only uses Btrfs only for compression so that Flatpak looks less horrendously inefficient. Fedora doesn't use snapshots.)
With no COW FS, RH had to invent a way to do transactional updates without filesystem support. Result, OStree. Git, but for binaries.
And yes, everyone developing FOSS uses Git, but almost nobody understands Git:
https://xkcd.com/1597/
You know that if there's an Xkcd about it, it must be true.
Embedding something you don't understand in your OS design is a VERY BAD PLAN.
With OStree your FS is a virtual one, it's not real, it's synthesized on the fly from a local repository. The real FS is hidden and can't be hand-edited or anything. It generates the OS filesystem tree on the fly, you see. OS-tree.
Use it just for GUI apps, that's Flatpak.
Use it for the whole OS, that's OStree. It is so mind-shreddingly complicated that you can't do package management any more, you can't touch the underlying FS. So you need a whole new set of layers on top: virtual directories on top of the main virtual directory, and some bits with extra pseudo-filesystems layered on top of that to make some bits read-write.
It's like the scene in the Wasp Factory where under the skull plate it's just writhing maggots. I recall in horror and revulsion when I see it.
So it's deeply bizarre to read blog posts praising all the cool stuff you can do with it.
Yawn. You again. Hater!1!! ;-> Mine is still running, after all these (two) years abusing it. Even with installing(not really, much more complicated) some Windoze into a subvolume of it by means of WinBtrfs. I can either boot that Windows natively, or from the same install into a VM, by means of what once were called bootmanager profiles. Automagically chosen. Without having to reserve space for NTFS, VFAT(excluding UEFI/boot), or anything else. Its just a fucking folder, like any other.
> But RH removed Btrfs from RHEL and Btrfs was the only GPL COW filesystem
There is NILFS(2) which is GPL. Lacks some features, but that could have been worked around, or implemented?
> I recall in horror and revulsion when I see it.
You have total recall of recoiling in horror? What a trip... ;-)
That aside, it's all cargo-cult anyway, because of this thing called FHS. Which seems crazy, because in a world of open-source, which can be compiled almost any way one could wish for, why not just use slashpkg, Gobolinux or similar stuff?
Yep, me again.
You know that line from the James Bond books?
“Once is happenstance. Twice is coincidence. Three times is enemy action.”
I saw Btrfs die so many times, I know what to suspect.
The official internal guidance was: give it all your disk space. Make the volumes so big that they will never fill up.
Well, if I could trust it, I might, but I know I can't trust it, so I want my data safe somewhere else. And the company wouldn't provide me with 2 or 3 disk drives so `/home` could live on another physical volume.
And once you've watched your primary work OS self-destruct again, well, if you're me, you reinstall with dual-boot so you have something else to fall back on if it happens again. As it did happen again. Repeatedly.
> You have total recall of recoiling in horror? What a trip... ;-)
OK, my bad. Brain/finger error.
> why not just use slashpkg, Gobolinux or similar stuff?
Slashpkg means DJB's thing?
https://cr.yp.to/slashpackage.html
That sounds fantastic. Thanks. I had never heard of this before.
Re Gobo -- yes please. I love that little distro.
With OSTree my team can deploy a commit over an existing ones to a field of 60k embedded Linux devices. Similarly bootc is a great alternative for deploying images for dedicated or virtualized servers.
CoW is strictly local. You can make a snapshot of local state sure and even atomically change it. But that's not we want.
So same OS, users, packages, flatpaks etc. And a mostly synced home dir too.
Is NixOS the only viable way to do this? I don't like the path mangling that Nix introduces.
It seems like an immutable distro customized via a Containerfile could work too? Except rebooting/reimagine for every change sounds tedious as hell.
I really like the NAS use case because I can build the ZFS kmods for that specific version of Fedora CoreOS in CI/CD. If there's any compatibility failure, then my NAS doesn't get an update and I get the CI/CD failed email. No downtime because of some kernel incompatibility.
For the laptop though, I feel like there's a better way that I haven't found. Some way not to require CI/CD, to build the next image and switch to it all locally. I haven't gone down that path yet, but it looks kinda like that Option 2 the author described. Maybe it's really just that easy.
I've really been enjoying this space.
OCI roughly assumes that layers will be laid out in some logical way, and that a given host will see opportunities to reuse across different instances, but with bootc, there will only ever be one instance.
OCI also assumes that individual layers are small enough that it is always worth pulling and unpacking a layer instead of some kind of authentication delta, which is great for a k8s cluster in a center, but not great for devices out on the edge, where you might want this kind of pseudo-immutable system even more.
I really want some standardized way for a manifest in OCI to say that "this content is also available in other format X here".
rpm-ostree worked fine for me