but for desktop use, it's probably fine. It's been the default for new Fedora installations for over a year now, and I don't see too many Fedora users complaining about their filesystem being hosed.
My current installation is still legacy EXT4 since I upgrade, but at some point I might blow it away and try a fresh install.
BTRFS might be gaining feature and reliability parity with ZFS, but I have never heard of any advantage of BTRFS over ZFS and in all the benchmarks that I have ever seen BTRFS is much slower than ZFS.
When the file system performance is important, XFS and EXT4 are the fastest. When the extra features provided by ZFS are important, it becomes the appropriate choice. For BTRFS, I am not aware of any application for which it would be the right choice.
I understand the first one may be a genuine pptimization needed.
But for the second one: when a file is striped across different physical disks with possibly different characteristics, how would you defrag it without knowing the read access pattern?
> when a file is striped across different physical disks with possibly different characteristics, how would you defrag it without knowing the read access pattern?
Looking at a single disk at a time, do your best to make sure all chunks of the same file are in order and adjacent.
You could be smarter than that, but just that would do fine.
The problem is that ZFS basically can't move things.
> The problem is that ZFS basically can't move things.
The one killer feature for me with btrfs is offline deduping (and by extension not having de-duping require a super computer to run). My data has enough duplication that it'd be nice to have, but not worth requiring hundreds of gigs of ram. Being able to fire off a job after importing a large set of data (or maybe yearly-ish) would be awesome and give me a few extra years without having to update my hardware.
That doesn't account for physical disk characteristics at all. You could have disks from different generations all be part of a pool, CMR vs SMR(!) (needs zone resets and large chunk writes), disks could degrade etc. Plus, when you update a file, you literally create new links and update the file out of place (true for any COW FS). So the read won't be sequential if you want to access two versions at the same time.
In that case you want to know if you optimize for hardware failure as opposed to for performance.
This could be fixed for specific usecases but not all block patterns in general. Just why I posted this.
If your system is all zfs, there shouldn't be much in the Linux cache, except for cache from NFS shares... At worst, if your uptime is over month, you could just run sync and echo 1 > /proc/sys/vm/drop_caches (but why, just reboot already, you probably have kernel upgrades anyways)
There's another issue where Linux pagecache doesn't support filesystem blocks larger than single hardware pages (and no large pages allowed), so ZFS would be hard limited to 4kB blocks on x86 etc.
Also a lot of other fancy features would be broken by pagecache design, so it's a no-go, and instead project focused on reducing duplication between the two.
This was fixed recently in Linux by Matthew Wilcox and others, in a way that can benefit all Linux filesystems. Folios of pages were added in 5.16, the page cache was converted to use folios in 5.17, and large-folios were in 5.18.
I don't know ZFS, I have been using Btrfs for years (desktop, simple setup). Defragmentation in Btrfs is not without issues. If you use subvolume (and I use Btrfs for subvolumes) and in particular snapshots, defragmenting a subvolume will deduplicate its files, increasing disk usage.
On the other hand, I have not had much need for defragmenting.
Any application on linux where snapshots, resizable partitions, compression or deduplication is useful? Zfs on linux is still not ready for primetime on root partitions, and for most uses except raid5/6 (which isn't important for 99% of desktop users) btrfs is an excellent option
There's a reason the other major non-debian distros have switched to it by default.
I'm still waiting for ubuntu to allow installing using btrfs + fulldisk encryption by default, which is currently a massive issue.
> Zfs on linux is still not ready for primetime on root partitions
Not sure where you got this from. OpenZFS on Linux is considered stable by the project. It has been rock solid, free of major issues, and considered more than ready for prime time since at least 0.8, and probably much earlier. It’s now at 2.1.
I’ve been using it across multiple environments, including my personal laptop and workstation which maintained the same root-on-ZFS install for 6+ years, including 2 full migrations to new hardware using zfs send. I’ve deployed much larger pools in professional settings and they’ve been rock solid.
Btrfs has had numerous data loss issues in that time. I can only recall one data loss issue in a ZFS on Linux release, several years ago, which was caught very quickly.
Which major distro allows using ZFS from its official installer except ubuntu (now deprecated) and maybe arch linux? Which distro uses ZFS by default? I think almost none.
Sure, ZFS might be stable for extra mass storage disks, but how many people use it as a filesystem for the linux root? Never seen anyone putting the main OS on zfs
Btrfs hasn't had any major data loss issues for the last 5 years (excluding the raid issues)
btrfs is much newer than ZFS, so obviously it took longer to be stable, if you're talking about 6+ years ago, Btrfs was obviously not ready yet, but now? It's been the default on OpenSUSE for 5 years and Fedora for 2.
> Which major distro allows using ZFS from its official installer…?
That says more about the distributions than about ZFS’s fitness or maturity as a file system.
It’s understandable that many distributions don’t bundle ZFS, since the licensing is still an open debate. Although that’s not so cut-and-dried [1], I wouldn’t expect broad support from distributions unless and until these issues have been tested and proved out one way or the other. No organization wants to insert themselves into a legal skirmish.
The best (and probably only good thing) to come out of Ubuntu’s support for ZFS is the vote of confidence in favor of ZFS’s legality in-base from a major corporate entity. We can hope more distributions pick up on this, but I’m not holding my breath.
But legal issues and technicalities say nothing at all about the relative technical merit of ZFS (or any other file system).
> how many people use it as a filesystem for the linux root? Never seen anyone putting the main OS on zfs
Quite a lot of users, actually. There’s a significant community around ZFS, including an ecosystem of some robust tooling. ZFSBootMenu is evidence enough that people want to run root on ZFS, and that root on ZFS is both feasible and highly desirable. Compare its feature list to what you can achieve with root-on-Btrfs, and ZFS pretty handily comes out ahead as the more powerful file system in terms of real world administrative capabilities.
Not arguing that ZFS is not more feature-complete than btrfs, I agree in general ZFS is a better file-system than btrfs, but for general users BTRFS is much more accessible than ZFS on linux, since you can just install a distro with it as default, without having to be an expert in partitioning and managing a linux system. ZFSBootMenu is very much a power-user tool, not for regular users.
It's also very niche, <500 stars ? Wouldn't call that 'Quite a lot of users'.
Although I'd still suggest single-block-device-btrfs on mdadm instead. The per-file RAID in btrfs (/zfs) can skip over unallocated sectors, but mdadm gets to do larger sequential writes with less random seeking. On spinning disks this can very quickly overtake the performance at even lightly filled volumes.
Depends on your use case, but for me, absolutely not. The btrfs manual says RAID5 “should not be used in production, only for evaluation or testing. The power failure safety for metadata with RAID56 is not 100%.”
The baked in ZFS support is what lead me to use Ubuntu on my Desktop for a few release cycles. Sure you can build the module yourself using most other distros but it was the path of least effort. Eventually, I realized that I don't really use ZFS' features much on my Desktop. It's nice for sure, but I just didn't find myself needing it as much as I do on my servers (where it's essential.) So I've left the Ubuntu camp anyway.
For me it means, that kernel, zfs, bootloader and initramfs-creating scripts are tested as a whole, by the distribution. For the user it means, that he can fearlessly update any of these packages and upon reboot, the system will successfully start.
That is not given if you have one-off custom integration of these parts. For a while, I had a CentOS box with openzfs kABI modules; having to tinker after updates to mount the pool was not fun.
Void Linux with ZFSBootMenu isn't far off. Multiple core Void team members uses and maintain ZFS support; kernel version bumps are blocked on OpenZFS support. ZFSBootMenu handles the bootloader, which can support fully upgraded ZFS pools and doesn't need pared-down zpool properties like Grub.
ZFSBootMenu is developed natively on Void, and their wiki has bootstrap instructions for a Void install with root on ZFS. Everything's well supported, including native zpool encryption. Plus gives you a multitude of tools right in the bootloader -- snapshot rollback/management, boot environment creation/management, booting older kernels, and a full recovery command line with zfs available. You can tweak the bundled dracut config to include other tools as well.
Even supports generating a unified, SecureBoot-signed EFI binary.
The experience is leagues ahead of the Ubuntu mess.
> For the user it means, that he can fearlessly update any of these packages and upon reboot, the system will successfully start.
NixOS currently has this! Since upgrades are atomic, if you have some other version constraint that pushes you towards a kernel version that ZoL doesn't target or support*, your system just doesn't finish building the upgrade and never switches over. If the system upgrades, that means the kernel was built and ZFS was added to your initial ramdisk successfully.
In case that succeeds but there are still ZFS issues (which I've never seen or heard of), NixOS will still let you boot the old, working system until you figure out what messed you up in the upgrade. Just select it in GRUB or whatever you're using.
Maybe NixOS overall is more DIY than what you're looking for, but the ZFS support feels very first class! I personally recommend giving it a try, if you're up for giving a declarative distro a chance.
--
*: I've never heard of this happening except to NVIDIA users, since NVIDIA's proprietary kernel module imposes restrictions on how new of a kernel you can use. On AMD, I've never had a ZFS-related update issue.
BTRFS is in-tree so I'm guessing that it is much easier to keep it in sync. Besides that I don't think many distros want to risk the legal trouble of shipping ZoL.
The only legal trouble comes from GPL side (with SFC leading the lawsuit), but it also goes against long-term consensus on how GPL applies (as ZFS is not derived work)
Perhaps everyone who cared about Zsys/ZFS had already left Ubuntu due to other things, like Canonical's insistence on using the Snap (mis-)feature.
Disappointing, because Zsys is quite good. I believe it would have gained prominence among competing ZFS boot environment managers if it hadn't been so tied to Ubuntu.
Ubuntu was the first Linux distro that I could actually boot, back in 2007. Steam and ZFS were reasons for me to continue with it in 2020, but now, everything I want to do can be done better on NixOS.
This is a good thing for ZFS and public perception of ZFS. Ubuntu has made some very poor choices with how they've integrated ZFS -
* zsys has both implementation and design flaws. They utilize two zpools - one with limited feature flags that GRUB can use and one general pool, and then try to tightly couple datasets between those two pools while coordinating snapshots. zsys routinely runs a system out of space due to not cleaning up snapshots. It's been known to destroy user data as well.
* Their packaging was and is still very sloppy. It's quite common to catch people running one version of kernel modules and another version of userland tools - often multiple releases apart. This has led to some serious problems.
(no direct source for this, typically noted in libera.chat#zfsonlinux)
* Ubuntu has created and applied bad patches to ZFS, resulting in destroyed pools.
If your exposure to ZFS is through Ubuntu and zsys, you'd naturally have a very poor opinion of it. ZFS, of course, isn't with out flaws. Ubuntu is not helping.
As a long time ZFS user, I was excited when Canonical promised first class ZFS integration. But it was clear from the first release that Ubuntu's ZFS project was a mess. Clearly some folks got excited about ZFS, but didn't absorb much of the existing knowledge or wisdom in the ecosystem. Issues like dataset/pool layout, how snapshots were handled on update, were handled in completely unusual and boneheaded ways.
The ecosystem already had the concept of boot environments, which more or less standardized what a rool-on-zpool layout ought to look like, and how to handle graceful, recoverable system upgrades. What Ubuntu could have brought to the table was a really high quality beadm replacement for Linux, moving Grub's ZFS support forward with support for more feature flags and boot environments, or investing in ZFSBootMenu.
Instead they went ahead with zsys, and it's still not entirely clear what they want or expect it to be. That they needed to write a daemon to enable "complex zfs dataset layouts" should've been an early clue that they were going down the wrong path. The dataset layout for a root filesystem should never be so complex, that you need to write a whole daemon to manage it.
The packaging issues and bad patches are almost besides the point.
I don't think they really need the two pool model, booting off efi(fat32) can load the modules, and importing the pool can be handled by the initrd. At least that's how I've been doing it on gentoo for years.
Edit: handcuffing yourself to grub seems pointless. Not to mention, grub is controlled by oracle devs who want nothing to do with zfs support and have actively ignored bringing the grub zfs in line with openzfs mainline.
This sounds cool, but it would take a fair amount of changes for me to use it, since I bake the cmdline into the kernel and boot the kernel directly from the efistub. Maybe I'll check it out some day though.
48 comments
[ 2.9 ms ] story [ 93.7 ms ] threadAll RAID1, backups to object storage.
https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...
but for desktop use, it's probably fine. It's been the default for new Fedora installations for over a year now, and I don't see too many Fedora users complaining about their filesystem being hosed.
My current installation is still legacy EXT4 since I upgrade, but at some point I might blow it away and try a fresh install.
When the file system performance is important, XFS and EXT4 are the fastest. When the extra features provided by ZFS are important, it becomes the appropriate choice. For BTRFS, I am not aware of any application for which it would be the right choice.
But for the second one: when a file is striped across different physical disks with possibly different characteristics, how would you defrag it without knowing the read access pattern?
Looking at a single disk at a time, do your best to make sure all chunks of the same file are in order and adjacent.
You could be smarter than that, but just that would do fine.
The problem is that ZFS basically can't move things.
The one killer feature for me with btrfs is offline deduping (and by extension not having de-duping require a super computer to run). My data has enough duplication that it'd be nice to have, but not worth requiring hundreds of gigs of ram. Being able to fire off a job after importing a large set of data (or maybe yearly-ish) would be awesome and give me a few extra years without having to update my hardware.
In that case you want to know if you optimize for hardware failure as opposed to for performance.
This could be fixed for specific usecases but not all block patterns in general. Just why I posted this.
Also a lot of other fancy features would be broken by pagecache design, so it's a no-go, and instead project focused on reducing duplication between the two.
On the other hand, I have not had much need for defragmenting.
Only in very limited instances. I think ZFS duplicates the inodes and dentries cache, and only file pages when you do something like mmap a file.
The ability to add capacity to a pool by adding a single drive at a time, or the fact that it is readily available on any Linux box.
zfs supports that now
There's a reason the other major non-debian distros have switched to it by default.
I'm still waiting for ubuntu to allow installing using btrfs + fulldisk encryption by default, which is currently a massive issue.
Not sure where you got this from. OpenZFS on Linux is considered stable by the project. It has been rock solid, free of major issues, and considered more than ready for prime time since at least 0.8, and probably much earlier. It’s now at 2.1.
I’ve been using it across multiple environments, including my personal laptop and workstation which maintained the same root-on-ZFS install for 6+ years, including 2 full migrations to new hardware using zfs send. I’ve deployed much larger pools in professional settings and they’ve been rock solid.
Btrfs has had numerous data loss issues in that time. I can only recall one data loss issue in a ZFS on Linux release, several years ago, which was caught very quickly.
Btrfs hasn't had any major data loss issues for the last 5 years (excluding the raid issues) btrfs is much newer than ZFS, so obviously it took longer to be stable, if you're talking about 6+ years ago, Btrfs was obviously not ready yet, but now? It's been the default on OpenSUSE for 5 years and Fedora for 2.
“I can’t believe Nixon won. I don’t know anyone who voted for him.” -- Pauline Kael
I use ZFS on root. Know lots of people who do. Might be your bubble?
That says more about the distributions than about ZFS’s fitness or maturity as a file system.
It’s understandable that many distributions don’t bundle ZFS, since the licensing is still an open debate. Although that’s not so cut-and-dried [1], I wouldn’t expect broad support from distributions unless and until these issues have been tested and proved out one way or the other. No organization wants to insert themselves into a legal skirmish.
The best (and probably only good thing) to come out of Ubuntu’s support for ZFS is the vote of confidence in favor of ZFS’s legality in-base from a major corporate entity. We can hope more distributions pick up on this, but I’m not holding my breath.
But legal issues and technicalities say nothing at all about the relative technical merit of ZFS (or any other file system).
> how many people use it as a filesystem for the linux root? Never seen anyone putting the main OS on zfs
Quite a lot of users, actually. There’s a significant community around ZFS, including an ecosystem of some robust tooling. ZFSBootMenu is evidence enough that people want to run root on ZFS, and that root on ZFS is both feasible and highly desirable. Compare its feature list to what you can achieve with root-on-Btrfs, and ZFS pretty handily comes out ahead as the more powerful file system in terms of real world administrative capabilities.
[1] https://blog.hansenpartnership.com/are-gplv2-and-cddl-incomp...
Although I'd still suggest single-block-device-btrfs on mdadm instead. The per-file RAID in btrfs (/zfs) can skip over unallocated sectors, but mdadm gets to do larger sequential writes with less random seeking. On spinning disks this can very quickly overtake the performance at even lightly filled volumes.
Maybe its just to much work?
But then other distros have implemented many of the features with BTRFS so it can't be that much effort.
What would this look like? In what ways do existing distros fail to live up to that vision?
That is not given if you have one-off custom integration of these parts. For a while, I had a CentOS box with openzfs kABI modules; having to tinker after updates to mount the pool was not fun.
ZFSBootMenu is developed natively on Void, and their wiki has bootstrap instructions for a Void install with root on ZFS. Everything's well supported, including native zpool encryption. Plus gives you a multitude of tools right in the bootloader -- snapshot rollback/management, boot environment creation/management, booting older kernels, and a full recovery command line with zfs available. You can tweak the bundled dracut config to include other tools as well.
Even supports generating a unified, SecureBoot-signed EFI binary.
The experience is leagues ahead of the Ubuntu mess.
https://docs.zfsbootmenu.org/en/latest/index.html
https://docs.zfsbootmenu.org/en/latest/guides/debian.html
https://docs.zfsbootmenu.org/en/latest/guides/fedora.html
Void remains our primary development platform, but we're working on improving documentation and support for other distributions.
NixOS currently has this! Since upgrades are atomic, if you have some other version constraint that pushes you towards a kernel version that ZoL doesn't target or support*, your system just doesn't finish building the upgrade and never switches over. If the system upgrades, that means the kernel was built and ZFS was added to your initial ramdisk successfully.
In case that succeeds but there are still ZFS issues (which I've never seen or heard of), NixOS will still let you boot the old, working system until you figure out what messed you up in the upgrade. Just select it in GRUB or whatever you're using.
Maybe NixOS overall is more DIY than what you're looking for, but the ZFS support feels very first class! I personally recommend giving it a try, if you're up for giving a declarative distro a chance.
--
*: I've never heard of this happening except to NVIDIA users, since NVIDIA's proprietary kernel module imposes restrictions on how new of a kernel you can use. On AMD, I've never had a ZFS-related update issue.
It is both a technical and legal problem.
Disappointing, because Zsys is quite good. I believe it would have gained prominence among competing ZFS boot environment managers if it hadn't been so tied to Ubuntu.
Ubuntu was the first Linux distro that I could actually boot, back in 2007. Steam and ZFS were reasons for me to continue with it in 2020, but now, everything I want to do can be done better on NixOS.
* zsys has both implementation and design flaws. They utilize two zpools - one with limited feature flags that GRUB can use and one general pool, and then try to tightly couple datasets between those two pools while coordinating snapshots. zsys routinely runs a system out of space due to not cleaning up snapshots. It's been known to destroy user data as well.
https://github.com/ubuntu/zsys/issues/196
https://github.com/ubuntu/zsys/issues/218
* Their packaging was and is still very sloppy. It's quite common to catch people running one version of kernel modules and another version of userland tools - often multiple releases apart. This has led to some serious problems.
(no direct source for this, typically noted in libera.chat#zfsonlinux)
* Ubuntu has created and applied bad patches to ZFS, resulting in destroyed pools.
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/190...
If your exposure to ZFS is through Ubuntu and zsys, you'd naturally have a very poor opinion of it. ZFS, of course, isn't with out flaws. Ubuntu is not helping.
As a long time ZFS user, I was excited when Canonical promised first class ZFS integration. But it was clear from the first release that Ubuntu's ZFS project was a mess. Clearly some folks got excited about ZFS, but didn't absorb much of the existing knowledge or wisdom in the ecosystem. Issues like dataset/pool layout, how snapshots were handled on update, were handled in completely unusual and boneheaded ways.
The ecosystem already had the concept of boot environments, which more or less standardized what a rool-on-zpool layout ought to look like, and how to handle graceful, recoverable system upgrades. What Ubuntu could have brought to the table was a really high quality beadm replacement for Linux, moving Grub's ZFS support forward with support for more feature flags and boot environments, or investing in ZFSBootMenu.
Instead they went ahead with zsys, and it's still not entirely clear what they want or expect it to be. That they needed to write a daemon to enable "complex zfs dataset layouts" should've been an early clue that they were going down the wrong path. The dataset layout for a root filesystem should never be so complex, that you need to write a whole daemon to manage it.
The packaging issues and bad patches are almost besides the point.
Edit: handcuffing yourself to grub seems pointless. Not to mention, grub is controlled by oracle devs who want nothing to do with zfs support and have actively ignored bringing the grub zfs in line with openzfs mainline.