Ask HN: Why isn't BTRFS the default FS in home-oriented Linux distributions?
I have been looking how BTRFS has been progressing. It supports CoW, snapshots, data deduplication, compression, data integrity, (...), a list of features that are unavailable in EXT4 (the current default FS in most home-oriented Linux distributions).
For granted, EXT4 keeps being faster than BTRFS in most synthetic benchmarks, which makes it a better choice for servers, but this difference in speed isn't noticeable in domestic workloads; domestic users would greatly benefit from the features offered by BTRFS. For example, Linux Mint has Timeshift, a backup solution that uses BTRFS volumes to make system snapshots.
This leads me to my question: Why haven't we switched to BTRFS as the default FS for home-oriented distributions? What features / reasons are holding us to keep using EXT4 as the default FS?
70 comments
[ 5.2 ms ] story [ 138 ms ] threadDitto for ZFS btw.
that said, lots of people do. arch and gentoo default install instructions suggest it just fine, telling you the pros and cons over ext4
Plus, my understanding is that Linus Torvalds has a dislike of ZFS because it came from Sun Microsystems.
> And honestly, there is no way I can merge any of the ZFS efforts until I get an official letter from Oracle that is signed by their main legal counsel or preferably by Larry Ellison himself that says that yes, it's ok to do so and treat the end result as GPL'd.
> Other people think it can be ok to merge ZFS code into the kernel and that the module interface makes it ok, and that's their decision. But considering Oracle's litigious nature, and the questions over licensing, there's no way I can feel safe in ever doing so.
> And I'm not at all interested in some "ZFS shim layer" thing either that some people seem to think would isolate the two projects. That adds no value to our side, and given Oracle's interface copyright suits (see Java), I don't think it's any real licensing win either.
> Don't use ZFS. It's that simple. It was always more of a buzzword than anything else, I feel, and the licensing issues just make it a non-starter for me.
[0] https://engineering.fb.com/2018/10/30/open-source/linux/
I don't remember hearing about any power failures --- either UPS and transfer switches all worked 100%, node replacement covered any issues 100%, or I just wasn't in the loop. This isn't common at home; most people have at least a few brief power blips every once in a while, and those with many often get a UPS, but home grade UPSes often fail. I'd expect at least one unexpected power failure per 3 years in a home environment.
Most of FB production is built from ephemeral containers. If a container host crashes and restarts and the filesystem is corrupt, the container would be rescheduled somewhere else, and the host would be sent for repair (reimaging) automagically, and a product team like mine would not really know unless they really looked, maybe just that the container moved. That's not realistic for home users.
The containers themselves were short lived for many reasons. I'd estimate most of them lived for less than a month. A home user distro install often lasts much longer.
That said, I don't remember seeing disk corruption, but I do remember a series of issues where we couldn't write to disk because it was full, with disk usage of maybe 60%. As I recall, there was only one couple week period where that happened. This was resolved in the moment by replacing affected containers with new ones on other hosts, and permanently by a fix in the host kernels; broken filesystems were not fixed, they were abandoned. This isn't great for home users either.
There was a separate class of operations for persistent data storage; although there was work to containerize that as well, I had little visibility. Persistent storage machines were different and I don't know if they used btrfs or not.
This all doesn't mean it's not a good filesystem for home users, just that Facebook's endorsement doesn't transfer as the use case is much different.
This is untrue. ZFS is ready for prime time. The problem here is that it is licensed under a license that is not compatible with the GPL, so it will never be imported into the kernel. Because it's an external module, most distros dont offer it in their installer etc.
If you want zfs, you can either install it during install in expert mode, or go BSD where it is a first class FS
also, there does not seem to be a proper guide to btrfs. only random of bits and bytes here and there.
zfs on the other hand is very well documented.
Not sure I agree. On ZFS I have to worry about an out of tree kernel module and all the implications of that, extra things like setting the correct ashift, making sure the recordsize is tuned individually in datasets to not destroy performance for specific worksets, ensuring ARC doesn't cause OOM behavior with applications which allocate a lot of memory at once (since it doesn't use the page cache), etc etc.
With BTRFS however off the top of my head the only thing I remember doing was making sure nodatacow was set for subvolumes containing databases and VM disks.
You are absolutely correct about the ZFS ecosystem containing superior tooling however.
About two years ago I tried it with second hand hdds thinking it would be good for checking the reliability of the drives. After writing some data in one of them and a reboot it failed to mount it again, total data loss while it would have been at most a partial data loss if using ext4 thanks to fsck. This happened 2-3 times as I was testing and didn't care about the data, on ext4 no issues at all. Still using btrfs as the root fs on a new ssd, the snapshots are really useful for backups, no problems in two years. But for reliability I would try zfs instead.
2 - higher risk of losing data
3 - harder to explain and understand vs ext4.
4 - has more special needs. in some cases you need way more free storage for rebalancing stuff around... you don't need that for ex4.
---
now ext4 is surely worse than btrfs... but for the average home user btrfs is more of a problem than a solution, probably.
The snapshot is also a point in time to derive backups from, and usually costs nothing to make, as opposed to making a dd image of the disk, or tarballing everything, etc.
Whether you think you need those is debatable, but it's quick and easy enough, works for me.
[0] https://github.com/jimsalterjrs/sanoid
[1] https://github.com/kimono-koans/httm
[2] https://wiki.archlinux.org/title/Snapper
Personally, I'm a happy user of btrfs installed by default by Fedora and haven't had any problems with it, but I'm also not using any of the advanced features.
[0] Source: casual conversation and poorly remembered anecdotes
btrfs has never really matured. It has only a fraction of the features of ZFS and generally a lot more unstable.
This is the one that really burns me. There was a moment there where Oracle could have re-licensed ZFS, gotten it merged directly into linux, and had the best file system for their Linux distro. They even did it for dtrace! Instead, whether it was because they were so attached to their own pet project or other reasons, they just did nothing and now we get to live with the consequences forever because they don't own all the copyrights on OpenZFS so they can't even change the license on it now if they wanted. (Strictly, they could relicense their own Solaris code for ZFS, but that would be quite an effort to rework and leave us with two incompatible implementations.)
The only thing(s) that define a "server" distro are support contracts and ego. Ubuntu server is no different than standard Ubuntu when both use and can run the exact same packages.
"Ask HN: Why isn't BTRFS the default FS in home-oriented Linux distributions?"
You can still use with with plain old MD raid.
BTRFS documentation still greatly trail behind ZFS, imo. It's RAID1 implementation is subpar in terms of monitoring.
Can I ask about the circumstances of losing data on ZFS? I've been viewing it as 100% rock-solid, so I'd very much like to know about any failure modes I'm missing.
There was one instance of issues with encryption, but the feature was in testing at the time, and has long since been rectified.
As for failure modes... the first issue above caused issues with send/receive, the second issue caused empty/sparse files to be written instead of data, and the encryption issue caused issues with send/receive and/or garbage data.
The corruptions were identified by scrub after an unexpected restart, but the files were already corrupted before that (the system that was using the iscsi target behaved strangely whenever it accessed the drive.)
Server: Lenovo 3U server Xeon E5 v2 with ECC ram. I forgot what's the controller.
Or is it a side effect of streaming everything? Ie who cares. Ie file systems are solved tech and nothing really new until permanent storage is as fast as RAM in all aspects, replaces it, and we get some new computing paradigm.
From the FS rather than the OS side, but I feel like support is pretty good?
ZFS supports all the big ones and many smaller options - Windows, Mac, Linux, FreeBSD, NetBSD, illumos (albeit with some slight incompatibilities depending on enabled features). The only OS that I kind of miss it on is OpenBSD.
BTRFS is, I grant, behind in this aspect, but it still has Windows and Linux, which is... a start, at least.
I am using it again but on a NAS with a backup battery.
There were some threads on HN related to BTRFS which made me look into its suitability as a RAID5 or RAID6 file system, and what I saw made me back-pedal very fast.
Its support for RAID1 or RAID10 (mirroring or stripe-mirror) is okay, but it has glaring issues in all other modes that have gone unresolved for many years. There are some fundamental design issues that might mean that these problems will never get solved, with data loss or data corruption being the inevitable consequence.
Storage is like multi-threading: you've either mathematically proved the correctness of your algorithm, or it is Wrong with a capital W. Storage is not like a web app where an occasional HTTP/500 is no big deal and recoverable. You stuff up something even a tiny bit, and the consequence is shredded or lost data and a very bad day for someone somewhere.
I'm just not seeing the right attitude from the team working on BTRFS. They've been very lax about data integrity issues, recovery from expected failures, etc...
My advice is: stay away.
Sorry, it's the only FS I've lost data with. It seems to have a problem when disks start getting hardware errors and some blocks become unreadable. I mean of course any FS suffers in this situation, but most don't lose the entire volume as easily as btrfs seems to.