Tell HN: ZFS silent data corruption bugfix – my research results
since I'm personally affected by this bug[1] and this scared me somehow, I thought I'd post my research results to have a discussion about the details.
I must say I lost a bit of trust in ZFS, since it was 'sold' as rock solid unbreakable file system and now what... the sheer complexity of ZFS let's me think that this was not the only bug of this kind existing undetected in the source code.
However the communication was clear and precise, other filesystems had similar problems and I just love the feature set of zfs. So I'll give it another shot hoping that future revisions will be safe(r).
Here are my conclusions so far:
1. ZFS had a silent data corruption bug recently discussed on HN [6][7]
2. `Silent` means, you can't do anything about it without upgrading zfs (scrubs, checks, etc. don't help)
3. There is no tool to check if your data has been corrupted
4. Only once written and untouched data should be relatively safe (e.g. backups), the bug mainly affects reading data [4]
5. Setting `zfs_dmu_offset_next_sync=0` reduces probability of being affected, but no guarantee [4]
6. The bug affected any version before 2.2.2 and 2.1.14[2], but were more likely between 2.1.4 and 2.2.1 because a behaviour change in coreutils
7. The causing problem already existed much longer (since 2006?) - this is a question mark [4]
8. do a `zfs version` to see the version number
9. If you're scared (I was), please read this comment: https://news.ycombinator.com/item?id=38553342
Please let me know, if I missed something or got something wrong.
Sources:
[1]: https://www.phoronix.com/news/OpenZFS-Data-Corruption-Battle
[2]: https://www.phoronix.com/news/OpenZFS-2.2.2-Released
[3]: https://www.theregister.com/2023/12/04/two_new_versions_of_o...
[4]: https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_bl...
[5]: https://news.ycombinator.com/item?id=38519382
89 comments
[ 3.1 ms ] story [ 74.9 ms ] threadHonestly, ZFS is the best thing on the (Free)BSDs only... On Linux it doesn't even use the page cache, and you conflict severely with L2ARC. I know there's a variety of people who don't care, but still for real users it's not an actual option.
I finally moved to btrfs earlier this year, and so far I'm glad I did.
I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Tailscale.
Replication is with rsync and borg, not snapshots.
Yes, it's more painful to replace disks after a failure, but once you get a bit used to it, it's really no big deal.
On my main workstation/laptop the dedupe and compression work much better than my experience with ZFS.
[1] https://news.ycombinator.com/item?id=16797644
I use LUKS / dm-crypt on entire drive partitions. Did that with ZFS too, and by the time ZFS got this feature I was already planning my migration to btrfs.
I like encrypting with LUKS, because I can have a drive configured with mine and my wife's password. Either one of us can take that drive, plug it in, and Gnome will put a nice graphical prompt asking for a password, then decrypt the drive and mount the filesystem.
If I get hit by a bus, LUKS makes it much easier for my family to get access to important data without having to have that data sit around somewhere in plaintext.
For me on Fedora, uptime is better and maintenance overhead is lower on btrfs.
The kernel fscrypt layer just needed a lot of refactoring first to support cow filesystems.
I backup my files with borg, but I still use snapshots during the Borg backup to ensure files are not modified during the process
Also a complicating factor with kernel upgrades is that while zfs release notes clearly delineate what kernel versions are supported that information doesn't appear to be meaningfully encoded in package metadata so if you use new enough kernels compared to the version of zfs for your distro it is possible to front run support. For instance 2.2.2 supports up to 6.6 but you could very well for instance install 6.7 and it might not work.
The somewhat broken thing is not encoding known data like which kernel is supported to automatically do the right thing in the packaging system not the filesystem. The lazy fix is to just manually handle kernel updates. The lazier one is to grab the release notes and update if latest kernel is <= supported.
Re btrfs raid 5/6, yes everyone knows about this, and this is why I have it only on my backup system. My primary data which holds 15TB of family photos and videos is on raid 1. The offsite is there only for the time my house burns down.[1]
[1] I watched my neighbor's house go up in flames 2 years ago, and it finally got me going on setting up remote backups. The fire spread to 3 other houses, and everything happened very, very quickly. No one got hurt, but multiple families got displaced for more than a year. Besides having backups , it's also a good reminder to have adequate insurance. One neighbor did not.
L2ARC isn't typically recommended for many use-cases. Anyway, what's the conflict you're referencing?
This is the case on FreeBSD as well, ZFS uses its own cache, but UFS and I guess other filesystems use the kernel page cache.
Even easier than that, type "zfs version"; it will report both the loaded kernel module and the userland version.
Here's a post by RobN (the dev who wrote the fix) on the ZFS On Linux mailing list
> There's a really important subtlety that a lot of people are missing in this. The bug is _not_ in reads. If you read data, its there. The bug is that sometimes, asking the filesystem "is there data here?" it says "no" when it should say "yes". This distinction is important, because the vast majority of programs do not ask this - they just read.
> Further, the answer only comes back "no" when it should be "yes" if there has been a write on that part of the file, where there was no data before (so overwriting data will not trip it), at the same moment from another thread, and at a time where the file is being synced out already, which means it had a change in the previous transaction and in this one.
> And then, the gap you have to hit is in the tens of machine instructions.
> This makes it very hard to suggest an actual probability, because this is a sequence and timing of events that basically doesn't happen in real workloads, save for certain kinds of parallel build systems, which combine generated object files into a larger compiled program in very short amounts of time.
> And even _then_, all this supposes that you do all this stuff, and don't then use the destination file, because if you did, you would have noticed that its incomplete.
> So while I would never say that no one has ever hit the problem unknowingly, I feel pretty confident that they haven't. And if you're not sure, ask yourself if you've ever had highly parallel workloads that involve writing and seeking the same files at the same moment.
https://zfsonlinux.topicbox.com/groups/zfs-discuss/Tcf27ae8f...
Here's another writeup by another ZFS dev Rincebrain https://gist.github.com/rincebrain/e23b4a39aba3fadc04db18574...
I think the only reason this has gotten so much attention is because it came up as a block cloning bug (which it's not) and that being a new feature created a massive scare that it's widespread. This isn't the first or the last bug ZFS has had - it's software.
Please correct me, but ZFS has none.
ZFS's scrub is both a check and a repair tool. It's already saved some of my data.
To me it was always weird to not have a separate tool for being able to do an offline ZFS repair.
With regards to data corruption, I mean, this is exactly why I moved to btrfs, because it's able to catch bit rot same as ZFS.
I think ultimately this is why ZFS data corruption bugs always are such a big deal... It's because in a lot of tech circles ZFS is put on this infallible pedestal where it can never do any harm to your data.
It just has a different name, and it can fix more kinds of corruption (including corrupted data, which `fsck` for other file systems can never do), and can do so on line. Of course zfs can fix those same errors on line while doing ordinary reads, so really all `zfs scrub` does is read everything.
`zfs scrub` is better than `fsck`.
`zpool scrub` walks every block in the pool, and implicitly, you do checksums and other things while doing that. That's it.
It's not doing any sort of logic bug repairs or cleanup or anything else.
It's also not checking that you can, say, decrypt things, since that would mean you needed the keys to scrub.
In truth fsck is a wart, a kludge, a bag on the file system design.
The key thing to realize is that you can't, actually, automatically fix every problem, sometimes you have found a logic problem which results in an impossible outcome and you need someone to manually clean it up.
In a world without flaws, it would be great to never need that. But the thing about theory and practice is that in theory, they never differ, but in practice...
The discussion was about the need for tools to make it easier to handle cases where you couldn't automatically handle repairing them, and your statement was that zdb is very useful, which is true, but it doesn't fix anything.
fsck for various filesystems has a bunch of common cases like "this is an orphaned file, should I save it or mark it free?", and something similar would indeed be useful for a number of failure cases in ZFS which require more explicit instruction on what to do about it because you can't easily automatically resolve it.
`zpool scrub` is very useful, but ZFS could still benefit from automated tooling to handle some common failure modes, not just let you write bespoke tooling every time.
This is an example of an automated fix that zfs just handles transparently, without needing to prompt the user. Or it would, if zfs could even have orphaned files, which it cannot.
I don’t know why you think this is such a win for fsck, which doesn’t even bother to give you any idea what the file was. It doesn’t try to show you the contents, and it probably doesn’t know what the file was called, or why it was deleted. Or even if it _was_ deleted; a file could be orphaned merely because the data was written but the write to the directory entry got lost. The user has nothing to go on and just guesses, or says `y` for everything. Useless.
> `zpool scrub` is very useful, but ZFS could still benefit from automated tooling to handle some common failure modes
This is precisely and exactly what scrubbing does! All failure modes that can be automatically fixed, whether they are common or not, are transparently fixed without even needing to unmount anything.
zdb is there for the really rare cases where there is so much damage to the filesystem that zfs cannot even mount it safely. Other filesystems don’t have anything like it.
If you use the zfs_recover parameter, then in a couple of cases, it will just permanently mark space as unallocatable forever because it can't figure out what owns it due to some errors, and you decided that was a better outcome than whatever error it was encountering. (That's what the "leaked" zpool property counts.)
Conceivably, you could write something to carefully figure out what, if anything, owns it, and allow it to be freed, or grab the contents of that region and drop it somewhere and then throw it out after the same safeguards, but by definition if you triggered that handling, something has gone wrong and you don't have a better automated intervention.
I wasn't arguing that ZFS has a case requiring orphaned files, but my point was that that was an example of "I don't know what to do with this, but I know enough to realize I can't decide what to do about this or just throw it out, so here, you do it."
Or if you, for example, wanted to do a destructive rewind on a pool because of some horrible edge case, then you might want some automated way to extract everything you're about to throw out, and that might require more work than just `zdb -R` if the pool won't import in the first place.
One example of something that would be useful to be able to do, particularly offline, would be when pools have some issue like spacemap corruption, you could conceivably walk the non-spacemap metadata to re-synthesize the spacemaps from whole cloth and write them out again. (And it's not an all-or-nothing thing, a number of versions many years ago would have very minor errors in how they computed spacemaps, which just mean zdb whines at you if you ask it to verify them even when the pool is offline, but don't interfere with the running otherwise.)
Could you convince the kernel code to do that for you on import if it's blocking import? Probably, though you'd probably want some out of band communication method to force it sometimes because I would bet there's any number of ways it might go awry and get too far into the woods of thinking it's "fine" before tripping some assertion.
Is it going to be faster to iterate on it from userland, particularly if you can simulate whether the import works from something like zdb with those conceptual spacemaps written out somewhere, versus rebooting on kernel panic? Absolutely.
But that's an example of a case you might have something so disgruntled it could be nice to reconstruct it outside of the normal import flow.
> All failure modes that can be automatically fixed, whether they are common or not, are transparently fixed without even needing to unmount anything.
`zpool scrub` doesn't fix anything except checksum errors. That's it. Any other class of flaw, it doesn't handle. I don't know why you think it does anything else, but I promise you, it absolutely does not.
Have you ever asked yourself why those operations need to be done offline?
It's not that ZFS doesn't have these capabilities, it's that ZFS is doing these similar checks and repairs as part of its normal operation and more. If you can't mount a ZFS filesystem, odds are a similarly borked ext4 filesystem isn't going to be repaired by fsck.
I got bit by some pretty bad foot guns in BTRFS that has meant I steer pretty far away, but my understanding is these have been somewhat mitigated. I hope it continues to work well for you.
The key differences are in two places:
1. Every administration task for ZFS is done online. You don't need to take your pool offline just to check for errors and repair them (if possible). (Mind that on FreeBSD, UFS can usually have a fsck done in the background while the file system is in-use. Just about zero Linux-native file systems have this capability)
2. "other filesytems" can only hope to detect and repair inconsistencies in their metadata structures. If your files are corrupted, they can't know and won't tell you that they are. ZFS checksums everything, including regular file data. It will repair regular file data too, if possible.
Debian bookworm users: You need to enable backports for zfs. Both bugs are fixed in backports but still present in stable.
Let's do away with the tribalism in both ways? Calling it perfect is quite a stretch. I'm not saying "don't use zfs", just "you want to be prepared for the worst-case if your data matters". 3-2-1 and "validate your backups" still apply.
Backups is the only (near) 100% way to insure you data I agree.
It's a fork of other work [1], modified and updated for Debian and Debian only. It also in absolutely no way has any best practices, which is why I'm doing a complete re-write with proper role dependencies, modularity, etc. When that's done, I'll be marking [0] as archived and linking to the new repo.
The new version also replaces the incredibly hacky shell script builder (hooray dialog) with Python, and updates Packer's JSON with HCL2.
[0]: https://github.com/stephanGarland/packer-proxmox-templates
[1]: https://github.com/chriswayg/packer-proxmox-templates
Its easy to feel like its being targeted with some kind of campaign, however the truth is rarely that exciting. The software is being used by more people, which means it will expose more bugs.
No software is perfect, it just now starting to be abused enough to be important enough to be talked about. This stage is entirely normal.
This bug is super hard to trigger, and has roots back to 2006 and only surfaced with a recent coreutils update. Yet there are numerous posts in the past few days verging on hysteria. I myself was very worried until I saw posts like the grand parent comment or the linked GitHub issues.
I don't believe there's actually a campaign of ZFS haters sitting around in a storeroom somewhere posting to phoronix and HN, but it does feel like a lot of strongly held opinions are being unleashed now that ZFS had this bug.
I'm pretty sure ZFS has been abused a lot - ten years ago I was helping test petabyte scale ZFS clusters, doing tons of concurrent IO and such. I'm pretty sure Sun and Oracle tested the crap out of it.
> filesystems. People are acting like upgrading to ZFS 2.2.0
> ate all their data like XFS used to back in the bad old
> days. I remember once the power went out at my house, and
> the entire XFS filesystem was irreparably damaged.
Different bugs manifest differently due to the coding, this is just the same.
ZFS wouldn't even have a hundredth of the testing of real-world use ext3/ext4 in hours. The large scale deployments of ext3/ext4 dwarf the usage of ZFS.
https://archive.ph/20120529150649/http://git.kernel.org/?p=l...
ZFS released: 2005
https://web.archive.org/web/20130619165135/https://blogs.ora...
This is Linux local bias. There is more to Unix than Linux.
Back around 2001 when work on ZFS began, I very strongly suspect that there were several orders of magnitude more storage on ZFS on Solaris in datacentres around the world than there were on all Linux filesystems combined.
Just because you are more familiar with the FOSS tool doesn't mean that it's more mature or more widely used or more deployed in production.
I don’t think we had any customers request ext4 bulk data storage. OS drives sure. Everything else was ZFS. Or weirder proprietary filesystems!
You simply can't begin to understand the amount of shitty iot and android devices that exist.
I think you have this 100% backwards. I heard similar comments to yours inside some large Linux vendors, too, and I think they betray a profound misunderstanding about the nature of large-scale Unix deployments over the years.
FOSS Unix is recent and mostly small-scale and transient deployments of containers, not long-term storage... but before that, there were decades of vast scale commercial Unix systems, which dwarf the FOSS stuff.
ZFS is an enterprise storage-management tool. It's not just another hard disk format, like ext4.
Not many people are running Linux boxes with root on ZFS because few distros support that.
Single-disk setups are not its strength so the number of single-disk setups of ZFS compared to ext4, which only supports single-disk setups, is irrelevant.
Even saying that, though...
I bet all FreeBSD deployments are on ZFS, and it's got a solid enterprise presence. Anything streamed off Netflix anywhere is being served by FreeBSD.
Oracle still supports Solaris. All Solaris, OpenSolaris, Illumos, Nexenta, OmniOPS, SmartOS, Tribblix... all on ZFS.
And it's still not its core area.
If you want to do a fair comparison let's hear about companies doing petascale to exascale arrays with double-digit to quadruple-digit numbers of drives.
How many of them do you reckon use ext4?
Here, let me quote myself.
> The software is being used by more people, which means it will expose more bugs.
I don't think I ever heard about this, apart from in the context of the ZFS bug. And although inline_data is niche, ext4 as a whole I would argue is not.
Actually lseek seems to have been broken on most filesystems at some point: https://bugs.gentoo.org/891125 https://github.com/gluster/glusterfs/issues/894
And apparently apart from modern coreutils using that, it is mostly gentoo users hitting the bugs in lseek.
'Et' is Latin for 'and'. The term means "and other things", not "andotherthings".
No, I have nothing important to add to the ZFS discussion.
Maybe I do not feel it like a big issue, since many Latin languages use it a single word (such as Italian with eccetera, Spanish, Portuguese), due to the evolution of the language.
I'm just joking. This is just a pet peeve of mine. Thank you for the fun lesson in etymology.
It makes it sound unlikely, but if I have a couple of VMs in datasets (all formatted as ext4 internally and some running DBs inside them) each is one big `raw` file which is getting a lot of reads and writes, I assume.
How are they at risk?
Also, what about ZVOLs mounted as ext4 drives in these VMs?
Uhhhh, databases?
First thing I'd suggest is really getting the terminology under your belt. Everything makes so much more sense when you use the correct terminology. For example, you don't extend a drive. That doesn't really make sense. I suspect what you mean is that you added a vdev to a zpool. Don't think about things in terms of disks and extending them, think about things in terms of one or more disks making up a vdev, and a zpool is, well, a pool of those vdevs. Zfs then works out how best to write data across the vdevs in the pool. Not disks.
Personally, I used (past tense) ZoL in 2014-2017 on Ubuntu. The issue is that the array eventually entered an unrecoverable state where it could no longer be mounted RW. That wasn't the end of the world, but the support from ZoL was to shrug at it. That was the end of that because without support and without pride, something that appears shiny is effectively useless.
Been running many XFS volumes over mdadm RAID10 near-2 arrays. Zero major problems in 5+ years with over 400 TiB online. SGI's, Redhat's, and more contributions to various Linux storage components are excellent.
My conclusion is that ZoL != Solaris ZFS. Once it left enterprise with controlled hardware and dedicated engineering & support teams, it regressed and devolved. Beware of fanboys where passion and tribalism exceeds evidence and reliability.
I would suggest that you should probably have a support contract with _somebody_ if you want a guaranteed turnaround time on responses. Plenty of people are paid to work on OpenZFS, but if you're not the one paying them, then it's always going to be a best effort thing.
Since the incorrect code dates back to Sun, I don't know that you should be using this specific bug to cast stones.
One corner case data loss bug in 20 years? Throw that baby out, the bath water is bad!
Back in (say) the 1980's, the code for a serious OS's filesystem was small enough that formal verification was at least imaginable. Similar for writing test code for ~all of the corner cases.
Not now.
When and where you value reliability - try to keep things simple, stick to code paths well-tested by the passage of the masses, do at least some testing of your own, and have a Plan B.
Since the problem is in the read path, having copied them off ZFS might result in them being incorrect on the destination, modulo all the caveats about this being quite rare and hard to hit unless you're very deliberately trying.
You can reproduce this back on 0.6.5 and pre-OpenZFS merge FreeBSD and illumos, if you really want to.