I might be misguided here (please correct me), but I feel like ZFS and hardware RAID had their place pre-SSD, and now they don't make much sense.
If you want an array of disks with a mutable configuration, I think the right choice is mdadm and LVM. They're reliable, relatively easy to use, let you move disks between machines, don't rely on hardware RAID, and support all file systems.
If you need something faster and more optimized like ZFS or btrfs, you're probably better off spending on SSDs anyway.
does that talk to the raid part so it can restore the correct sector when two raid1 mirrors disagree? And it seems to come with its own journal, so stacking several journaled systems could lead to write amplification, no?
You would use dm-integrity on each individual disk below the RAID, and when it detected the corrupted file it would return an error, which would then be used as a signal to the RAID system to read from the other disk.
You have to manually set up dm-integrity on the disks, however - it is not used by default when creating an mdadm/lvm setup.
That setup would mean more overhead. Separate journals for each disk, one checksum copy on each disk instead of only up to your redundancy level, more disk writes.
It's not "more" overhead, just a different distribution of it. You can actually use erasure coding instead of conventional RAID once the lower-level blocks are protected by their own checksums.
It is certainly more overhead when you have to keep a journal for each disk and a checksum for each cluster on each disk instead of one checksum per stripe or extent and a per-filesystem journal.
In other words zfs and btrfs incur less overhead by having the raid, checksumming and journalling layers talk to each other. Implementing them separately in as in device-mapper comes with inefficiencies.
Price per gigabyte. Far more economical to use ZFS on spinning disks with a huge NVME + RAM cache than it is to buy 80TB of NVME. As long as you don’t need fast random access to the whole dataset all-SSD is wasted.
I think most people use ZFS for data integrity, not performance.
I make backups regularly (manually, because I want them to be offline), but I'd rather not have to use them if one of my drives die. Unless I literally just finished making the backup, I'm going to lose some amount of work.
My mirrored ZFS pool means that (A) a drive can die and I won't lose _any_ data, no matter how recent, and (B) I'm confident none of the data on either drive is corrupted, because ZFS checks it regularly (every time it's accessed, and during weekly scrubs).
Exactly. Anyway, my home server is having 6Tb redundant storage (3xToshiba 3Tb) and 10Tb Hitachi drive, non redundant. What would the cost of SSDs be? Over $1000 not to mention the controlle 4 SATA port vs 19+ port controller is a huge difference in price.
I do use SSDs, where I need speed - L2ARC ;), far better use than buying 19Tb of SSDs, not to mention the price.
Anyway, I do use ZFS on laptop for linux root. Reason? Snapshots. Kvm + ZFS, perfect match. /.zfs/snapshots? Fabulous.
RAID that knows about which blocks are occupied and which are not will take a shorter time to rebuild because it'll need to read and write fewer blocks. For `mdadm` to do that, it'd need to know a lot about the internals of the filesystems that are mapped to the block device.
Filesystems like BtrFS and ZFS know what is being used and what is not and won't waste time reading or writing to blocks that don't hold any currently used data.
ZFS is optimized first for data integrity, not speed.
It's like you're saying, why do we need ECC when there is fast DDR4 memory?
ZFS is a filesystem not just a layer like LVM. You have to compare it to LVM + some filesystem. Which filesystem would you suggest using with LVM to get ZFS-level integrity?
(Technically you can use ZFS on top of LVM but that would be a weird thing to do.)
I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for.
With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? If the concerns about "data integrity" are really about time until a disk is replaced, then it makes sense. Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
>I use ECC and mdadm+LVM+ext4. I haven't had any issues yet, but this thread is making me wonder if there's some data integrity problem that I'm setting myself up for.
Nothing in your setup there protects you from bitrot.
>With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing?
Drive rebuild time is primarily a function of a variety of things - speed of the disks, processing power involved, etc. There's nothing magic about any solution here. ZFS does get a leg up in that it knows what blocks are in use, but hardware raid and mdadm are quite fast when they find empty blocks too, so it's not some overwhelmingly massive improvement.
>Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
I think you're fundamentally confused about what journaling provides a filesystem. A journal helps you in situations where your system crashes, loses power, etc. and you had not finished committing your changes to the filesystem. It does not protect you from data integrity issues that are unrelated to this sort of situation - a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal.
> a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal
Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Is the difference that ZFS scrubs in real-time?
Sorry if I'm being dense here, but I did a fair amount of reading about this, and I'm trying to understand what I'm missing.
>Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot.
Nope. Parity bits aren't used to detect bitrot. If you get a URE when you go to read, the parity bits will be used to get you data. If you read successfully, nothing happens at all. If you scrub, and the data is found to be inconsistent, md assumes that the parity is wrong and overwrites it.
>If a read error is detected during this process, the normal read-error handling causes correct data to be found from other devices and to be written back to the faulty device. In many case this will effectively fix the bad block.
>If all blocks read successfully but are found to not be consistent, then this is regarded as a mismatch.
>If check was used, then no action is taken to handle the mismatch, it is simply recorded. If repair was used, then a mismatch will be repaired in the same way that resync repairs arrays. For RAID5/RAID6 new parity blocks are written. For RAID1/RAID10, all but one block are overwritten with the content of that one block.
Thanks for the explanation. I didn't realize md didn't use the parity checks unless there's a URE.
Is it correct to say that ZFS parity is used to reconcile on each read even without UREs, but with md parity is only used when there are UREs? If that's true, it makes much more sense why ZFS is considered the right choice.
Also, do you know if adding luks in the mix changes that?
My data goes md->lvm->luks->ext4 and I hadn't considered which part of that is supposed to catch the error if it wasn't a URE.
It makes sense that md would update parity if there was no URE. It also makes sense that LVM wouldn't know or care about data integrity.
I would think luks would require data integrity to work properly, but I'm not sure what happens if there is bitrot in a luks block.
ZFS has native encryption now that is solid. So you can even avoid luks if you can live with some higher meta data leakage. There was a great talk about the details you can find on yt.
luks won't change that at all, it'll still be at the md level overall. The main encryption modes used by luks will lose a block to bitrot and such but won't lose the rest of the drive. They essentially generate a new key (really a new IV/nonce I think?) for every block through various means. This is the only way to allow for random access to any part of the disk without having to decrypt the previous blocks, and rewriting the rest of the disk whenever you change a part in the middle. I think you can setup an authenticated luks mode, but I've not looked at it and that should catch errors but that's all it would do, catch them.
I went ~2 years without running a scrub when I first started using ZFS many years ago; when I ran my first scrub, it found 2 file corruptions, one in a random picture back when I still copied pictures from a digital camera to a hard drive, one in a random file in the browser cache.
(If I'd been scrubbing regularly, these almost certain would have been found before corruption crept into the mirrored copy of the file, and been recoverable.)
This is a great point about how pernicious corruption is: it replicates. Detection is more important than correction, at least you can stop the corruption from spreading. It's typical for people to reuse storage hardware, and each derivative inherits prior corruption as well as contributing its own. Most people don't keep a long term unchanging archive, and even that archive likely has no means of unambiguous verification of data integrity.
To be honest I'm not sure if the corruption was replicated directly, or if it's just that the same file got unlucky on two different drives -- I can't recall if this was before or after a drive failure.
> Your setup won't save you from e.g. bad cables silently corrupting data, or drives simply returning bad data, etc. etc.
I thought this was what RAID was designed to prevent. I don't know what a cable failure would look like in my setup in practice, but I would think the disk would start failing parity checks against the rest of the array and be flagged as bad.
Well, each RAID implementation has different designs and goals in mind, but the original design was simply meant to replace expensive single drives.
Using more spindles you could get more performance out of a bunch of cheap disks, and to help with the fact that these cheap disks failed more often, you made them redundant.
In general, dealing with data corruption that does not generate an error is not within the purview of RAID implementations. ZFS and btrfs handle it because they're more than just RAID implementations. mdadm can do similar things in a RAID setup if dm-integrity is used on all of the drives (but it is not used by default when just setting up mdadm/lvm/etc.)
RAID is primarily about redundancy and speed, not really end-to-end integrity. (Though some setups will certainly have some measures against corruption.)
ZFS is really designed for end-to-end integrity. Once you're stored data on ZFS, you're basically guaranteed that you'll always get the exact same thing back (or an error)... no matter where in the "chain" of I/O the failure occurred.
On the last round of zpool scrubs of the 13PB-used set of zfs filesystems here (raidz3, mostly SAS drives, all hosts with ECC RAM), 50MB of data was repaired.
50MB isn't much compared with 13PB, but silent corruption is a real thing.
One thing is that ZFS tells you exactly what file is corrupted. I recently took over a ZFS setup that was built with plain ZFS on top of a 24-drive wide RAID6. Drives failed, rebuilt the RAID6, nothing appeared to be wrong. ZFS scrub revealed two corrupted files. I was able to pull backups of those two files, replace them on the FS, and rerun the scrub.
Obviously I moved off the RAID6 soon after, but this is a powerful example of why the checksuming of ZFS is so useful.
RAID5 corruption is non-recoverable - you can't tell which is right, the data or the parity.
RAID6 is recoverable - you can get consensus between the data and the two parity bits.
ZFS stores hashes of each block, so as long as you have two copies, you can just pick the data+hash that's correct and restore the corrupt block, even with only two copies of the data.
The core ZFS feature being references here is storage checksums. The general argument is that as storage volumes have increased in size and/or workload, the conventional checksum methods have not been changed to increase their reliability in a way that prevents real world corruption on real world storage during their real world lifetimes.
A RAID system can only reliably recover when an IO error is reported. In this case, it can assume which element of the stored data is no longer trustworthy and use the other parts to reconstruct it. A RAID scrub operation that finds inconsistencies, without a reported IO error, cannot really determine why it is inconsistent. It doesn't know which element is corrupt and which other elements to use to reconstruct it.
In effect, ZFS adds another data integrity layer over the physical storage, so you can have a lower probability of silent failures where the disk controller or IO channels do their own "parity" or other checks and report a successful read when they actually returned corrupt values.
Something like mdadm or LVM would need to add a new layer of checksums over the physical devices to provide this level of protection. Each block/chunk/stripe written to disk would need another integrity checksum written as well. During operation, these checksums would be verified before deciding that a read operation was actually successful. If the verification fails, a virtual IO error is synthesized to flag the read as bad, so that the other RAID recovery techniques can then be applied while knowing which block/chunk/stripe is invalid.
The issue is not a loudly failing disk. If you get read errors, or if the disk is simply offline, it's easy to recalculate the data.
But what if you have silent data corruption: a single bit?
Even if you have parity (raid5) or a duplicate (raid1), how do you know which part is correct, and which part to repair? Without checksums.
I've had to restore 1TB from Crashplan. They apparently had checksums in their software, and something like 10-15 files were damaged.
Without checksums, you simply don't know if your data is damaged, even on a single hard drive. ZFS gives you data integrity with checksums, and redundancy with mirrors or raid-z.
This is why a regular scrub is recommended, you verify the checksums and repair if possible.
I was won over to btrfs for my new NAS (~6 years ago!) when I saw a demo (which I expect is also valid on zfs) where the presenter starts with a mirrored setup, finds the exact block offset location of the file's contents and then overwrites a byte in it with a zero using /usr/bin/dd against the raw disk, /dev/sdX.
He then reads the file, uncorrupted, with a kernel warning about a bad checksum that was fixed (from the mirror) during the read.
This demonstration is why I switched to btrfs (and ECC) for my new home server/NAS.
That, along with compression sealed the deal. I was disappointed about not having dedupe at the time, but it probably really wouldn't have saved me that much space. (I keep meaning to run the dedupe tool to see what it might save me now.)
Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent.
Notably, Apple's new filesystem introduced in 2017 does not detect data corruption; it only checksums metadata. https://en.wikipedia.org/wiki/Apple_File_System#Data_integri... Apparently Apple Engineers are of the opinion that Apple SSD and HDD media simply don't have silent data corruption. They have some pretty smart engineers, so maybe they're on to something. It does beggar the question, though: why have metadata checksums at all? http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/
Anyway, I'm not exactly supporting threat models that ignore silent data corruption. Just pointing out that they're not exactly as ridiculous as they might seem on their face.
> Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent.
Linux md will do the completely wrong thing with this: it will treat that as a drive error, and if you accumulate a lot in a short time (such as rebuilding an arrary) then it will conclude that this is a failed drive and kick it out of the array.
Well, with modern disk sizes it will almost certainly kick out all the drives sooner or later, making it essentially impossible to ever rebuild an array after a disk failure if you weren't scrubbing regularly before the disk failure (and dicey even if you were).
I totally believe you but this has never actually been a problem for me with Linux md. I tend to replace drives when the very first error is detected, and for spinning rust I usually get a very small number of errors initially. Also, md on my system seems to automatically run scrubs weekly. Currently observing 0 for Raw_Read_Error_Rate and Reallocated_Sector_Ct for all drives, with 24k Power_On_Hours (>2.5 years).
Weekly scrubs put you in a much better situation than not having them. I suspect your distribution must have done something to close the gap, because vanilla md does not put them in place and the tutorials do not always make it clear that setting up scrubs is not a nice-to-have thing but something that's absolutely critical if you want to have a hope of avoiding data loss.
How many array rebuilds have you done? You only see the read errors when you go to actually read the sector in question, and even in a week I wouldn't want to rely on not having any sectors go bad.
Given enough writes, any SSD will eventually fail.
As a related anecdote, I did some stress-testing on a ZFS filesystem with two Kingston NVMe SSD-s in mirrored configuration and at one point I got two random read errors on one drive, which subsequently got corrected by ZFS.
Silent failures might not be the best phrase to use for these situations, but ZFS does let you know that corruption has taken place and you have to acknowledge that message for it to disappear.
If the SSD controller isn't bugged, lifetime write limits don't result in silent data corruption; they just result in errored writes and inability to persist new data. Apple may have decided to only qual and sell SSDs that they've tested to failure, rather than throwing darts at random.
It's not clear to me if you're describing silent data corruption or detected data corruption (at the controller level), but either way Apple isn't selling Kingston NVMe SSDs; they have their own SSD controller and vendor the raw flash.
The journal contains information that can be replayed on a crash to hopefully avoid an file system check and repair to make the file system consistent. It's intended to address how long fsck can take as file systems get bigger. It has nothing to do with data integrity per se. Recently ext4 adds journal and metadata checksums (feature metadata_csum) to detect corruption, but data is not checksummed by the file system. Each drive sector is checksummed. The expectation by ext4 (and XFS) when it comes to data integrity is that it's the responsibility of the hardware or some other layer, like dm-integrity.
The data integrity problem is you don't have a way of unambiguously determining data integrity. You have to trust the hardware. Obviously that's good enough most of the time, but it can transiently misstep without outright failing: bit flips, torn or misdirected writes.
ssd's still can become corrupt (bitrot) and can still die.
File systems like zfs and btrfs checksum every block to ensure that if they do become corrupt, they can get repaired. (otherwise, if you had a mirror of disks, and one disk says the value is 0010 and the other says its 0001, wich one is right?)
Additionally, the 'raid' concept is not just about speed. Its also about capacity, and how many disks can fail. I might have 48 drives in a single server. I certainly don't want an LVM stripe across all drives and lose data. I also don't want raid1, losing the capacity of 24 of the drives. I want it so that any 3 drives can fail in that system, and I can still recover the data.
ZFS might not have the single disk, raw synthetic benchmark performance of a stupid file system with no correctness/integrity built in, but that use case isn't really what anyone wants or needs.
ZFS isn't really a filesystem. It's more like a storage backend, that happens to support the POSIX interface. The original intent of ZFS was to completely rethink the volume management/filesystem layering nonsense, and to provide for storage what the MMU/VM system provides for memory.
So it can do much more interesting things like memory backed read caching (via ARC), SSD backed read caching (via L2ARC), and SSD backed write buffering (via ZIL/SLOG), as well as the normal RAID stuff.
LVM software RAID has the same problems as ZFS's implementation and hardware RAID. When drives fail, high capacity arrays will take over 24 hours to rebuild themselves and are likely to run into additional read errors due to the heavy load. Expanding the capacity of the array is an incredibly complex process:
It should be possible to buy disks of any size, add them to the computer and tell the file system to use them. Drive failures shouldn't increase the risk of additional failures.
There are two kinds of LVM snapshots: thick and thin. You're probably thinking of conventional (thick) snapshots, rather than the newer thinly provisioned snapshots. Snapshotting an LV will FIFREEZE the file system, then snapshot, then FITHAW the file system. So depending on how dirty things are, it might take a while for the snapshot to happen, but otherwise they are nearly instant to create and destroy.
I am aware of the difference - and even the much-touted thins are very poor by comparison, as anyone who has had the displeasure of trying to use them in anger can attest.
I have but only small scale: single 2T drive and data, with maybe a dozen snapshots. I use Btrfs for most everything these days, with ZFS archive. The biggest issue I had, a while back now, with thinp is if the metadata volume gets full...kaboom. And it's non-trivial to repair.
Isn't that a bit of rewriting the history? Originally the roadmap was that these kinds of features would be added based on the mythical block pointer rewrite. Eventually BPR and thus all features requiring it were put on ice.
What specifically are you replying to? I'm not sure I recognize something in the article this applies to (but that could just be me interpreting what was presented differently than you).
> One fundamental reason is that ZFS is philosophically and practically opposed to rewriting existing data on disk; once written, it wants everything to be completely immutable
ZFS was not philosophically opposed to those ideas. They were just dropped at some point.
What's funny is now ZFS is used on SSDs, where that kind of rewriting is constantly happening behind the scenes (even if you're not writing it's constantly doing housekeeping for degraded blocks, especially on modern denser flash).
I wouldn't call it rewriting history. ZFS had plans to eventually develop the more complicated code to handle reshaping pools, but that never happened and the described system is the one still in place.
Yeah, I think Sun/Oracle recognized that there was a subset of their user base that would greatly benefit from BPR and all the associated features that come from it, but that it wasn't their core customers, so it wasn't included in the initial implementation (in my experience, most enterprise customers would rather replace an entire array that was full than expand or reshape it.)
Had Sun remained the powerhouse it was, the BPR would probably probably have eventually gotten done, but with the smaller resources devoted to ZFS development now vs in the Sun days, it's gotten kicked way down the priority list.
Enterprise users are most often using enterprise servers with fancy built in disk enclosures, and they're likely to fill those up with disks. Expanding the storage by replacing with higher capacity disks makes sense; expanding by using more disks isn't as likely. And changing the configuration to have fewer disks is also less likely; you paid for all those slots, so they're going to be filled.
> as you have less redundancy during the expansion.
If the old device is not faulty then the replace operation will keep both old and new in the array until it is complete.
> An enterprise ought to just migrate the entire pool/content to somewhere else and redo it completely.
That's also an option with send/receive, but that involves downtime. Drive replacements can be done on a live system. Server grade hardware generally supports hotplug.
Or growing by adding a whole additional DAS full of disks, and yes... zfs supports adding vdevs to pools. So if you have a 6disk raidz2, you can append another 6disk raidz2 to it to "grow" your pool. More commonly though, is probably adding new raid1 pairs to a raid10.
This was true when a tray of disk was 10s of thousands of dollars. In the world of 32TB SSDs that are tens of thousands of dollars for a single disk, not so much. As drives have gotten larger, the need to add only a handful of drives to an existing RAID set has become a desirable feature even for enterprises.
One thing I like about btrfs is offline deduplication. So whenever you want you can search for duplications and then tell the filesystem about what you find. This avoids many of the performance and ram impacts of ZFS deduplication.
Yes. For example when you're using snapshots but download new versions of something on the live volume then most installer software just overwrites the old files with nearly or fully identical new files. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots.
Lots of software is not as CoW-friendly as it could be, offline dedup fixes that with relatively low overhead.
This has nothing to do with blocks or files. The point is that you can do that only when it's needed (not often) without paying the price (ZFS dedup tables) all the time.
You can do it when needed by file on any filesystem, fdupes is a common program used for the process. The only advantage btrfs has is searching for duplicate blocks rather than files. In the few cases where that saves a meaningful amount of space, ZFS's live deduplication would probably be worth the RAM.
>fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions
Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done.
>fdupes will not let you reconcile duplicates in snapshots on ZFS
Why not? The files are kept in a hidden folder in the base directory.
>ZFS deduplication is block-based
It is also live. You don't gain Mich with offline block deduplication over file, though it is better. Live deduplication is a feature even if unnecessary for most.
> Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done.
For files that are used by anything, they might be written to. I can't have a write to one file appear in a completely different file just because they had the same contents at one point. And what if my duplicate files have different owners or permissions?
> Why not? The files are kept in a hidden folder in the base directory.
Those are read-only views, not real folders.
> You don't gain [much] with offline block deduplication over file, though it is better.
In general, perhaps not. But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments.
>I can't have a write to one file appear in a completely different file just because they had the same contents at one point.
The act of saving should destroy the link, though I know that isn't always the case. Still, this is a benefit of a CoW filesystem, not offline dedupe. Permissions may be a problem, I'm not great with them.
>Those are read-only views, not real folders.
Then mount them writable,
>But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments
Live deduplication is valuable in some instances, otherwise there are still offline alternatives available.
> this is a benefit of a CoW filesystem, not offline dedupe
It's one method of gaining CoW. And ZFS does not support CoW between files. But whatever feature you lump it under, it's an advantage Btrfs has.
> Then mount them writable,
You can't.[1] And even if you could, you can't make a hard link between subvolumes.
[1] You can 'clone' a snapshot to make a writable subvolume, but the original read-only snapshot can never be deleted while the clone exists, so this doesn't help you fix duplicates.
Hard links are essentially the filename that points to an inode. Creating a hard link creates just another filename associated with the same inode. Once a hard link, always a hard link, until destroyed.
Reflink has its own fs metadata including inode, with (initially) shared extents. Those shared extents can have their blocks individually and independently modified, per file. The point at which there are no more shared blocks, they're not reflinks.
IIUC: If you hard link two files, changing one file changes the other. If you dedupe two blocks, it will copy-on-write, meaning that changing one block does not change the other.
It's not an intrinsic property of file-dedupe vs block-dedupe. It's just how it's conventionally done.
hardlinks are paths pointing to the same inode - the same metadata, the same contents - they are the same file - just happen to have multiple addresses.
Inodes that happen to share blocks are not the same file. ie totally different.
I'm not sure what you're comparing to. Both filesystems do deduplication at a block level. But Btrfs can do reflink copies and ZFS can't. And Btrfs can do after-the-fact deduplication and ZFS can't.
I'm not sure the saving is much bigger. I would expect that in most cases if the two files differ most of the blocks will be different (e.g. add a byte to a file at the beginning, now no block is the same).
I think that block-level deduplication has more predictable performance though: for example if you write some blocks to a deduplicated file, with block-level deduplication you just write the new blocks, with file-level deduplication you have to rewrite the whole file.
Man, I wish zfs had this. Or a periodic scan, maybe run with a flag as part of, or after, a scrub.
The current implementation of dedup in zfs was the biggest disappointment I had in setting up my current storage box. Even with plenty of horsepower, I quickly turned it off and accepted the net reduction in capacity.
how does offline dedup work? Does it require reading all file data to locate duplicates? That sounds expensive to do anywhere except nearly idle systems.
Any filesystem with data checksumming should only need to build an index of the checksums to find duplicates, which sounds far cheaper. Does btrfs do that?
Well it's up to you. Generally you can look for duplicated blocks OR files, for any subset. So you could look at only files over 1 month. Or maybe any file over 2GB you could look for block level deduplication and for anything under 2GB you could look for file level deduplication.
The utilities can be written in any language, there's several popular ones called dupremove, bedup, btrfs-dedupe, bees, and dduper.
Bedup specifically mentions "It integrates deeply with btrfs so that scans are incremental and low-impact."
For my use I'd likely ignore all files less than a few weeks old for deduplication purposes. Any virtual machine images I'd use block level deduplication and everything else I'd use file level deduplication.
Doesn't seem much different overhead wise than the normal scrub that linux software RAID, ZFS, and btrfs does anyways. I'm all for not storing old files twice, but would like to avoid the overhead of checking for deduplication on every write.
If they did, the dedup process for a multi tb drive would just be a few seconds, and the overhead would be so small it could be run on the whole drive every minute all day long.
Btrfs uses data and metadata checksums, those pre-existing checksums allow for checking for duplications without having to read the underlying files. So sounds like we are violently agreeing.
Yes you can deduplicate 1TB quickly if you already have the data and metadata data checksums.
ZFS (with dedupe on) checks the checksum of every written block against every existing block in the pool, thus the slow writes and being very ram intensive.
Btrfs instead calculates data and metadata checksums for integrity then allows using them afterwards for deduplication. You could of course check for block level checksums for 100% of your storage. Or you could decide to dedupe only old files, only VMs, or only non-VMs. Or even a hybrid of block level for VMs, but file level for non-VMs.
Is the issue with ZFS deduplication really due to the fact that it is done online ? From what I understand the way that ZFS works, the amount of ram used to keep deduplication metadata would be necessary even if the deduplication was done offline, i.e. it is necessary also for read operations, not only for write operations.
I believe the ZFS deduplication consumes a ton of ram (somewhere around 1GB of ram per TB of disk) to keep the block checksums in ram. Said list is checked for every block write. I don't believe it's necessary to check for reads, since the file metadata already tells you which blocks to look for.
As a result ZFS with deduplication on uses a ton of ram and has slow writes. This is so big an impact that most folks just turn it off, despite the tempting lure of "free" disk space.
There was some talk about improving ZFS deduplication considerably by not trying to be perfect. That is, try to deduplicate against a subset of the data, not against all the data.
This allows the dedup table (which stores the hashes of existing data to find identical blocks) to be limited in size to whatever you want, and especially prevent it from eating all the RAM. The tradeoff is that the smaller it is, the deduplication hit rate drops.
Here's a talk by Matt Ahrens about it, which also includes some other tricks to improve dedup performance:
What I like about ext4 and xfs is that they are relatively simple and have a clearly defined set of capabilities that anyone can understand (within reason). They are not volume managers or anything else fancy (I have LVM/md etc for that). However, if I do have a machine that is not housed on a hypervisor with a SAN or other decent RAID then I still go for simple.
I do use zfs in preference to ufs on my pfSense boxes running non RAID on mSATA type storage but that is not a scientific choice but based on some quite iffy hearsay.
I have my opinion and you have yours. When it comes to filesystems then I want simple (and good backups.) Fast is nice but reliable is better. If I want faster I buy a bigger RAID controller or whatever.
Unfortunately ext4 and xfs are fast, but not reliable. They don't have data-checksumming, so it's very hard to know when you need to go to your backups.
Their backup options are drastically more complex because there's no way to take an atomic snapshot.
The complexity of data-integrity and backups are pushed a layer up, whereas more complex filesystems manage data-integrity and atomic snapshotting themselves, and can do so more easily due to existing at the correct layer.
As far as I know it is a fundamental design difference between btrfs and ZFS. Btrfs has extent back references. Given a location on disk btrfs has an efficient data structure that gets from the disk location to the items using that disk location. E.g. if a sector on disk gets corrupted it has a command that lets you list all the the files using that sector. With ZFS you'd have to iterate through all files, list block locations and check if one of them is affected. Of course such an additional index comes with a performance cost for managing that index (any kind of write or delete operation plus more metadata space used). On the top of my head this is used for rebalance, quota groups, device removal, send/receive (finding reflinks) and file system shrinking.
For a similar trade off compare hard links on Windows vs. ext4 and co. on Linux. On Linux there is no command to list all hard links to a file while Windows (NTFS) has one (FindFirstFileName etc.). This is why creating hard links on Windows is much slower and there is a limit of 1024 hard links to a file. To find all hard links to a file on Linux one would need to iterate over all files and check if they have the same inode number, however.
I thought most *nix filesystems were like how you describe ZFS (and ext2/3/4) where all the metadata is at the beginning of the disk. At least with the ext family, doesn't the metadata reference the first 10 blocks of a file (where the last two blocks are pointers to double linked lists and the last a reference to a triple linked list). ...
It's been a long time since I took that filesystems class, but I recall it's something like that?
That's about right. But with ext2/3/4 there are no snapshots and when you have a reference to a block you know that's the only reference, so it's not a problem there.
> This is why creating hard links on Windows is much slower
Are you actually sure that's why? Usually the Windows bottleneck is in the I/O subsystem overhead rather than the file system itself. Note that FindFirstFileName requires opening the file first, which is the operation that incurs a lot of overhead irrespective of what you're doing after. I read that on Linux this is avoided through the use of a cache (hash table?), which Windows doesn't have. Once the file is opened, the I/O performance on Windows and Linux is comparable.
Ok, to be correct it is one of the factors. (one extra random write per hard link is a lot). Add another extra write to the Windows change journal and you probably have 3x the random writes per hard link compared to Linux (discouting active shadow copies which can additionally double that to 6x).
That's what makes it impractical to use for anything other than very big and rare events. But on top of that you need a way to plan the changes and make them completely safe to do, in a way that doesn't collide with any other changes made to this presumably-live filesystem. And you have to store tons of data from the scrub somewhere to use during this process.
> That's what makes it impractical to use for anything other than very big and rare events.
Which is totally fine. Growing an array is a big and rare event. I'd be totally fine if I have to take the array offline or read-only for the duration. I just want the option to do so at all.
bcachefs is looking promising to me. I hope that it doesn't unlearn any of the lessons we've learned over the years from ZFS, which is itself excellent in many ways.
The indirection that exists in Btrfs is the internal virtual address space and block groups. Metadata and data extents are referenced in virtual addresses, bytes. That address is translated by the chunk (block group) tree into two things: a physical device, and location on that device. Conceptually it's not altogether different from ext4 on LVM, and hence the old critique that Btrfs is a layering violation. But these things are integrated on Btrfs, and the chunk is what makes so many things flexible, as well as a PITA when it comes to ENOSPC (mostly solved these days).
> One fundamental reason is that ZFS is philosophically and practically opposed to rewriting existing data on disk; ...
That's not quite the issue.
The issue is that PHYSICAL and HASH block addresses are stored together, intermingled, which means that the PHYSICAL addresses get hashed into the copy-on-write tree, which means any block can't be relocated (i.e., have its physical address(es) changed) without rewriting the path to the block containing a pointer to it as well.
I.e., ZFS is not content-addressed storage. In CAS the hash of a block is its pointer, and the physical addresses are stored separately (not intermingled with hashed data) and not hashed into the tree.
IF instead ZFS had used ONLY the hash value as the blkptr_t, and then had stored the physical addresses at the end of metadata blocks and then NOT hashed the physical addresses into the tree, THEN relocating would be a lot easier. Not trivial, mind you, just easier. Not trivial because of snapshots and dedup: you can find a singular pointer to any data block in the absence of snapshots and dedup, and rewrite the physical address stored nearby, but you can't easily find duplicate pointers, so to relocate blocks in CAS w/ snapshots and dedup you have to leave behind forwardings, or have a database of relocations (which is costly).
> (In the grand tradition of computer science we can sort of solve this problem with a layer of indirection, where the top layer stays immutable but the bottom layer mutates. This is awkward and doesn't entirely satisfy either side, and is in fact how ZFS's relatively new pool shrinking works.)
Yes, and proper CAS also has a layer of indirection, and some awkwardness around snapshots and dedup (see above). For mirror drive evacuation there's nothing to do. For shrinking operations you only need to relocate blocks whose physical addresses are beyond the end of the new device (but you'll need a fair bit of swing storage for the interim relocation or to use as forwardings until you've found and fixed all pointers to any one relocated block).
I've said this many times in the past, and always someone complains that not hashing the physical addresses is unsafe, as if the hash of the contents is not safe but hash of contents and physical address could somehow be so much safer (it can't be much safer, if at all). And again, see CAS.
Even before ZFS I wanted there to exist a RAID array where you could just swap out a disk to increase capacity.
I wanted it in part for me, but I also wanted it for all the people in my life who were doing a terrible job of taking care of their important data.
“Don’t worry,” I wanted to tell them. “Get one of these, slap some drives in it. When you need some more space, buy a new drive, take the smallest one out and put the new one in.”
Some of the early PR for ZFS was unclear on whether that was possible, but I was excited. This was by far the closest we’d gotten. Later it was a feature we’d have soon. I waited to adopt. Apple was going to support it, here we come. Apple was not going to support it. Okay, NAS could still happen. Sun craters and I lose track, but check in every so often, and still nothing.
In the grand scheme of things, it is not the biggest tech disappointment I’ve suffered. But it feels like it should be in the top ten, even if it’s #10.
142 comments
[ 4.0 ms ] story [ 74.6 ms ] threadhttp://open-zfs.org/w/images/6/68/RAIDZ_Expansion_v2.pdf
https://www.youtube.com/watch?v=Njt82e_3qVo
https://github.com/zfsonlinux/zfs/pull/8853
If you want an array of disks with a mutable configuration, I think the right choice is mdadm and LVM. They're reliable, relatively easy to use, let you move disks between machines, don't rely on hardware RAID, and support all file systems.
If you need something faster and more optimized like ZFS or btrfs, you're probably better off spending on SSDs anyway.
Tell me why I'm wrong, HN :)
You have to manually set up dm-integrity on the disks, however - it is not used by default when creating an mdadm/lvm setup.
But you can start hacking together a lot of the functionality the more dm modules you start using.
In other words zfs and btrfs incur less overhead by having the raid, checksumming and journalling layers talk to each other. Implementing them separately in as in device-mapper comes with inefficiencies.
I make backups regularly (manually, because I want them to be offline), but I'd rather not have to use them if one of my drives die. Unless I literally just finished making the backup, I'm going to lose some amount of work.
My mirrored ZFS pool means that (A) a drive can die and I won't lose _any_ data, no matter how recent, and (B) I'm confident none of the data on either drive is corrupted, because ZFS checks it regularly (every time it's accessed, and during weekly scrubs).
I do use SSDs, where I need speed - L2ARC ;), far better use than buying 19Tb of SSDs, not to mention the price.
Anyway, I do use ZFS on laptop for linux root. Reason? Snapshots. Kvm + ZFS, perfect match. /.zfs/snapshots? Fabulous.
https://www.znapzend.org/
Filesystems like BtrFS and ZFS know what is being used and what is not and won't waste time reading or writing to blocks that don't hold any currently used data.
That's taken care of by TRIM/DISCARD, which all modern filesystems support.
It's like you're saying, why do we need ECC when there is fast DDR4 memory?
ZFS is a filesystem not just a layer like LVM. You have to compare it to LVM + some filesystem. Which filesystem would you suggest using with LVM to get ZFS-level integrity?
(Technically you can use ZFS on top of LVM but that would be a weird thing to do.)
With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing? If the concerns about "data integrity" are really about time until a disk is replaced, then it makes sense. Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
Nothing in your setup there protects you from bitrot.
>With RAID6 I can have 2 drives fail without data loss. It does take several days to rebuild when a drive dies, so maybe that's the ZFS feature I'm missing?
Drive rebuild time is primarily a function of a variety of things - speed of the disks, processing power involved, etc. There's nothing magic about any solution here. ZFS does get a leg up in that it knows what blocks are in use, but hardware raid and mdadm are quite fast when they find empty blocks too, so it's not some overwhelmingly massive improvement.
>Otherwise, what's the data integrity concern? Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
I think you're fundamentally confused about what journaling provides a filesystem. A journal helps you in situations where your system crashes, loses power, etc. and you had not finished committing your changes to the filesystem. It does not protect you from data integrity issues that are unrelated to this sort of situation - a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal.
Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Is the difference that ZFS scrubs in real-time?
Sorry if I'm being dense here, but I did a fair amount of reading about this, and I'm trying to understand what I'm missing.
Nope. Parity bits aren't used to detect bitrot. If you get a URE when you go to read, the parity bits will be used to get you data. If you read successfully, nothing happens at all. If you scrub, and the data is found to be inconsistent, md assumes that the parity is wrong and overwrites it.
https://linux.die.net/man/4/md
>If a read error is detected during this process, the normal read-error handling causes correct data to be found from other devices and to be written back to the faulty device. In many case this will effectively fix the bad block.
>If all blocks read successfully but are found to not be consistent, then this is regarded as a mismatch.
>If check was used, then no action is taken to handle the mismatch, it is simply recorded. If repair was used, then a mismatch will be repaired in the same way that resync repairs arrays. For RAID5/RAID6 new parity blocks are written. For RAID1/RAID10, all but one block are overwritten with the content of that one block.
Is it correct to say that ZFS parity is used to reconcile on each read even without UREs, but with md parity is only used when there are UREs? If that's true, it makes much more sense why ZFS is considered the right choice.
Also, do you know if adding luks in the mix changes that?
My data goes md->lvm->luks->ext4 and I hadn't considered which part of that is supposed to catch the error if it wasn't a URE.
It makes sense that md would update parity if there was no URE. It also makes sense that LVM wouldn't know or care about data integrity.
I would think luks would require data integrity to work properly, but I'm not sure what happens if there is bitrot in a luks block.
I'm also surprised ext4 doen't have checksumming, but apparently it's only for able to checksum for metadata and it's disabled by default: https://serverfault.com/questions/456319/how-to-get-checksum...
Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence.
>Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence.
That wiki page is pretty yikes - I was under the impression dm-integrity was farther along.
Some misc other stuff I like about ZFS:
l2arc/zil on ssd/optane devices for fast read/write caching Easy NFS/SMB sharing zdb ( https://manpages.debian.org/testing/zfsutils-linux/zdb.8.en.... )
I don't think you would be disappointed if you checked it out.
(If I'd been scrubbing regularly, these almost certain would have been found before corruption crept into the mirrored copy of the file, and been recoverable.)
Btw, journaling has nothing to do with integrity, though I believe it is checksummed these days.
I thought this was what RAID was designed to prevent. I don't know what a cable failure would look like in my setup in practice, but I would think the disk would start failing parity checks against the rest of the array and be flagged as bad.
Using more spindles you could get more performance out of a bunch of cheap disks, and to help with the fact that these cheap disks failed more often, you made them redundant.
In general, dealing with data corruption that does not generate an error is not within the purview of RAID implementations. ZFS and btrfs handle it because they're more than just RAID implementations. mdadm can do similar things in a RAID setup if dm-integrity is used on all of the drives (but it is not used by default when just setting up mdadm/lvm/etc.)
ZFS is really designed for end-to-end integrity. Once you're stored data on ZFS, you're basically guaranteed that you'll always get the exact same thing back (or an error)... no matter where in the "chain" of I/O the failure occurred.
50MB isn't much compared with 13PB, but silent corruption is a real thing.
Obviously I moved off the RAID6 soon after, but this is a powerful example of why the checksuming of ZFS is so useful.
A RAID system can only reliably recover when an IO error is reported. In this case, it can assume which element of the stored data is no longer trustworthy and use the other parts to reconstruct it. A RAID scrub operation that finds inconsistencies, without a reported IO error, cannot really determine why it is inconsistent. It doesn't know which element is corrupt and which other elements to use to reconstruct it.
In effect, ZFS adds another data integrity layer over the physical storage, so you can have a lower probability of silent failures where the disk controller or IO channels do their own "parity" or other checks and report a successful read when they actually returned corrupt values.
Something like mdadm or LVM would need to add a new layer of checksums over the physical devices to provide this level of protection. Each block/chunk/stripe written to disk would need another integrity checksum written as well. During operation, these checksums would be verified before deciding that a read operation was actually successful. If the verification fails, a virtual IO error is synthesized to flag the read as bad, so that the other RAID recovery techniques can then be applied while knowing which block/chunk/stripe is invalid.
If you don't have checksums on your data, how do you know it's good (even without drive failures)? Your RAID rebuild could be recreating junk.
> Ext4 is a journaled file system, which I thought was the equivalent of ECC for filesystems.
Journaling helps with speeding up post-crash reboots: instead of doing a full fsck, it just replays the log.
From experience i can tell this can happen. nervous trauma twitch
But what if you have silent data corruption: a single bit? Even if you have parity (raid5) or a duplicate (raid1), how do you know which part is correct, and which part to repair? Without checksums.
I've had to restore 1TB from Crashplan. They apparently had checksums in their software, and something like 10-15 files were damaged.
Without checksums, you simply don't know if your data is damaged, even on a single hard drive. ZFS gives you data integrity with checksums, and redundancy with mirrors or raid-z.
This is why a regular scrub is recommended, you verify the checksums and repair if possible.
He then reads the file, uncorrupted, with a kernel warning about a bad checksum that was fixed (from the mirror) during the read.
This demonstration is why I switched to btrfs (and ECC) for my new home server/NAS.
That, along with compression sealed the deal. I was disappointed about not having dedupe at the time, but it probably really wouldn't have saved me that much space. (I keep meaning to run the dedupe tool to see what it might save me now.)
This sort of gets into it but isn't a fantastic source: https://www.enterprisestorageforum.com/storage-hardware/driv...
See also https://en.wikipedia.org/wiki/Data_corruption#Silent and https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHAND... .
Notably, Apple's new filesystem introduced in 2017 does not detect data corruption; it only checksums metadata. https://en.wikipedia.org/wiki/Apple_File_System#Data_integri... Apparently Apple Engineers are of the opinion that Apple SSD and HDD media simply don't have silent data corruption. They have some pretty smart engineers, so maybe they're on to something. It does beggar the question, though: why have metadata checksums at all? http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/
Anyway, I'm not exactly supporting threat models that ignore silent data corruption. Just pointing out that they're not exactly as ridiculous as they might seem on their face.
Linux md will do the completely wrong thing with this: it will treat that as a drive error, and if you accumulate a lot in a short time (such as rebuilding an arrary) then it will conclude that this is a failed drive and kick it out of the array.
How many array rebuilds have you done? You only see the read errors when you go to actually read the sector in question, and even in a week I wouldn't want to rely on not having any sectors go bad.
As a related anecdote, I did some stress-testing on a ZFS filesystem with two Kingston NVMe SSD-s in mirrored configuration and at one point I got two random read errors on one drive, which subsequently got corrected by ZFS.
Silent failures might not be the best phrase to use for these situations, but ZFS does let you know that corruption has taken place and you have to acknowledge that message for it to disappear.
It's not clear to me if you're describing silent data corruption or detected data corruption (at the controller level), but either way Apple isn't selling Kingston NVMe SSDs; they have their own SSD controller and vendor the raw flash.
The data integrity problem is you don't have a way of unambiguously determining data integrity. You have to trust the hardware. Obviously that's good enough most of the time, but it can transiently misstep without outright failing: bit flips, torn or misdirected writes.
By the way, make sure you check for timeout mismatches if you're using consumer drives. Affects Btrfs, mdadm and LVM raid, and I assume ZFS on Linux. https://raid.wiki.kernel.org/index.php/Timeout_Mismatch
File systems like zfs and btrfs checksum every block to ensure that if they do become corrupt, they can get repaired. (otherwise, if you had a mirror of disks, and one disk says the value is 0010 and the other says its 0001, wich one is right?)
Additionally, the 'raid' concept is not just about speed. Its also about capacity, and how many disks can fail. I might have 48 drives in a single server. I certainly don't want an LVM stripe across all drives and lose data. I also don't want raid1, losing the capacity of 24 of the drives. I want it so that any 3 drives can fail in that system, and I can still recover the data.
It certainly made me reconsider my ZFS practices.
ZFS isn't really a filesystem. It's more like a storage backend, that happens to support the POSIX interface. The original intent of ZFS was to completely rethink the volume management/filesystem layering nonsense, and to provide for storage what the MMU/VM system provides for memory.
So it can do much more interesting things like memory backed read caching (via ARC), SSD backed read caching (via L2ARC), and SSD backed write buffering (via ZIL/SLOG), as well as the normal RAID stuff.
Granted I was a much nooblier noob when using mdadm.
https://raid.wiki.kernel.org/index.php/Growing
It should be possible to buy disks of any size, add them to the computer and tell the file system to use them. Drive failures shouldn't increase the risk of additional failures.
ZFS was not philosophically opposed to those ideas. They were just dropped at some point.
Had Sun remained the powerhouse it was, the BPR would probably probably have eventually gotten done, but with the smaller resources devoted to ZFS development now vs in the Sun days, it's gotten kicked way down the priority list.
That is an insanely time consuming operation that also increases the risk of data loss as you have less redundancy during the expansion.
An enterprise ought to just migrate the entire pool/content to somewhere else and redo it completely.
Expanding by replacing with higher capacity drives just feels like a hack and afterthought that only hobbyists would actually make use of.
If the old device is not faulty then the replace operation will keep both old and new in the array until it is complete.
> An enterprise ought to just migrate the entire pool/content to somewhere else and redo it completely.
That's also an option with send/receive, but that involves downtime. Drive replacements can be done on a live system. Server grade hardware generally supports hotplug.
Which you probably should.
Lots of software is not as CoW-friendly as it could be, offline dedup fixes that with relatively low overhead.
Running a dedupe based on file rather than block would do the same thing. You don't gain much if anything in your example using btrfs.
What is being argued here?
* fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions
* fdupes will not let you reconcile duplicates in snapshots on ZFS
* ZFS deduplication is block-based
Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done.
>fdupes will not let you reconcile duplicates in snapshots on ZFS
Why not? The files are kept in a hidden folder in the base directory.
>ZFS deduplication is block-based
It is also live. You don't gain Mich with offline block deduplication over file, though it is better. Live deduplication is a feature even if unnecessary for most.
For files that are used by anything, they might be written to. I can't have a write to one file appear in a completely different file just because they had the same contents at one point. And what if my duplicate files have different owners or permissions?
> Why not? The files are kept in a hidden folder in the base directory.
Those are read-only views, not real folders.
> You don't gain [much] with offline block deduplication over file, though it is better.
In general, perhaps not. But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments.
The act of saving should destroy the link, though I know that isn't always the case. Still, this is a benefit of a CoW filesystem, not offline dedupe. Permissions may be a problem, I'm not great with them.
>Those are read-only views, not real folders.
Then mount them writable,
>But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments
Live deduplication is valuable in some instances, otherwise there are still offline alternatives available.
It's one method of gaining CoW. And ZFS does not support CoW between files. But whatever feature you lump it under, it's an advantage Btrfs has.
> Then mount them writable,
You can't.[1] And even if you could, you can't make a hard link between subvolumes.
[1] You can 'clone' a snapshot to make a writable subvolume, but the original read-only snapshot can never be deleted while the clone exists, so this doesn't help you fix duplicates.
That's bullshit. hard links are the same exact inode, with all that entails. reflinking is at the data block level. Totally different.
Reflink has its own fs metadata including inode, with (initially) shared extents. Those shared extents can have their blocks individually and independently modified, per file. The point at which there are no more shared blocks, they're not reflinks.
It's not an intrinsic property of file-dedupe vs block-dedupe. It's just how it's conventionally done.
Inodes that happen to share blocks are not the same file. ie totally different.
Of course this only works if the data is read-only.
It won't reclaim any space on ZFS if you're using snapshots.
I think that block-level deduplication has more predictable performance though: for example if you write some blocks to a deduplicated file, with block-level deduplication you just write the new blocks, with file-level deduplication you have to rewrite the whole file.
The current implementation of dedup in zfs was the biggest disappointment I had in setting up my current storage box. Even with plenty of horsepower, I quickly turned it off and accepted the net reduction in capacity.
[1] I put the wrong command here originally.
Or actually detects duplicate blocks? Because that would be still be something, knowing how much deduplication would benefit you, if implemented.
Any filesystem with data checksumming should only need to build an index of the checksums to find duplicates, which sounds far cheaper. Does btrfs do that?
The utilities can be written in any language, there's several popular ones called dupremove, bedup, btrfs-dedupe, bees, and dduper.
Bedup specifically mentions "It integrates deeply with btrfs so that scans are incremental and low-impact."
For my use I'd likely ignore all files less than a few weeks old for deduplication purposes. Any virtual machine images I'd use block level deduplication and everything else I'd use file level deduplication.
Doesn't seem much different overhead wise than the normal scrub that linux software RAID, ZFS, and btrfs does anyways. I'm all for not storing old files twice, but would like to avoid the overhead of checking for deduplication on every write.
If they did, the dedup process for a multi tb drive would just be a few seconds, and the overhead would be so small it could be run on the whole drive every minute all day long.
Yes you can deduplicate 1TB quickly if you already have the data and metadata data checksums.
ZFS (with dedupe on) checks the checksum of every written block against every existing block in the pool, thus the slow writes and being very ram intensive.
Btrfs instead calculates data and metadata checksums for integrity then allows using them afterwards for deduplication. You could of course check for block level checksums for 100% of your storage. Or you could decide to dedupe only old files, only VMs, or only non-VMs. Or even a hybrid of block level for VMs, but file level for non-VMs.
As a result ZFS with deduplication on uses a ton of ram and has slow writes. This is so big an impact that most folks just turn it off, despite the tempting lure of "free" disk space.
This allows the dedup table (which stores the hashes of existing data to find identical blocks) to be limited in size to whatever you want, and especially prevent it from eating all the RAM. The tradeoff is that the smaller it is, the deduplication hit rate drops.
Here's a talk by Matt Ahrens about it, which also includes some other tricks to improve dedup performance:
https://www.youtube.com/watch?v=PYxFDBgxFS8
http://open-zfs.org/w/images/8/8d/ZFS_dedup.pdf
I do use zfs in preference to ufs on my pfSense boxes running non RAID on mSATA type storage but that is not a scientific choice but based on some quite iffy hearsay.
I have my opinion and you have yours. When it comes to filesystems then I want simple (and good backups.) Fast is nice but reliable is better. If I want faster I buy a bigger RAID controller or whatever.
Their backup options are drastically more complex because there's no way to take an atomic snapshot.
The complexity of data-integrity and backups are pushed a layer up, whereas more complex filesystems manage data-integrity and atomic snapshotting themselves, and can do so more easily due to existing at the correct layer.
For a similar trade off compare hard links on Windows vs. ext4 and co. on Linux. On Linux there is no command to list all hard links to a file while Windows (NTFS) has one (FindFirstFileName etc.). This is why creating hard links on Windows is much slower and there is a limit of 1024 hard links to a file. To find all hard links to a file on Linux one would need to iterate over all files and check if they have the same inode number, however.
It's been a long time since I took that filesystems class, but I recall it's something like that?
https://en.wikipedia.org/wiki/Inode_pointer_structure has a summary
https://docs.microsoft.com/en-us/archive/blogs/askcore/the-f... talks about the NTFS equivalent
Are you actually sure that's why? Usually the Windows bottleneck is in the I/O subsystem overhead rather than the file system itself. Note that FindFirstFileName requires opening the file first, which is the operation that incurs a lot of overhead irrespective of what you're doing after. I read that on Linux this is avoided through the use of a cache (hash table?), which Windows doesn't have. Once the file is opened, the I/O performance on Windows and Linux is comparable.
I mean... you just described a scrub. I'd gladly sit through a scrub if it meant that I could grow a raidz vdev.
I'm suspecting there's some other hurdle.
Which is totally fine. Growing an array is a big and rare event. I'd be totally fine if I have to take the array offline or read-only for the duration. I just want the option to do so at all.
https://m.youtube.com/watch?v=ZF8V7Tc9G28
https://youtu.be/Njt82e_3qVo
https://github.com/zfsonlinux/zfs/pull/8853
That's not quite the issue.
The issue is that PHYSICAL and HASH block addresses are stored together, intermingled, which means that the PHYSICAL addresses get hashed into the copy-on-write tree, which means any block can't be relocated (i.e., have its physical address(es) changed) without rewriting the path to the block containing a pointer to it as well.
I.e., ZFS is not content-addressed storage. In CAS the hash of a block is its pointer, and the physical addresses are stored separately (not intermingled with hashed data) and not hashed into the tree.
IF instead ZFS had used ONLY the hash value as the blkptr_t, and then had stored the physical addresses at the end of metadata blocks and then NOT hashed the physical addresses into the tree, THEN relocating would be a lot easier. Not trivial, mind you, just easier. Not trivial because of snapshots and dedup: you can find a singular pointer to any data block in the absence of snapshots and dedup, and rewrite the physical address stored nearby, but you can't easily find duplicate pointers, so to relocate blocks in CAS w/ snapshots and dedup you have to leave behind forwardings, or have a database of relocations (which is costly).
> (In the grand tradition of computer science we can sort of solve this problem with a layer of indirection, where the top layer stays immutable but the bottom layer mutates. This is awkward and doesn't entirely satisfy either side, and is in fact how ZFS's relatively new pool shrinking works.)
Yes, and proper CAS also has a layer of indirection, and some awkwardness around snapshots and dedup (see above). For mirror drive evacuation there's nothing to do. For shrinking operations you only need to relocate blocks whose physical addresses are beyond the end of the new device (but you'll need a fair bit of swing storage for the interim relocation or to use as forwardings until you've found and fixed all pointers to any one relocated block).
I've said this many times in the past, and always someone complains that not hashing the physical addresses is unsafe, as if the hash of the contents is not safe but hash of contents and physical address could somehow be so much safer (it can't be much safer, if at all). And again, see CAS.
I wanted it in part for me, but I also wanted it for all the people in my life who were doing a terrible job of taking care of their important data.
“Don’t worry,” I wanted to tell them. “Get one of these, slap some drives in it. When you need some more space, buy a new drive, take the smallest one out and put the new one in.”
Some of the early PR for ZFS was unclear on whether that was possible, but I was excited. This was by far the closest we’d gotten. Later it was a feature we’d have soon. I waited to adopt. Apple was going to support it, here we come. Apple was not going to support it. Okay, NAS could still happen. Sun craters and I lose track, but check in every so often, and still nothing.
In the grand scheme of things, it is not the biggest tech disappointment I’ve suffered. But it feels like it should be in the top ten, even if it’s #10.