Is ZFS worth considering at this point? Btrfs seems to be quite featureful and stable, with a plethora of grocers and retailers all over the world using it. Wal-Mart, Kroger (and subsidiaries), all of IBM's old point of sale customers (now Toshiba SuperPOS iirc) all use it on things as small as deli scales all the way to running their point of sale systems and backends.
That's by far from the only failure case to worry about. In addition to random dataloss, the most frequent failure cause I dealt with was complete unbalancing making the filesystem read-only. This happened every ~48 hours of continuous use! I've not read that this is fixed yet, but I haven't used it in the recent past due to this making it unsuitable for production use.
I've used it for backups for five years now. My USB backup drives have gotten full quite a lot of times because of old snapshots (rsync isn't able to detect moving directories), and I've had zero problems with it – just have to delete some old snapshots and wait a bit and then there's space again.
I don't use RAID5, which is where I've heard btrfs is dangerous, just for the snapshots and dedupe (with https://github.com/jbruchon/jdupes/ )
It was used at my old job on a 1TB backup drive, containing lots of hourly snapshots from around 100 other machines that also had btrfs on it (btrfs send). The snapshots were also frequently thinned (i.e. only a few of the older ones kept) to make room for new ones. The filesystem broke at least two times in two years when it was nearly full. None of the fixes we found with web searches worked. We went with formatting and copying the backups from another location, losing only a little bit of data that was not too important. That was around 2015-2017, and the version of Btrfs was about what's in Debian jessie.
Has Facebook ever made it completely clear what they use btrfs for? I believe it was in one of the episodes of bsdnow where they talked about it "only" being used for administrative systems.
I think that's a couple of years old. If you want more modern info, I think https://www.youtube.com/watch?v=ME4RhxwPO5A (skip to about 8m30) is pretty good, as is https://code.fb.com/open-source/linux/. We also talk a lot about the benefits of having cgroup2 work well with BTRFS, which (imo) isn't something that particularly matters with administrative systems, but more for data processing/serving systems. I don't think we've released numbers or percentages, but it's safe to say every request you make to Facebook.com is processed by 1 or more machines with a btrfs filesystem. We've got a ton of value out of it, I hope we talk more about its successes publicly soon :)
That video's discussing btrfs layering, which is very neat.
I am _very_ curious what sort of experience FB has had with btrfs.
If FB has decided to go all-in with btrfs, you probably have the most accurate raw data there is to have.
Of course, any hyper-scale deployment of a technology is going to produce remarkable/exponential statistics, and these will (sadly/annoyingly) need careful parsing to normalize in a way that is generally accessible and unsurprising. (We live in a very knee-jerk world, and all. Sigh)
All this to say, I at least am looking forward to whatever sorts of numbers you end up being able to share.
--
After considering the bit about btrfs in the video, and your mention of "1 or more machines", I wonder if you aren't using btrfs on things like load balancer type systems - or, more abstractly, node configurations that can always safely be "thrown away", whether by hardware failure or explicit decommission. But then I realize the chances are you probably use such a model (individual hardware failure must be acceptable; everything important must be n-way redundant) because nothing else makes sense at scale. And then I wonder... exactly what point in that spectrum does btrfs fit in? I wonder if/how you can answer that question.
zfs used to have a better compression algorithm (lz4) before btrfs landed zstd with kernel version 4.15. Btrfs still doesn't have stable support for RAID 5+6 https://btrfs.wiki.kernel.org/index.php/RAID56
- fsync on btrfs is extremly slow. The situation has improved in last 3 years, but it is still much slower than ZFS. I just did a simple test and BTFS is 12x slower than ZFS (on my customer SSD, for small writes).
- RAID5/6 mode is only experimental in btrfs, while it is really stable in ZFS.
- I don't have concrete data for that, but in my experience, BTRFS has high latency (>1 second) even for small file operations when under load.
- While that should not be a problem for production systems, I have some crappy hardware where BTRFS oopses or corrupts data, while other filesystems (Ext4, ZFS) work fine.
Yes the 12x slower seems about right. I remember Debian installations taking >1h while ext4 took < 10min. (Installations and package operations are particularly bad since dpkg does many fsync()s when unpacking). I think that was on a spinning drive, though. Anyway at my old job (where we had mainly spinning drives) we used apt and dpkg only with eatmydata. eatmydata is a command which uses LD_PRELOAD hackery to remove fsync() systemcalls.
This was really painful. Back when I was a Debian developer tracking unstable, I would update every evening and it would sometimes take multiple hours to complete. The fsync performance is a truly awful design flaw, and while eatmydata does make it faster, it's a terrible and dangerious hack! (Which everyone used!)
I even had to add eatmydata support to the schroot tool as a command-prefix option to allow every command to be run via eatmydata when using btrfs snapshots. (I've since dropped btrfs support entirely; it was too unreliable with large snapshot turnover rapidly unbalancing the filesystem. Unusable in production.)
I believe the correct solution to this problem would be that the installers would snapshot the system, install all packages without any fsync()ing at all, then finally one sync() and remove the snapshot. Optionally keeping snapshots if user want to roll back from a broken upgrade (for whatever reason). Again, as others have written here, btrfs is great if your software plays along with it, otherwise it might not be so great.
While this is a good idea in theory (and is possible on e.g. FreeBSD with ZFS with boot environments), dpkg can't do this easily. The main problem is that it's possible and supported for the managed files to be placed upon multiple filesystems. Separate /usr, separate /var, separate /usr/share, whatever combination you choose. This means that dpkg needs to force file synchronisation across all mounted filesystems and it can only do this robustly by issuing fsyncs.
When there's only a single filesystem, and that filesystem is btrfs (or ZFS), it should however be possible to optimise this away and delegate everything to the filesystem. But even here, maintainer scripts may issue their own fsyncs as they update their own databases, kernel images or whatever.
> dpkg needs to force file synchronisation across all mounted filesystems and it can only do this robustly by issuing fsyncs
Not if file-change notifications were supported robustly by dpkg and the kernel (to a lesser extent). Getting to that would, however, require massively restricting the compatible-kernel-versions set of dpkg, and would also probably require undoing some of the more . . . misguided pieces of history with regard to file-change notification systems in Linux.
I don't think this is correct. File-change notifications wouldn't provide any information which isn't already known. dpkg, after all, is entirely responsible for unpacking the .deb files and doing the file modifications. It's fully aware of what was written, in what order, and when.
The problem is that the system state needs checkpointing for every package state change. It must allow for recovery on failure, termination, abortion or power loss, amongst other scenarios. And the package database must remain in sync with the filesystem state.
When every managed file is on one snapshot-able filesystem, this could be rolled back atomically, and the fsyncs skipped. But as soon as you have a non-snapshot-able filesystem or multiple filesystems in use, the fsyncs can't be skipped.
fsync performance on ZFS on Linux isn't as performant as it should be either. On my Optane 900p I have almost 10x better fsync performance on FreeBSD compared to Linux.
Is Btrfs worth considering? ZFS is one of the most advanced, stable and feature-complete filesystems there is, and is cross-platform. Btrfs is best known for being flakey and in mainline.
Had there not been uncertainties about licensing which prevented ZFS from being in mainline there wouldn't be any Btrfs. How can you ever really trust a filesystem developed by adding features first, trying to add stability and usability later anyway?
I don't even really understand why ZFS being outside mainline is such an issue. On Debian Stretch, I installed ZFS like any other package, and didn't even realize the kernel was getting recompiled until I read about the licensing situation later on.
I did think it was odd that installing the ZFS package took such a long time, but that was a one-time wait.
If you're busy and can't be arsed to research past "this has been rock-stable for a decade, so should be safe to use", ZFS is the best choice out there.
If you care about data not randomly failing, btrfs is not an option.
While working in that specific industry it seems like there is a motivation to specifically go out of their way to make things unreliable, extra complicated and utilize bad technologies.
The fact that IBM and Toshiba on POS is frightening.
I'm curious that they're talking about implementing encryption at the filesystem layer. This has typically been done elsewhere. And anyway, key management is an issue. Seems like it would be beyond the scope for a filesystem. But I had similar thoughts right before I started learning about the write anywhere layout used by NetApp all those years ago, where mixing layers had enormous benefits (ZFS too).
The idea of incorporating a flash translation layer (FTL) is interesting, but there is the hardware support issue. Meaning that yes, you can still buy raw NAND flash memory, but what are you going to connect it to? NAND flash controllers which can present a useful interface to the host processor (like USB) already incorporate a FTL.
Similarly, eMMC memory has a FTL layer baked in, and presents itself as a block-addressable device.
Raw NAND flash controllers are increasingly rare on modern microcontrollers.
ZFS encryption works in a similar fashion, and ext4 also has encryption (which is used by default in Android).
Personally I'm not a huge fan of this because unlike the clear benefits of giving filesystems control over raw devices, full disk encryption has requirements that can't really be provided partially. You want to ensure no metadata about the filesystem structure or files will be known without knowing the encryption key but this is clearly not possible without having block device layer encryption. ZFS encryption allows for all sorts of useful operations on encrypted filesystems without having the key. This is cool, but also obviously provides a lot of information about the filesystem structure. Also dedup tables aren't encrypted in this setup. ext4 only encrypts filenames and contents, which is worse in some respects.
> Also dedup tables aren't encrypted in this setup.
This sounds like low-hanging fruit for a security researcher to try their hand at: comprehend the dedupe table format, and see what you can derive from it (basically the real-world version of https://xkcd.com/704/).
The only metadata stored in the clear in bcachefs is the superblock and the very first part of the btree node and journal entry headers - just the checksum, magic number (identifying it as a btree node/journal header), and a flags field which mainly contains the checksum/encryption type.
So an attacker can tell roughly how much metadata you have, but literally nothing about the contents.
> the write anywhere layout used by NetApp all those years ago
I've never heard of "write anywhere" before but it sounds interesting. What is/was this?
> ... incorporating a flash translation layer (FTL) is interesting, but ... what are you going to connect it to? NAND flash controllers ... already incorporate a FTL. ... Raw NAND flash controllers are increasingly rare on modern microcontrollers.
My read of the FTL bit is that of hopeful optimization that supports the ideal-case - say, the filesystem equivalent of all the revving-up everyone is doing to support RISC-V. It's clearly leaning pretty heavily on the bit about existing precedent. My practical hope is that the code stays in good, well-tended condition up to the point it can be usefully taken advantage of - and that someone notices it to good effect sooner rather than later.
A really good reason to use encryption at the filesystem level is that you can use stream ciphers instead of block ciphers (because you have a place to store nonces). The current disk encryption is based on XTS, which has a section in wikipedia on it's weaknesses [1].
I'm very excited for bcachefs, I've been trying it out on my personal laptop for a while (until I was forced to reinstall).
It was a very pleasant experience and I hope that it gets mainlined in the close future. Linux desperately needs ZFS with better integration and more flexibility.
It wasn't bcachefs' fault (in fact, I'm copying my laptop's install back on a bcachefs partition), it was a accumulative problem where in the end the bcachefs-enabled kernel and various tools decided to go on strike and forced me to go back on a btrfs based partition. (I booted into a post-mount environment and ran rsync)
> zfs is block based, not extent based, whereas all other modern filesystems have been extent based for years: the reason they did this is that extents plus snapshots are really hard.
then:
>Snapshots: In progress. bcachefs's snapshot implementation is going to be significantly more capable than any competing implementations. When it's done, ...
Multiple ZFS challengers can get easily to the point where they have 80-90% of the cool features that ZFS has. Then everything slows down to crawl because getting the same performance and reliability + features is super hard. When you leave the essential but hard features 'for later' they are harder to add later without big rewrite (or losing performance).
Ticking all the boxes with good performance becomes exponentially harder as number of features grow. There comes time to order things: sacrifice performance in this to attain this.
It's not about being excellent or not, or if he can implement something that technically works.
He or any other excellent guy don't have an idea how the implementation he designs works in multiple workloads and different machines and disk drives. There will be time period of several years of adjusting and tinkering and compromising.
bcachefs is specifically trying to avoid the btrfs garbage fire by building slowly on top of well-tested bcache. I am confident it won't end up like btrfs where RAID5/6 is still horribly broken yet people claim the filesystem is totally stable and fine for production use.
On that note, are there any fancy file systems that have acceptable Windows/Mac support? Or are we all doomed to using NTFS or exfat for USB drives that need to work on Linux/Mac/Windows?
> Installing a binary release
>
> Enable unsigned drivers:
> bcdedit.exe -set testsigning on
That seems only slightly less scary than the btrfs on Windows project (which suggests disabling SecureBoot). At least it's from a ZFS contributor (while btrfs on Windows seems like a from-scratch rewrite, ie. all the bugs all over again).
From the comments here I take it people aren't actually using anything with features like snapshots/cow/compression on Windows :-/
This is somewhat like downloading an open source Android app from github, and saying it's scary that you need to go into settings and enable sideloading.
Microsoft has somewhat ridiculous driver signing requirements. I become angry about this perhaps once every four months.
There are ext2/3/4 drivers for Windows and macOS, but you need to install them. If you want seamless support, your best choice is FAT32. NTFS or ExFAT are good, though not all Linuxes support them of out the box.
The problem with Ext?, NTFS, and ExFAT is that you always need a driver on one of the 'big 3'.
FAT32 "seamless support"? It doesn't even have journaling. The other day I wanted to copy a 8 GB file. Forget FAT32. Just use Ext4. It isn't patent encumbered, and there are drivers available for all of the big 3. You can boot a live CD/DVD/USB to mount the data in no time. This is easier than installing any driver, and you won't have to uninstall or maintain it afterwards.
Try UDF, it's supported by Linux, Mac and Windows natively since it's the default FS for a DVD (ie, any computer OS that understands a DVD understands UDF). It supports files bigger than 4GiB, unicode filenames and up to 2TiB volumes with 512 byte sectors. It doesn't have journaling though but it is fairly reliable even without.
> The ability to create ReFS volumes was removed in Windows 10's 2017 Fall Creators Update for all editions except Enterprise and Pro for Workstations,[4] which would seem to indicate Microsoft is no longer intending ReFS as a general replacement for NTFS, at least in the near future.
ReFS isn't meant as a mainstream FS. It's a pure storage FS that is meant to specifically work with backup applications for space saving purposes and with Hyper V. You can't even boot an OS off it, and the overhead from the block cloning technology simply cannot be compatible with normal OS usage.
Apple has APFS, which might get a decent Linux driver soon, since Apple has recently published a fairly detailed description of how it works. Theoretically, nothing prevents someone writing a Windows driver too, which would allow APFS to become the modern standard for cross-platform requirements. Only I suspect that such Windows driver won't be free or common enough to make this practical.
There is a Windows driver for APFS from Paragon available. [1] Proprietary and commercial though. They also provide a free APFS to HFS+ conversion tool. HFS+ can be read from Linux though AFAIK without journaling support (essentially HFS).
I suspected somebody like Paragon would have a driver for it, but it being commercial means the adoption rate is probably minuscule and would remain so.
I don’t get how there isn’t more demand for this, to me it’s such an obvious pain-point for personal computing these days. I guess the number of people dual booting OSes or even using external drives at all these days is so small? I feel like thumb drives are still very common though and getting large enough that it’s worth having a decent file system on them.
I'm not sure I agree with the idea that having a single open source project to rule them all is an inherently bad thing. In proprietary software, if Banana corp says that the next version of their Cavendish OS drops x86 support, you might find yourself doomed. But if Debian decides to drop x86 tomorrow, you go "Cool, good luck, I'll just fork." The project itself is never a single point of failure.
For filesystems, it's a tough call. They are one of the most, if not the most, important parts of the system with respect to requiring absolute reliability. Being stable isn't necessarily a bad thing.
In the case of OpenZFS, where it's implemented by multiple operating systems with differing requirements, maybe the different priorities and goals of the different teams will result in cross-pollination of ideas and implementation details rather than it stagnating top-down. I.e. there's scope for some differences in features and details, while still adhering to the common design.
We need to disambiguate the word "you" here. When "you" = an IBM or Apple or Microsoft, then forking is a great option, and a great reason to pick a program with a license that allows it. When "you" = some guy sitting at home in his PJs on Sunday morning who reads that his favorite filesystem is moving in a direction he doesn't like, you're basically SOL unless you decide to devote your life to maintaining that FS now.
Maintaining an entire Debian architecture is probably not a one-person job.
But, because Debian has such a broad array of contributors, I'm extremely confident they won't drop x86 support unless/until basically everyone else in the world has already moved on, and quite possibly not even then.
Whereas Apple really could just up and drop x86 any day now, and I'll just be stuck.
ZFS is not suitable for desktop computers. as for example that one anecdote in the comments of the article shows, ZFS doesn't work well on a single disk. the author of the comment is confused by the behavior, and doesn't even consider that this might be by design.
i considered getting EEC RAM for my machine just because of ZFS, but i could not find any 16GB strips to fill my board.
that's two strikes against desktop use, and enough to let me prefer btrfs.
EDIT: i am very happy to see so many replies proving me wrong. when i last researched this topic i was not able to find any counter arguments to needing EEC.
I have been using ZFS for over two years, on any desktop that I control, as both root file system for FreeBSD setups and backup file system in many configurations. In almost all cases I don't have EEC ram and I don't need it really. Your comment is either outdated or wrong, ZFS is THE go-to filesystem on desktop for me.
Edit: oh and it works beautifully with single disk setups too.
Wait until you get a 1-bit flip in a spacemap record.
Then you just won't be able to import your pool (insta-panic). And good luck with fining and fixing that flipped bit.
Does anyone have any pointers to a resource for single disk ZFS setups? Last time I looked, most search results were entry points into a persistent flame war.
“There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error.
I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS.”
I'd say the main problem is that there isn't a fsck for ZFS. If important metadata gets damaged you'll have to re-create the whole file system.
W.r.t. to btrfs they are at least working on a fsck tool, even though it can't repair much yet (what it can repair is e.g. some bitflips caused by bad RAM: https://patchwork.kernel.org/patch/4116411/ ).
And stuff like ext4 or ntfs can get back to a state where one can write to it again in most cases.
Another problem is that many applications don't handle EIO well, i.e. they don't give you an option like "Continue with corrupt data", e.g. if a video file gets damaged it doesn't matter much, it'll just be a few damaged pixels, but the whole video player stops playback because it doesn't handle the IO error.
If there is any kind of corruption on a Btrfs file system, it will report EIO and path to the corrupt file, it will not handoff corrupt data to user space. The application can't really do that much but say something like I/O error, check kernel messages, or some such. It can't ask the kernel to continue with corrupt data, Btrfs driver has no such option.
There is this myth over the Freenas forums that ECC is a must-have for ZFS.
If you read closely the posts that are supposed to prove this claim, they never give strong evidence that the pool losses are directly linked to the lack of ECC memory.
It always ends up with cyberjoke stating that the issue was linked to lack of ECC.
Yes, I really hate that FUD. It makes people decide to use less safe filesystem because they don't have ECC RAM thinking ZFS somehow will corrupt their data, when in reality ZFS is still safe (and safer than other filesystems). ZFS claims to keep your data safe, but it can't prevent data being corrupted in RAM. If your RAM is corrupted your data might be as well no matter what filesystem you use. ECC RAM makes sure that memory is also safe and that ZFS can deliver its promises.
you could add additional checks in the code to make sure the data is correct. the impression i got was that ZFS didn't add those checks because it would trust the RAM.
HAMMER2/DragonFly is already looking great with cow/snaphots, compression, live dedup etc and one of its design goals is to work great on low memory systems. Once it is feature complete with clustering support it'll be a great choice to have.
Also the author do not mention that LLNL zfs Linux port is not as production ready than FreeBSD one.
Personally I use nilfs2 on top of lvm since is Linux mainline from many many years, it a logfs like Hammer volumes, so it protect effectively against accidental overwrite/deletion and can be live resized both in grow and shrink so coupled with LVM can be a poor man's pooled storage... But Hammer on Linux will be a success like OpenSSH!
From my reading in the past primarily because cross OS portability was not a design goal for hammer. This lets them take advantage of tighter integration with DragonFly's VM and VFS subsystems.
Hmm, I saw Matt Dillon's post on FreeBSD mailing list offering Hammer to be ported to FBSD, but looked like there was not much enthusiasm about from the FBSD team. It's possible it was Hammer1 and because of that he integrated Hammer2 more tightly with DFBSD.
HAMMER2 does seem to be the best alternative, but I haven't had an opportunity use it for a heavy workload. I'd be very interested in anecdotes or even guidance about how best to demonstrate its capabilities.
It's quite stable by now. I have backed up and restored around 4.5TB of data and multiple machines backup files to it over SAMBA - I see nice benefits from compression and dedup, the performance is fine and I haven't lost anything.
Opinions are good, bad and ugly. Sometimes all at once.
Thanks for letting us know one nerd's take on openzfs.
The only thing standing in zfs way (to 1st tier Linux support) is a company springing up and heading the charge. 75 percent sure this will happen eventually with ZoL at some point in a decade's time.
However, another point of measure is money. Synology is huge and seems to think btrfs is good enough for business. They compete with ixsystems directly. We could go all day and into 2020 with points and counter points. The licensing issue is weak but will eventually be moot with enough money and time behind the right project. I'm not in any position to judge that last point for sure just offering my IT pro/business take on it.
To be fair bsd-ish-ness has a role to play on the backend and is bullet proof and battle tested for the right mission. But the same can be said for Linux too. The last point I'll make is that today, I can run the Linux kernel from my msft windows 10 machine natively. No one here would have saw that in 2000. Does bsd, zfs have a similar anecdotal Hero Epic like this; where the enemy capitulated so thoroughly?
Dont' we already have the beginnings of such support with Canonical's support of ZFS in the Ubuntu Linux kernel? I'm writing this reply on a system running ZFS as the rootfs.
Regarding Linux kernel on Windows 10, if you're referring to The Linux subsystem, there's no Linux kernel there, it's a Windows subsystem implementing the Linux system calls directly. Unless you're referring to full Hyper-V virtualisation, where you can run FreeBSD just as you can run Linux.
I also didn't think the canonical effort went anywhere? Maybe I'm wrong. Re:wsl You are correct thank you for the catch. I was missing a few key words.
I'm not sure what you mean by "went anywhere". It's perfectly functional as it stands.
The main missing piece is proper support in the installer for using it as a root filesystem. Currently this requires manual setup, which is tedious, but even this is functional once done.
Syscall ABI translations are pretty common in Unices, with Linux itself having had support for a few of them, and a lot of others (including Solaris/illumos, FreeBSD, NetBSD) supporting Linux as well.
The special thing about the "Linux on Windows" subsystem is just how different the host OS is, which requires Microsoft to do some quite extensive changes to the Windows kernel in order to support it. Unix to Unix is relatively easy by comparison.
OTOH, we also have Wine, which is pretty much exactly the same sort of thing as Windows 10's Linux subsystem.
I was always hoping for more adoption of NILFS [0] - a log structured FS with continuous snapshots. In theory it should never lose data and that's a reasoning behind it not having fsck. I've seen a blog post that was debating this choice and presenting a few situations when fsck would be necessary, but I can't find it.
It would be great for /home for many normal users - they could always go back. That could also be a problem when one wants to make sure that a sensitive file is really removed. It could be also used at least for some kind of archive. Possibly also for development.
I was curious how "never lose data" works given that HDDs do not provide infinite capacity: the log is circular. This makes NILFS good for NAND flash.
The continuous per-fwrite snapshotting functionality sounds awesome - I've had a few (minor but annoying) accidents with rm of late, so I'd definitely appreciate an rm-proof undelete.
I was trying to figure out if NILFS had any sort of COW functionality, then realized, d'oh, of course it effectively does, via the log.
To address the elephant in the room - checksumming - unfortunately NILFS doesn't (yet?) do this. The WP article links to https://www.spinics.net/lists/linux-nilfs/msg01063.html, which states that the CRC32Cs that are written to disk are used only for the remount/recovery stage, and are apparently not suitable for realtime verification.
Using it on root and nearly all personal fses I second, especially hoping that it's development speed up a bit, the cleaner it's still a pain and a small wrapper to diffs between cp's can be a very nice addition!
Unfortunately the mindshare it could have had was killed by Linus announcing btrfs was the future, long before it was close to ready. Almost a decade of wasted opportunity...
anecdotal report: Im running a medium sized MariaDB database of appointments and invoice backups on a 4 disk BTRFS filesystem with deduplication. snapshots work, its stable, and it performs well.
I feel like so much of BTRFS though was death by committee. It was trying to do every single thing ZFS did, regardless of whether or not the specific feature from ZFS was mediocre to begin with.
focusing on a release cycle and communicating would have helped it alot instead of vague things like "it is ready enough" and "some features arent ready"
Is it just me or is it absolutely insane that zfs and btrfs are the only common filesystems out there that do data checksumming? I don't want or need the extra complexity of either of them in a lot of cases, but I'd sure as hell like to know if my data is corrupt...
Yeah it should be standard. Equally crazy is that APFS is brand new and didn't add it. I belive the justification was that the hardware does it on flash memory for their devices, but ZFS really exposed how unreliable hardware is.
I mean Apple has their own flash controllers, so they’re really the only company that owns the FS and the flash controller. So really they can depend on it.
What about external hard disks with terrible USB controllers? That's where data checksumming would be even more important and where Apple doesn't produce the hardware.
This has more to do with the implementation details of TimeMachine which relies on features of HFS+ not available in APFS. It's not Apple telling you not to use APFS on external drives.
There have been at least two occasions in the past three years that I remember where Apple had to recall or issue firmware updates for SSDs due to corruption issues.
It’s a big IF to depend solely on flash based error checking. I think Apple likely figured the cost of implementation and system overhead wasn’t worth it for Macs and iOS devices.
I’d be very surprised if they use APFS or a non checksumning file system on their iCloud server farm though.
A lot of applications and databases do this themselves.
Part of the problem is that the app might know what the proper checksum is and it would be nice to be able to write a document ahead of time and tell the FS that you already know the checksum.
Won't save a massive amount of CPU but it would be worth it if it was easy to implement.
There is however no point in having data checks if there are no way of recovering, so running ZFS only make sense if you have multiple disks. And don't forget to scrub your pool!
OK, so you know your HDD is corrupted, and you now have to take the site down to recover from backup, which is also old, so there will be data missing. Compared to if the FS can just fix itself. Sometimes not knowing can be a bliss though. :P
This is like saying "there's no point in ECC if it can't correct every error" ... which simply isn't true. It's still far better to know that corruption exists -- so you can know something is invalid and potentially take action -- than to have corruption silently hanging around doing corrupted data things.
That is not a fair comparison as ECC is Error Correcting, while just check-summing is not. If you however have a mirror or raid-Z the file system will be able to correct.
You can set the number of copies to keep at filesystem level. This is in addition to any redundancy provided at pool level (if any). It won't save you from a failed disk, but it will be able to transparently correct any checksum error.
It's a travesty that so few people care about moving to more reliable filesystems. A decade later and we're still stuck with ext4 and now btrfs, which I do use nowadays but I'm absolutely aware of the high risk of data corruption because the underlying code is terrible.
Windows is even worse with NTFS, which is a complete pile of garbage. I've fixed hundreds of computers with NTFS corruption over my lifetime and it's mind-boggling that Microsoft is fine with this shit.
The thing about ext4 is that it's sufficiently mediocre at what it does.
It's fairly crash resistant, has well worn and battle-tested codepaths and generally seems to not loose data outside bitrot. So why pick btrfs with it's dataloss stories or ZFS with it's complex management?
It's an effort to switch to a new FS as default, especially if either itself or the integration is not as well tested or completed yet.
So everyone just sticks to the thing they know how it sucks.
Maybe it’s because I’ve already been using it for years, but I don’t really consider ZFS any more difficult to manage than the DM/LVM/*FS stack I work with on most Linux systems.
Compared to just running gdisk and using raw partitions - sure, but I’ve been using LVM for over a decade now.
ZFS is a lot more complicated if you're expecting the average consumer to handle it. It doesn't even handle 1-disk-storage expansion.
Even Synology and QNAP now have plug-and-play solutions to expand a RAID one disk at a time. You can even start plugging in higher capacity disks and once you have enough extra parity, the higher capacity becomes used magically.
No need for wasteful RAID10/50/60 setups. Just plug in more drives.
That doesn't give you RAID protection, sadly, with mdadm and LVM you can add a single disk into an existing array, ZFS requires you to pull up a new RAID array (vdev) and at minimum 2 devices if you want something better than a RAID 0.
It is in the works but the question is how many years until it's stable and available to all end users and how well it works (IIRC from a talk on this topic, it had some limitations as well).
This also affects the RAID1 setup, any storage expansion requires 2 disk to pull up a new RAID1 into the pool.
I don't know where you read that, but don't be fooled. That functionality has been "in the works" for quite a long time. But it hasn't yet appeared.
In the past it has been said that something called "block pointer rewrite" was a prerequisite. Just google for that.
Here's some 10 year old information about it. At that time it was already a "frustratingly frequent request." I stopped searching when I found something 10 years old, but I'm sure you with some effort you could go back much further than that.
> No need for wasteful RAID10/50/60 setups. Just plug in more drives.
Those RAID set ups are only "wasteful" if you don't need that level of redundancy (ie the number of disks that can fail at the same time). Most home users only really need 1 redundancy disk (actually most home users don't even need that) but a production database server is another matter entirely.
For clarity, the reason we have redundancy drives isn't to mitigate the need for backups; it's the mitigate the likelihood of downtime while you rebuild your storage array from a backup. Which is why most home users don't even need 1 redundancy disk - they just need to be disciplined in backing important stuff up.
Scenario 1: Replace smaller disc with bigger disc. Add new disc as mirror to vdev. Wait for it to resilver. Remove original disc from vdev. Pool will autoexpand to new disc size or can be manually expanded otherwise.
Scenario 2: Expand virtual storage. Use LVM lvresize, or expand VM virtual disc. Pool will autoexpand to new volume size; may need manual triggering if the system doesn't pick up the new device size automatically. Or you can do the same as for physical discs. Add new virtual storage as mirror, resilver, then drop the old storage.
ZFS and "complex management"? there comes nothing close in the open source market than ZFS and its easy and straightforward to use management tools in my opinion.
You don't need a handful of tools just to do one simple task, just hack away at the zfs and zpool commands and everything gets done. No need for parted, mdadm, lvm or support of your needs compiled into the kernel.
No matter on which operating system you are currently, you only need those two tools and they work the same everywhere.
It comes down to when you need to anything with ZFS but simply use it for data storage.
Expanding a RAID vdev is not possible, instead the recommendation is to waste 50% of storage for a pool of mirrored devices to get the same protection as a RAID1 of two drives, which isn't much for some people.
For the avarage home user, such a scaling option is already no longer viable.
This is topped with ZFS' poor performance under low RAM conditions. FreeNAS barely managed 10MB/s when running with 2GB of RAM and an old Pentium 4 Dual Core. Not everyone can afford to throw hardware at the problem.
On the other hand, Linux performs under the same conditions with 8 times the speed for simply copying files and mdadm/lvm/unraid allow me to dynamically grow the pool of harddrive space a single drive at a time.
(plus mdadm/lvm is included in basically every larger distro by now, no need to install it or compile support into the kernel)
>Expanding a RAID vdev is not possible, instead the recommendation is to waste 50% of storage for a pool of mirrored devices to get the same protection as a RAID1 of two drives, which isn't much for some people
Or make a new vdev. It takes a bit of planning, but it's not some massive hurdle.
>This is topped with ZFS' poor performance under low RAM conditions. FreeNAS barely managed 10MB/s when running with 2GB of RAM and an old Pentium 4 Dual Core. Not everyone can afford to throw hardware at the problem.
FreeNAS is enterprise software, the default settings expect you to read and follow the system requirements. The developers aren't spending their time supporting it on less powerful systems, but ZFS (and likely FreeNAS) can run fine if you configure it for that hardware.
>>Expanding a RAID vdev is not possible, instead the recommendation is to waste 50% of storage for a pool of mirrored devices to get the same protection as a RAID1 of two drives, which isn't much for some people
>Or make a new vdev. It takes a bit of planning, but it's not some massive hurdle.
I had a 3 disk, 6TB RAIDZ (3x 3TB drives). I bought a fourth 3TB disk when I started running low on space, not knowing that this limitation existed. I ended up using an intermediate, 8TB external drive to copy data off, re-create the RAIDZ with 4 disks, and then copy the data back.
Sadly enough, the current answers are to either do what you did, or replace disks one by one.
RAIDZ expansion is coming though[0]. This is the right answer to your problem, but an answer at some point in the future doesn't help you with your problem today.
You should have read almost anything about ZFS before investing several hundred dollars and your data into it. It sucked for you, but seems like a reasonable ask.
I'm surprised you chose ZFS without knowing about it, it seems to be mentioned in every discussion about ZFS.
ZFS is no harder to use than ext4, if all you want is a file system. But then once you start needing multiple volumes or multiple disks per storage pool (eg LVM, mdadm, etc) then ZFS really does come into it's own for being by far the easiest tool for solve those problems (that I've used anyway).
The hardest part on Linux if you're running a distro that doesn't officially support ZoL so you'd have to compile you're own kernel module. But even that isn't the problem it used be in the old days of Linux since we usually have bootstrapping scripts to handle the pain now (however if you can run FreeBSD then that's always a good choice as well. It's genuinely a really nice platform to use).
If you only need a single disk filesystem, it's basically the same plus some minor RAM hunger (doesn't run well with 1-2GB regardless of how many disks you have).
ZFS isn't quite easier to user for multiple disks or volumes, rather, there is a very tiny set of configurations that everyone recommends (RAID1 mirrors, leaving consumers unable to grow pools 1 disk at a time because ZFS is fundamentally unable to grow a RAID vdev). LVM and mdadm manage these jobs much better and are composable to your needs.
> doesn't run well with 1-2GB regardless of how many disks you have
I've read people say that but I've never had any issues when I used to run ZFS on various systems with 2GB RAM. So I wonder how many of those people actually had practical experiences running ZFS on low memory systems and how much of it is people just looking at some of the recommended memory specifications (which are high) and start making assumptions about it's performance on low memory systems.
> RAID1 mirrors, leaving consumers unable to grow pools 1 disk at a time because ZFS is fundamentally unable to grow a RAID vdev
You actually can grow a ZFS vdev - exactly by replacing one disk at a time (albeit you don't get the gained capacity until all the disks in the vdev have been replaced).
What you cannot do is shrink it - but that's not a unique problem to ZFS and is a pretty uncommon use-case anyway. Also you cannot grow a ZFS "raidz" vdev by adding disks - but you cannot do that with any typical RAID5/6 array either. However if you want to add disks to a raidz then you can workaround the RAID5/6 striped parity problem by adding the additional disks into a new vdev and adding that new vdev into the existing ZFS volume (named "tank").
The kind of storage arrays that grow by adding solitary disks work quite different to your typical RAID. They'll often have designated parity drives instead. See solutions like "unraid".
> LVM and mdadm manage these jobs much better and are composable to your needs.
LVM and mdadm are not file systems though, so they only solve part of the problem. The fact ZFS handles volume management, software RAIDing, file system operations (and a bunch of other stuff that your typical fs doesn't yet support (eg snapshotting)) and does so from surprisingly intuitive command line tools is what edges ZFS for me in terms of usability.
Ultimately though, "easier" is such a broad term that we're essentially just arguing personal opinion. :)
>start making assumptions about it's performance on low memory systems.
I've ran it for real. Once a few more background services start running, ZFS bogs down and keels over. I've had to reboot the NAS dozens of times because the entire filesystem had become totally unresponsive.
>You actually can grow a ZFS vdev - exactly by replacing one disk at a time (albeit you don't get the gained capacity until all the disks in the vdev have been replaced).
Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time.
>The kind of storage arrays that grow by adding solitary disks work quite different to your typical RAID. They'll often have designated parity drives instead. See solutions like "unraid".
Not only unraid, there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too.
>surprisingly intuitive command line tools is what edges ZFS for me in terms of usability.
I found the commands anything but intuitive though I guess that's a difference of a few years of experience, I'd agree that would be personal opinion if it's easier.
> I've ran it for real. Once a few more background services start running, ZFS bogs down and keels over. I've had to reboot the NAS dozens of times because the entire filesystem had become totally unresponsive.
Was that an off the shelf NAS? If so then I'm guessing it wasn't exactly high spec'ed even putting aside the RAM? eg did it have a 32bit CPU? I've heard that does a massive amount to dent ZFS's performance (I wouldn't know because I've only ever ran on it 64 bit systems).
As I said, I've not had any issues with raidz2 on 2GB RAM. I've even ran virtual machines on one OpenSolaris box with ZFS and 2GB RAM so these weren't always dedicated storage servers either.
edit: just read elsewhere you were running FreeNAS on a Pentium 4. Yeah that would do it. FreeNAS wasn't optimised for low end systems (in fact they outright tell you not to!) and the P4 is pretty low end CPU as well. I still think ZFS would have ran on that but you'd have been better off going with vanilla FreeBSD instead.
> Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time.
...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system. That process can literally take several days on some storage servers. So ZFS might require you to add vdevs to the pool (tank) but at least it can all be done on live systems without any downtime. That in itself is a massive boon for ease of use in my opinion.
To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway. And if you are the type of person that would need to do that then you wouldn't go for a traditional raid5/6 style pool either. Thus the argument becomes a comparison of apples and oranges as you'e comparing completely different types of software RAIDing.
> Not only unraid
I didn't say "exclusively unraid". I said "solutions like unraid". I'm fully aware there are other solutions out there.
> there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too.
When you said "there are other systems that provide almost this" and "once you have added enough [disks] to provide parity, the additional capacity will become used too." I really had to question what your actual point of this discussion was because just a paragraph earlier you were calling my points "unacceptable" because ZFS required you to add more than 1 extra disk to provide parity; and now you're saying your preferred solutions also does the same?
I hope I'm just being presumptuous here but it's starting to feel like you're just arguing for the sake of arguing. :-/
> I found the commands anything but intuitive though I guess that's a difference of a few years of experience
Maybe. I've been a Linux and UNIX sysadmin for a fair few years now so experience is not something I'm short of. :)
Honest question: what is your experience of ZFS outside of the troubles you had on your NAS? I could totally see why the problems you described would taint your opinion of ZFS but I definitely wouldn't say what you experienced is typical for ZFS.
In any case, I'm not going to argue that everyone should be running ZFS - it's just technology, you can run whatever you want to run.
> To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway.
OpenZFS is currently working on adding code to allow for vdev expansion[1]. I am a fan of ZFS as well, but I don't get not wanting to admit its limitations. Not being able to expand vdevs is a limitation and it's being addressed.
I actually did acknowledged that limitation. Maybe I wasn’t all that clear but my actual point was that requirement is where most traditional raids often fall apart (even mdadm requires days of downtime to grow a vdev) so its usually better to grow the pool instead of the vdev anyway.
Or in other words: if growing the vdev is a hard requirement then a traditional raid might not be the best option regardless of ZFS. It’s about picking the right tool for the job rather than moaning that ZFS doesn’t do every job perfectly (which seems to be the argument the GP was making).
No it was a repurposed office computer running a 12 year old Pentium 4 Dual Core (about 2.8GHz per Core). 2GB of DDR2.
As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB.
>...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system.
If you use mdadm and ext4 you can do it online.
>To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway.
There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions.
The home desktop is possibly the largest filesystem market next to "disposable virtual machine".
>and now you're saying your preferred solutions also does the same?
They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage.
On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs.
>what is your experience of ZFS outside of the troubles you had on your NAS?
I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since.
> As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB.
That's pure conjecture. Plus I actually have ran ZFS on a Pentium 4 on FreeBSD. For a while that was my personal backup server.
> If you use mdadm and ext4 you can do it online.
Not according to any of the docs I've read in the past. Happy to be proven wrong if you have a citation though.
> There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions.
> The home desktop is possibly the largest filesystem market next to "disposable virtual machine".
I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al?
The issue I have with your complaints is that you're flip-flopping across a multitude of industries from desktop to enterprise, which seems grossly unfair for making a balanced opinion. I'm not going to deny that ZFS will lack some desirable features compared to some speciality tools but like I said before, it's about picking the right tool for the job rather than arguing that ZFS is garbage because it doesn't win every single argument in every specific speciality.
I mean you're not even willing to take on board any different configurations of ZFS nor alternative platforms it ships on - instead only basing your biases on one failed attempt at running FreeNAS (something which openly states they need beefy hardware to run) on equipment that is several generations old while then comparing that one specific ZFS instance against every alternative tool you've read about. That hardly seems fair, don't you think?
> They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage.
No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air.
But to assume you meant 2TB of extra storage when you add a 2TB HDD: in your situation you don't get redundancy so your data becomes at risk if you have a failure. You can actually expand your ZFS tank the same way if you really wanted (but no-one who cares about their data would want to) by adding a HDD vdev (unraided) to an existing tank.
The issue is when you want to expand an existing raidz while retaining redundancy. To do that on any of the existing FOSS tools I've used, you'd have to either buy more than one HDD or do an offline restripe (as is the case with mdadm). I guess at least mdadm allows you to do it - even if it takes the storage pool offline for several days. Personally I've always opted to buy multiple drives and expand the pool (tank) by adding an additional raidz vdev because that not only expands the storage array but also then gives me another failure option (eg I can then lose 2 disks simultaneously without losing data). As you start to expand the number of disks you have, you equally increase the likelihood that more than 1 disk might fail. So having that extra redundancy disk is desirable to me.
> On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs.
Actually no. You have to stand up two vdevs but a vdev can be a raidz, mirror, a solitary HDD or even a file on an existing file system. So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it).
> I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpoo...
>I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al?
QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs).
>That hardly seems fair, don't you think?
It's the real world experience I had with the software and I think it's totally fair to point out and complain about the shortcomings.
>No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air.
Reread what I wrote, I mentioned adding 2x2TB not 1x2TB.
> in your situation you don't get redundancy so your data becomes at risk if you have a failure
Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data.
And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array.
>So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it).
Again, what Syno offers (and what can be replicated on any Linux system) allows you to add a single drive to expand the storage without loosing parity and more drives to span any drive increases with parity. This allows adding a single drive at a time and at worst you loose any additional capacity without parity yet (in the 1TB/2TB example, that would be loosing 1TB of storage until the second drive is added)
That's resizing the filesystem not the pool (eg `mdadm --grow`). You're also missing the point about what I said. To run those commands you must first `umount` to pool. Hence it being an offline job.
> QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs).
That's completely irrelevant to the point I was making. How many home users do you know who configure their own home raids using LVM and mdadm?
> Reread what I wrote, I mentioned adding 2x2TB not 1x2TB.
You actually said neither. You said, and I quote, "2TB" (ie without any accumulation prefix).
> Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data
You can actually do that in ZFS. In fact what you've just described were is the exact solution that you called "unacceptable" in ZFS!
> And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array.
Indeed. Lots of commercial tools do support that. I thought we were comparing ext4, LVM and mdadm to ZFS though?
I'm guessing you just glossed other my post where I said you were flip-flopping all over the place? It's easy to pick any specific tool ("tool a")and compare it to a thousand other tools and argue that "tool a" is crap because it doesn't work as well as tool b, c and d in very specific edge cases but all you're doing is trying to win an argument via death from a thousand papercuts rather than forming a sane basis for analysis.
To be honest I think it's probably best we give up on this discourse because your points are all over the place. This feels very much like you've got an axe to grind and no practical experience to back up your prejudices.
>To run those commands you must first `umount` to pool. Hence it being an offline job.
No you don't need to unmount the pool under mdadm. You can do a live resize of the array. See the link I posted.
>How many home users do you know who configure their own home raids using LVM and mdadm?
Not many, hence me mentioning Syno and QNAP.
>You said, and I quote, "2TB" (ie without any accumulation prefix).
I said, quote, "if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage."
>In fact what you've just described were is the exact solution that you called "unacceptable" in ZFS!
Again, this is not what ZFS does, the mdadm solution stripes multiple RAID arrays over the same drive, which to my recollection is not a recommended configuration in ZFS by a long shot.
>Lots of commercial tools do support that. I thought we were comparing ext4, LVM and mdadm to ZFS though?
The commercial tools I mentioned previously use ext4, LVM and mdadm native functionality.
> doesn't run well with 1-2GB regardless of how many disks you have
By default, sure. There is a FreeBSD (I think) guide to tuning ZFS that lets you tune it for low memory. It's a long time since I've done this so I can't remember which settings, but I ran an old machine tuned for its low max memory, with 4 drives in RaidZ, and it was just fine.
The FreeNAS forum doesn't help much with this as the default opinion there seems to be 16GB ECC. :)
> So why pick btrfs with it's dataloss stories or ZFS with it's complex management?
As you mentioned, the bitrot. Also, my understanding is that a COW filesystem can be more reliable than a traditional journaling filesystem in a system crash since nothing is overwritten. Plus there is compression and all of the features provided by a COW filesystem.
Bitrot is rare. Over the average lifetime of most consumer desktops, the loss of any personal documents is very unlikely unless it makes up a majority of the disk. And if it does, it's most likely that it affects negligible parts of the file (ie, corruption in a frame of video).
The risk of bitrot becomes interesting when you talk about long timeframes like a decade.
Considering the capacity and price of a 2TB disk, compression isn't the immediate need of the consumer either, unless they have a NAS, where Syno and QNAP already provide integrated solutions with btrfs and ext4 both.
The modern ext4 journaling is also very battle tested, it's a very simple and resilient physical journal, data loss in ext4 happens largely when the disk is already failing or has been cheap enough to lie about the state of data being written or not.
For the average consumer that sums up to "it doesn't matter enough". Something like bcachefs could change the game by being btrfs+ZFS but without the dataloss risk.
> Bitrot is rare. Over the average lifetime of most consumer desktops, the loss of any personal documents is very unlikely unless it makes up a majority of the disk. And if it does, it's most likely that it affects negligible parts of the file (ie, corruption in a frame of video).
> The risk of bitrot becomes interesting when you talk about long timeframes like a decade.
I think you are making a lot of assumptions here about frequency of bitrot, especially in the face of failing hardware. Also, consumer desktops are not the only use. There are bunch usages where the risk of bitrot is much more concerning. Or were you intending your original post to be only about consumer usage?
>Considering the capacity and price of a 2TB disk, compression isn't the immediate need of the consumer either, unless they have a NAS, where Syno and QNAP already provide integrated solutions with btrfs and ext4 both.
This kind of feels like you are saying that it isn't needed because it is already being used when it is needed. Am I misunderstanding? Consumers may want compression for smaller hard drives they already own in order to get more out of them. Also, compression can speed up disk IO in some workloads.
> The modern ext4 journaling is also very battle tested, it's a very simple and resilient physical journal, data loss in ext4 happens largely when the disk is already failing or has been cheap enough to lie about the state of data being written or not.
The issue with ext4 resilience isn't about quality of the implementation. It is a fundamental limitation of no COW filesystems. Since things are overwritten, if the system crashes mid write you can end up with partial data. COW filesystems always give you the old data or the new data. This can be worked around but that requires that all of your applications do this workaround.
> For the average consumer that sums up to "it doesn't matter enough". Something like bcachefs could change the game by being btrfs+ZFS but without the dataloss risk.
What makes you think bcachefs is going to be any better? A lot of functionality still has to be implemented. What makes you think that the implementation of this functionality will be any less error prone?
You also haven't addressed the large pile of new features provided by BTRFS/ZFS. Snapshotting alone can make it worth the price of admission for consumers. With snapshotting, you can set up a system where rollbacks after any upgrade can occur seamlessly.
>The issue with ext4 resilience isn't about quality of the implementation. It is a fundamental limitation of no COW filesystems. Since things are overwritten, if the system crashes mid write you can end up with partial data. COW filesystems always give you the old data or the new data. This can be worked around but that requires that all of your applications do this workaround.
You do know how journalling works, yes? Unless the drive lies about the state of data being written, in which case not even COW will save you, a journal is just as good as COW. Exceptions being coding errors but those can occur on COW with the same devastating result.
Maybe you don't know as much about filesystems as you want to appear to?
> You do know how journalling works, yes? Unless the drive lies about the state of data being written, in which case not even COW will save you, a journal is just as good as COW. Exceptions being coding errors but those can occur on COW with the same devastating result.
Yes, I'm aware of how journaling works. The problem with your description is that it requires both data and metadata journaling. This is rarely the case because data journaling requires that you write all of your data to the disk twice (which also requires additional seeks as well). So, nearly all in-place filesystems default to only metadata journaling for performance reasons (this is the case for ext4). This means that, in practice, if you get a system crash mid file write, you will end up with a partially updated file. On a COW filesystem you are always guaranteed the old file or the new file.
> Maybe you don't know as much about filesystems as you want to appear to?
That is possible, although I'm not sure how much I want to appear to know :P. More seriously, this is extremely condescending. Why is this necessary? If you think I'm wrong, just tell me I'm wrong. There is a pretty decent chance I'm wrong about most things. So, condescension shouldn't be required.
I've had a single issue across many EBS drives partitioned with only XFS, in several mdadm configurations. The data-loss occured during a maintenance window, so I'm convinced it was an EBS issue and not XFS.
I've also used it on physical hardware and never experienced dataloss, even losing drives in RAID configurations (I will admit I was lucky enough to never lose 2 drives in my normal RAID10 setup).
I've had several filesystems become corrupted, but the XFS community is extremely helpful, and the tooling that comes with it has never failed me in at least letting me figure out what's actually wrong.
Ehm, what distro supports xfs out of the box and what are the guides like for setting it up? Also, it seems to be missing features like COW or data checksums (bitrot says hello), and data deduplication seems to still be experimental. Doesn't seem like the next-gen filesystem we should be hoping for.
When you're installing you can select something like "manually set the partitions/disks" and format XFS with basically any distro. It's part of the kernel, and I can't remember installing a distro without that option.
If by that you mean "these distros support XFS, but I don't use them for other reasons (package managers)" . . . OK. Don't use them. But don't criticize a filesystem for unrelated tradeoffs that OSes that support it happen to make.
> Doesn't seem like the next-gen filesystem we should be hoping for.
XFS was an early journaling file system. It wasn't designed to be next-gen, it was designed to be fast, service soft-realtime I/O demands and handle large volumes well.
We have some CentOS 7 qemu/kvm servers for hosting dev/test VMs, where the backing VM images are stored to a local RAID array formatted with XFS. Basically a 2-socket server with 128-256GB RAM and an LSI MegaRAID controller with battery-backed, write-back caching. Each virtual disk in a VM is a raw-formatted file in the host filesystem.
We have seen host kernel panics which damage the filesystem such that xfs_repair cannot fix it. It could be mounted read-only, though I am not sure I would trust the content. We just reformatted and rebuilt content from external sources.
What I believe caused this panic was a habit of creating sparse VM disk images. Later, under heavy write loads from multiple VMs, the host OS filesystem seemed to choke under all the out-of-order allocation needed to place new data into those sparse disk images. When we instituted a policy of always preallocating the VM disk files, we never saw this problem repeated.
> btrfs, which I do use nowadays but I'm absolutely aware of the high risk of data corruption because the underlying code is terrible.
Is the underlying code terrible? I haven't heard any particular concerns about code quality (aside from the original implementation of RAID 5/6). Most of the issues I've heard about were due to people using features that weren't ready yet. I've heard a lot fewer of these issue since BTRFS started providing clear statuses for individual features[1]. Do you have a different impression?
I've used btrfs as my main fs (with ample backups) since 2012. Plenty of file system corruption and backup restores in the intervening years, but I went from having it happen 4x a year in 2012-2013 to once a year by 2015 and never since 2016.
Its totally anecdotal to say just because my ~10 hard drives with it being stable for two years means its improved but I'd consider putting some faith in it now. A bit. With backups.
The challenge here is that ZFS is dominate by Oracle right now, and if anyone thinks that Oracle is going to act in the best interest of the community, rather then killing the golden goose to peel the meat off it's bones, take a look at what Oracle has done with Java recently. Or what they did with Solaris. Or what they do with Oracle Cloud.
in addition, we know that there are patent landmines out there because NetApp and Oracle have a (undisclosed) settlement on the issue. Using ZFS is a probably good way to get a visit from your legendary friendly Oracle sales agent and auditor. Oracle could fix all of this with a quick re-license or patent indemnification, but they haven't and almost certainly won't.
My Synology (and all modern Synologies) uses BTRFS, and it seems pretty solid so far. I would have preferred that ZFS had matured and been dominate in FS, but there are some really risky things that come with ZFS.
ZFS is not dominated by Oracle, Open-ZFS is the source of record for anything other than Solaris and critical features are being added that Oracle cannot use without opening their branch back up under the CDDL.
The CDDL also includes a patent agreement, Oracle can’t retroactively waive it.
>Go look at what the Java community is going through right now.
You did not understand what happened with the latest Oracle Java news and you are spreading FUD.
People have a choice what Java implementation to use, they can pay for latest Oracle implementations, use Open JDK (it is included in Debian and most distributions), you can get support from RedHat and other big vendors. There is no risk of getting in trouble by using Open JDK, MS and Google had trouble because they "forked" Java
Is btrfs still a thing? Reading other people's experiences, atleast a couple distros dropping it as a default, and my personal experience of it crashing completely. I'm surprised anyone would even consider it as an option. It simply isn't a reliable file system.
As personal experiences go, btrfs works great for me.
I've been using it on a 6 drive NAS since 2012, and on two laptops with SSDs.
In my experience "crashes" are symptoms of people who didn't read the documentation and don't understand how to recover.
For example, on the laptops I've often run out of disk space. And btrfs can run into a situation where it can't clean up partly used blocks without more disk space. To recover you have to add more space. I usually used the swap partition, but a USB stick also works. Or a RAM drive, but that's risky.
Some other issues I've had to help people with is snapshot deletion. Deleting a btrfs snapshot appears instant but none of the used space comes back. That's because it is doing it in the background and it can take a really long time. OTOH, ext4 does exactly the same thing with background delete of very large files.
>In my experience "crashes" are symptoms of people who didn't read the documentation and don't understand how to recover.
aka most users. Filesystems must be reasonably resilient to incorrect usage. This is a problem across the board; if you're not careful setting up a ZFS pool, it's easy to accidentally create a scenario where you're unwittingly striping across multiple devices, greatly increasing the likelihood of a catastrophic data loss event without ensuring the user is fully aware.
I've lost multiple terabytes with btrfs on multiple different systems. It was essentially a random occurance. These were all cases of simply using it as a regular filesystem. I read the docs I attempted recovery. They didn't work. And quite frankly I shouldn't need to take those steps anyways. As for your example that seems insane someone should have to jump through hoops like that to utilize a hard drive. Btrfs is junk and I am glad it has been removed as the default on a couple distros.
Yeah that's every filesystem ever. I helped a guy out just a couple of months ago because his Debian virtual machine had plenty of disk space but he couldn't install anything.
Because it was out of inodes. What "regular user" aka Nodejs / React developer knows about inodes? Not that guy.
You’re confusing the expansion of a ZFS pool with the expansion of a RAID-Z vdev. Throwing new disks into a pool has always been possible, adding them to an existing RAID-Z vdev isn’t (yet, block pointer rewrite and RAID-Z expansion are coming).
I run exclusively with mirrored vdev’s and have never run into issues growing my pool.
There was just a talk on it this year, it’s coming but between the time it takes to go stable in OpenZFS and the time it takes to hit downstream (FreeBSD, ZoL) it might not be for a while yet.
It's been able to do online enlarge and extend forever, and it will even automatically enlarge if you replace disks in a vdev one at a time allowing for resilvering.
Perhaps you're thinking of shrinking, which it still cannot do without rebuilding the whole FS infrastructure.
While I definitely sympathize with the desire here, mirrored disks are almost always the best way to go. It has saved my butt just in my home setup a couple of times, most recently when a power connector got loose and started shutting a disk on and off basically at random. Mirrored disk mean quick recovery from such scenarios and relatively-limited impact during degradation.
My opinion after trying both parity-checked redundancy and true mirrored redundancy is that any serious data storage should be strictly on mirrored volumes (preferably with disks of substantially different ages so you don't risk them dying within a couple weeks of each other, possibly before a replacement gets installed).
I once read "50% storage efficiency is plenty, don't be greedy". Although painful for the home user, that seems like the right attitude to take for important stuff, especially as volume sizes increase. It can be a very slow and very fraught process to recover several TBs from parity data.
I've been burned by Btrfs enough times to avoid it.
I had a VM running OpenSuse Tumbleweed - I know, rolling release, don't expect stability, etc - and it would consistently freeze after being up and unused for a few days. Rebuilt the VM but used XFS instead of Btrfs, it was rock solid. This happened within the last four months.
At work, we have a number of machines using Btrfs. At some point they get into a state where the weekly Btrfs scrub process hangs the machine for hours at a time. The solution in every case was to move that filesystem onto a non-Btrfs volume. This is an ongoing issue, but I'm not in the team responsible so I don't know what they're doing with this. Anecdotally, this seems to be a problem when there is a lot of writing going on with the volumes. Machines that use Btrfs root but do most of their work on NFS volumes don't seem to experience this issue.
I also have a laptop running OpenSuse Tumbleweed with Btrfs. One day it somehow got itself into a state where it would hang when mounting the filesystem read/write, but it would be fine if it was mounted read-only. I can't recall exactly what happened to cause it, but I'm pretty sure I didn't do anything exceptional with the computer - I think I rebooted it from the GUI while it was doing a scrub in the background, but that's just a guess. I think this happened about a year ago.
This is all disappointing because some of the things Btrfs enables are awesome. In particular, OpenSuse's snapper has saved me countless hour of work. I haven't played with the send/receive stuff, but it seems like a great solution for backups.
In the meantime, I'm using ZFS on a variety of FreeBSD boxes with a variety of workloads (including one running more VMs than its RAM should support) and it's been rock solid.
At this point it will be a long time before I give Btrfs another look. It has the potential to be really good, and I want it to be good, but I don't believe it's there yet.
Yes, this is similar to my fairly recent experience with btrfs. While it's competitive on paper and there are several meta-arguments in its favor (being in mainline, for example), the cold hard facts just don't stack up. btrfs fares far worse and requires much more tuning than a generic ZFS installation.
A great example is that if you ever try to run a database or a large virtual machine on a btrfs mount, you'll quickly become acquainted with chattr +C for "nocow". Any write-intensive applications grinds the filesystem to a halt until you disable the copy-on-write functionality for that file specifically, making it so snapshots, etc., are ineffective and the FS stops tracking the changes that would make them possible.
Whereas on ZFS, like any application that uses fixed block sizes, virtualization and databases _may_ require some alignment tuning to avoid a read-modify-write cycle, and ZFS works without fuss from that moment on.
I know that Facebook is investing heavily in btrfs right now, but to be frank, I don't see the point. btrfs doesn't have its youth as an excuse anymore and the core issues that exist seem like they're pretty fundamental.
It's a gargantuan waste of effort when people could just take the tack that Ubuntu has taken and decide all that FUD around the CDDL, kicked up by fears that the commercial Unices were plotting to take their revenge on Linux, that maybe all that isn't super relevant anymore, and they can just start shipping ZoL.
Facebook uses btrfs extensively [1]. I was tempted to try it out again (burnt my fingers on it a couple of years back) but from the experiences mentioned here it looks like it's not really production-ready yet.
I gave up on btrfs a long time ago. People were considering it 'production ready' when I experienced an issue that rendered a volume unusable, in the form of kernel panics on boot. This rubbed me the wrong way and I just moved back to ZFS-on-Linux which I've never had as much as a single issue with - not even once!
Today I'm running ext4 because maintaining ZFS setups is exhausting since it is not upstreamed.
Still... I hope there's a future in one of these volume-manager-meets-filesystem filesystems. Maybe bcachefs will end up being the answer some day?
Quick note on btrfs: unless you've been dealing with the latest kernels in the last few months, reserve judgement. Facebook has put a truly ridiculous number of hours on that filesystem in the last year or so and fixed a concomitant number of issues.
From what I'm seeing in this thread, basically two things:
People who have been running btrfs in production (reckless, imho) on kernels that are 2.5+ years old or worse. And then they're surprised and disappointed when it loses data.
For everyone who's posted here with a btrfs data loss story, it would be helpful to quantify your anecdote with what kernel you were running.
Top post in this thread is about btrfs is on Tumbleweed, which is not an old kernel. My experiences have been similar. Btrfs is a great backup stress tool. I have no idea how Facebook is running it in production…
"Btrfs has played a role in increasing efficiency and resource utilization in Facebook’s data centers in a number of different applications. Recently, Btrfs helped eliminate priority inversions caused by the journaling behavior of the previous filesystem, when used for I/O control with cgroup2 (described below). Btrfs is the only filesystem implementation that currently works with resource isolation, and it’s now deployed on millions of servers, driving significant efficiency gains."
No one should underestimate the power of employing a system's maintainer. I'm sure one feels much more confident knowing that the primary author of the filesystem is available full-time to resolve any issues they may encounter.
It's very common for MegaTechCos to hire maintainers like this, but us in the hoi polloi need to be cautious not to jump headfirst into a situation where we can't call upon the experts/inventors to get us out of any jam. We have to rely on our own abilities, the mailing list archives, and hopefully some relevant random blog posts that aren't too outdated.
This should be taken into consideration much more often than it is. If anything, maintainer employment by a MegaTechCo should nix out their implied endorsement of the technology rather than taken as proof that it's viable.
ZoL and Btrfs are the only two filesystems that I've ever had rendered unmountable without any reported disk error. In both cases, ECC ram was in use, and mirrors were used.
Btrfs was a known bug, and the kind folks on #btrfs were able to help me recover (though it was a quite involved process).
For ZFS, the consensus in #zfs and #zfsonlinux was that with the error message I was getting "I hope you have backups."[1]
I see XFS mentioned here, but in the admittedly unusaly use-case of editing a file that when compiled and modprobed may hang the kernel, I found that vim was insufficiently fsync-ing to ensure I had valid data (either old or new copy would have been acceptible).
I don't know what my point was with this comment except "All filesystems are buggy, so backup your data"
1: It should be noted that in the case of ZFS, I was on vacation and it was my laptop that had no critical data, but I absolutely needed it working for other things, so I only spent a couple hours trying to recover; The btrfs issue happened at home where I had plenty other systems to use and where I was more motivated to recover the data.
I've been super happy with ZFS for over a decade. All the pain I've had with it has been related to deduplication requiring massive amounts of RAM. When you don't have the RAM, it tends to be flaky. But I've never run into data loss with it, despite my best attempts.
My current backup server would need ~50GB of RAM just for the in-RAM deduplication tables.
I've been thinking of trying btrfs for the backup server, but the deduplication story is not something that's obvious, it all seems to be third-party, which makes me skeptical. But it worth trying. I might just try a test Dragonfly setup, because my simple testing of HAMMER seemed very promising, years ago.
I've had pretty bad experiences with btrfs in the distant past. Around a decade ago I had all my company laptops using btrfs and it worked great, as long as we didn't get >80% usage or something. It worked well for around a year. But then the reliability of btrfs dropped, and we had repeated data loss and switched back to ext.
> I've had pretty bad experiences with btrfs in the distant past. Around a decade ago I had all my company laptops using btrfs and it worked great, as long as we didn't get >80% usage or something. It worked well for around a year. But then the reliability of btrfs dropped, and we had repeated data loss and switched back to ext.
btrfs wasn't considered stable until 2012. Using an experimental filesystem for all your company laptops seems to be asking for trouble.
You really can't blame ZFS for needing ram for dedup. The raw math around looking up checksum just demands it. There are some other systems that can achieve fast dedup, but instead of ram they would require a ton of super fast SSDs to store the tables.
There are some other types of dedup that don't guarantee 100% dedup, but then the dedup levels would very depending on data sets -- this may be fine for VMs and user data. They work by being using data locality, predicting which dedup table would needed to dedup the next N bytes. But it is not likely we are going to see this any time soon because the patents on this sort of dedup is fairly recent.
> You really can't blame ZFS for needing ram for dedup. The raw math around looking up checksum just demands it. There are some other systems that can achieve fast dedup, but instead of ram they would require a ton of super fast SSDs to store the tables.
This is only true for real-time, "live" dedup, which is an essential feature for almost nothing.
btrfs' offline dedup (which, despite the name, can be done on a mounted filesystem) is far more rational. You fill the disk with data, and then dedup it later, at a schedule that works for you. No extra RAM needed.
> btrfs' offline dedup (which, despite the name, can be done on a mounted filesystem) is far more rational.
You still have to process it, and you are still going to use a lot of ram -- and you are going to want to use a lot of ram once you get into larger data sets otherwise the process will take days.
Most sane deduplication systems use a Bloom filter to avoid direct lookups to huge tables of content hashes. Not sure if ZFS does this or not, but even NTFS deduplication does.
I get it during the Sun years, but why the hell does Oracle continue developing Btrfs when they can just release ZFS under a GPL-compatible license and be done with it? It reallys seems like they are working against their own interests here.
They’re still selling Solaris licenses to enterprises. If they keep the FUD churning they can continue to suck what is essentially free money from those customers.
Does a GPL-compatible ZFS really detract from their FUD churning Solaris business, though?
Those customers can't redistribute Solaris any more than they can redistribute Linux with a CDDL-licensed ZFS. If they really wanted to switch to Linux they could do so today and be in the exact same legal position license-wise. Not to mention they could switch to FreeBSD and be able to redistribute, but again, I doubt this is something very many Solaris users care about. Perhaps the cost of porting their applications away from Solaris is too high, or their middle managers are Oracle fanboys, whatever the reason is, a GPL-compatible ZFS does not threaten that business model.
I think a more likely reason why they pour money into the Btrfs death march while having a turnkey solution just sitting around in the next office over waiting to be singed off by the legal department as CDDL+GPL or Apache or whatever is that each project is the pet of a different pointy-haired boss and they refuse to cooperate internally.
224 comments
[ 4.8 ms ] story [ 222 ms ] threadI don't use RAID5, which is where I've heard btrfs is dangerous, just for the snapshots and dedupe (with https://github.com/jbruchon/jdupes/ )
(I work at fb)
That video's discussing btrfs layering, which is very neat.
I am _very_ curious what sort of experience FB has had with btrfs.
If FB has decided to go all-in with btrfs, you probably have the most accurate raw data there is to have.
Of course, any hyper-scale deployment of a technology is going to produce remarkable/exponential statistics, and these will (sadly/annoyingly) need careful parsing to normalize in a way that is generally accessible and unsurprising. (We live in a very knee-jerk world, and all. Sigh)
All this to say, I at least am looking forward to whatever sorts of numbers you end up being able to share.
--
After considering the bit about btrfs in the video, and your mention of "1 or more machines", I wonder if you aren't using btrfs on things like load balancer type systems - or, more abstractly, node configurations that can always safely be "thrown away", whether by hardware failure or explicit decommission. But then I realize the chances are you probably use such a model (individual hardware failure must be acceptable; everything important must be n-way redundant) because nothing else makes sense at scale. And then I wonder... exactly what point in that spectrum does btrfs fit in? I wonder if/how you can answer that question.
Doesn't appear to be merged yet, but it's coming.
- RAID5/6 mode is only experimental in btrfs, while it is really stable in ZFS.
- I don't have concrete data for that, but in my experience, BTRFS has high latency (>1 second) even for small file operations when under load.
- While that should not be a problem for production systems, I have some crappy hardware where BTRFS oopses or corrupts data, while other filesystems (Ext4, ZFS) work fine.
I even had to add eatmydata support to the schroot tool as a command-prefix option to allow every command to be run via eatmydata when using btrfs snapshots. (I've since dropped btrfs support entirely; it was too unreliable with large snapshot turnover rapidly unbalancing the filesystem. Unusable in production.)
When there's only a single filesystem, and that filesystem is btrfs (or ZFS), it should however be possible to optimise this away and delegate everything to the filesystem. But even here, maintainer scripts may issue their own fsyncs as they update their own databases, kernel images or whatever.
Not if file-change notifications were supported robustly by dpkg and the kernel (to a lesser extent). Getting to that would, however, require massively restricting the compatible-kernel-versions set of dpkg, and would also probably require undoing some of the more . . . misguided pieces of history with regard to file-change notification systems in Linux.
The problem is that the system state needs checkpointing for every package state change. It must allow for recovery on failure, termination, abortion or power loss, amongst other scenarios. And the package database must remain in sync with the filesystem state.
When every managed file is on one snapshot-able filesystem, this could be rolled back atomically, and the fsyncs skipped. But as soon as you have a non-snapshot-able filesystem or multiple filesystems in use, the fsyncs can't be skipped.
Had there not been uncertainties about licensing which prevented ZFS from being in mainline there wouldn't be any Btrfs. How can you ever really trust a filesystem developed by adding features first, trying to add stability and usability later anyway?
I did think it was odd that installing the ZFS package took such a long time, but that was a one-time wait.
While working in that specific industry it seems like there is a motivation to specifically go out of their way to make things unreliable, extra complicated and utilize bad technologies.
The fact that IBM and Toshiba on POS is frightening.
Checkout some of the writing about it here: https://www.patreon.com/bcachefs
I'm curious that they're talking about implementing encryption at the filesystem layer. This has typically been done elsewhere. And anyway, key management is an issue. Seems like it would be beyond the scope for a filesystem. But I had similar thoughts right before I started learning about the write anywhere layout used by NetApp all those years ago, where mixing layers had enormous benefits (ZFS too).
The idea of incorporating a flash translation layer (FTL) is interesting, but there is the hardware support issue. Meaning that yes, you can still buy raw NAND flash memory, but what are you going to connect it to? NAND flash controllers which can present a useful interface to the host processor (like USB) already incorporate a FTL.
Similarly, eMMC memory has a FTL layer baked in, and presents itself as a block-addressable device.
Raw NAND flash controllers are increasingly rare on modern microcontrollers.
Personally I'm not a huge fan of this because unlike the clear benefits of giving filesystems control over raw devices, full disk encryption has requirements that can't really be provided partially. You want to ensure no metadata about the filesystem structure or files will be known without knowing the encryption key but this is clearly not possible without having block device layer encryption. ZFS encryption allows for all sorts of useful operations on encrypted filesystems without having the key. This is cool, but also obviously provides a lot of information about the filesystem structure. Also dedup tables aren't encrypted in this setup. ext4 only encrypts filenames and contents, which is worse in some respects.
This sounds like low-hanging fruit for a security researcher to try their hand at: comprehend the dedupe table format, and see what you can derive from it (basically the real-world version of https://xkcd.com/704/).
So an attacker can tell roughly how much metadata you have, but literally nothing about the contents.
I've never heard of "write anywhere" before but it sounds interesting. What is/was this?
> ... incorporating a flash translation layer (FTL) is interesting, but ... what are you going to connect it to? NAND flash controllers ... already incorporate a FTL. ... Raw NAND flash controllers are increasingly rare on modern microcontrollers.
My read of the FTL bit is that of hopeful optimization that supports the ideal-case - say, the filesystem equivalent of all the revving-up everyone is doing to support RISC-V. It's clearly leaning pretty heavily on the bit about existing precedent. My practical hope is that the code stays in good, well-tended condition up to the point it can be usefully taken advantage of - and that someone notices it to good effect sooner rather than later.
https://en.wikipedia.org/wiki/Write_Anywhere_File_Layout
[1] https://en.wikipedia.org/wiki/Disk_encryption_theory#XTS
It was a very pleasant experience and I hope that it gets mainlined in the close future. Linux desperately needs ZFS with better integration and more flexibility.
Was it due to bcachefs in some way?
Is there anything you felt was still missing back when you were using it?
> zfs is block based, not extent based, whereas all other modern filesystems have been extent based for years: the reason they did this is that extents plus snapshots are really hard.
then:
>Snapshots: In progress. bcachefs's snapshot implementation is going to be significantly more capable than any competing implementations. When it's done, ...
Multiple ZFS challengers can get easily to the point where they have 80-90% of the cool features that ZFS has. Then everything slows down to crawl because getting the same performance and reliability + features is super hard. When you leave the essential but hard features 'for later' they are harder to add later without big rewrite (or losing performance).
Ticking all the boxes with good performance becomes exponentially harder as number of features grow. There comes time to order things: sacrifice performance in this to attain this.
But even if that's not true, bcachefs still has an excellent feature set already with great performance, especially in long-tail latency.
He or any other excellent guy don't have an idea how the implementation he designs works in multiple workloads and different machines and disk drives. There will be time period of several years of adjusting and tinkering and compromising.
https://github.com/openzfsonwindows/ZFSin
That seems only slightly less scary than the btrfs on Windows project (which suggests disabling SecureBoot). At least it's from a ZFS contributor (while btrfs on Windows seems like a from-scratch rewrite, ie. all the bugs all over again).
From the comments here I take it people aren't actually using anything with features like snapshots/cow/compression on Windows :-/
Microsoft has somewhat ridiculous driver signing requirements. I become angry about this perhaps once every four months.
FAT32 "seamless support"? It doesn't even have journaling. The other day I wanted to copy a 8 GB file. Forget FAT32. Just use Ext4. It isn't patent encumbered, and there are drivers available for all of the big 3. You can boot a live CD/DVD/USB to mount the data in no time. This is easier than installing any driver, and you won't have to uninstall or maintain it afterwards.
It's better than FAT and widely supported.
https://en.wikipedia.org/wiki/ReFS
[1]: https://github.com/maharmstone/btrfs
https://github.com/maharmstone/btrfs
(I have never used it, just have seen this refered in the ReactOS release announcement...)
[1] https://www.paragon-software.com/home/apfs-windows/
It’s all network. So all I need is working file-transfer protocols.
In the case of OpenZFS, where it's implemented by multiple operating systems with differing requirements, maybe the different priorities and goals of the different teams will result in cross-pollination of ideas and implementation details rather than it stagnating top-down. I.e. there's scope for some differences in features and details, while still adhering to the common design.
I'm a bit confused by this. Irem wrote:
> But if Debian decides to drop x86 tomorrow, you go "Cool, good luck, I'll just fork."
When Systemd made its way into debian, debian _was_ forked by devuan (and possibly by others) to keep it out. So...is your point exactly?
edit: Maybe I misunderstood entirely and you were using Systemd as an example _supporting_ Irem's post...
There are areas where it just doesn't work, such as in the embedded space.
Maintaining an entire Debian architecture is probably not a one-person job.
Whereas Apple really could just up and drop x86 any day now, and I'll just be stuck.
another point that shows that ZFS is unsuitable for the desktop is it's reliance on EEC RAM: https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...
i considered getting EEC RAM for my machine just because of ZFS, but i could not find any 16GB strips to fill my board.
that's two strikes against desktop use, and enough to let me prefer btrfs.
EDIT: i am very happy to see so many replies proving me wrong. when i last researched this topic i was not able to find any counter arguments to needing EEC.
greetings, eMBee.
Edit: oh and it works beautifully with single disk setups too.
That's like saying you don't need health insurance because you haven't got ill.
Claim that you can only use ZFS with ECC memory to feel safe is false.
I used to use non-ECC memory on my laptop when running Belenix, and I now use ECC memory on my servers at Hetzner where I run my own Illumos builds.
(I'm one of the maintainers of Belenix, an early opensolaris based distro that was the foundation of Solaris 11. I'm working on reviving it again)
“There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error.
I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS.”
https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
W.r.t. to btrfs they are at least working on a fsck tool, even though it can't repair much yet (what it can repair is e.g. some bitflips caused by bad RAM: https://patchwork.kernel.org/patch/4116411/ ).
And stuff like ext4 or ntfs can get back to a state where one can write to it again in most cases.
Another problem is that many applications don't handle EIO well, i.e. they don't give you an option like "Continue with corrupt data", e.g. if a video file gets damaged it doesn't matter much, it'll just be a few damaged pixels, but the whole video player stops playback because it doesn't handle the IO error.
If you read closely the posts that are supposed to prove this claim, they never give strong evidence that the pool losses are directly linked to the lack of ECC memory.
It always ends up with cyberjoke stating that the issue was linked to lack of ECC.
Moreover, One of the OpenZFS developers confirms this is a myth here : https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
TL;DR: Zfs is as vulnerable as any other FS to bit flips. So if you care about your data, use ECC, regardless of your file system.
TL;DR is that a few stuck bits is won't corrupt your pool like the forum post claims.
Also, in the highly unlikely event that bad RAM would make ZFS corrupt data, how would btrfs perform better on the same hardware?
Although looks like it'll be DragonFly only.
Also the author do not mention that LLNL zfs Linux port is not as production ready than FreeBSD one.
Personally I use nilfs2 on top of lvm since is Linux mainline from many many years, it a logfs like Hammer volumes, so it protect effectively against accidental overwrite/deletion and can be live resized both in grow and shrink so coupled with LVM can be a poor man's pooled storage... But Hammer on Linux will be a success like OpenSSH!
Got it migrated over to HAMMER 2 recently and it's pretty sweet.
Here's some benchmark comparison - https://www.phoronix.com/scan.php?page=article&item=8-linux-...
Thanks for letting us know one nerd's take on openzfs.
The only thing standing in zfs way (to 1st tier Linux support) is a company springing up and heading the charge. 75 percent sure this will happen eventually with ZoL at some point in a decade's time.
However, another point of measure is money. Synology is huge and seems to think btrfs is good enough for business. They compete with ixsystems directly. We could go all day and into 2020 with points and counter points. The licensing issue is weak but will eventually be moot with enough money and time behind the right project. I'm not in any position to judge that last point for sure just offering my IT pro/business take on it.
To be fair bsd-ish-ness has a role to play on the backend and is bullet proof and battle tested for the right mission. But the same can be said for Linux too. The last point I'll make is that today, I can run the Linux kernel from my msft windows 10 machine natively. No one here would have saw that in 2000. Does bsd, zfs have a similar anecdotal Hero Epic like this; where the enemy capitulated so thoroughly?
The main missing piece is proper support in the installer for using it as a root filesystem. Currently this requires manual setup, which is tedious, but even this is functional once done.
The special thing about the "Linux on Windows" subsystem is just how different the host OS is, which requires Microsoft to do some quite extensive changes to the Windows kernel in order to support it. Unix to Unix is relatively easy by comparison.
OTOH, we also have Wine, which is pretty much exactly the same sort of thing as Windows 10's Linux subsystem.
It would be great for /home for many normal users - they could always go back. That could also be a problem when one wants to make sure that a sensitive file is really removed. It could be also used at least for some kind of archive. Possibly also for development.
[0] https://en.wikipedia.org/wiki/NILFS
The continuous per-fwrite snapshotting functionality sounds awesome - I've had a few (minor but annoying) accidents with rm of late, so I'd definitely appreciate an rm-proof undelete.
I was trying to figure out if NILFS had any sort of COW functionality, then realized, d'oh, of course it effectively does, via the log.
To address the elephant in the room - checksumming - unfortunately NILFS doesn't (yet?) do this. The WP article links to https://www.spinics.net/lists/linux-nilfs/msg01063.html, which states that the CRC32Cs that are written to disk are used only for the remount/recovery stage, and are apparently not suitable for realtime verification.
I feel like so much of BTRFS though was death by committee. It was trying to do every single thing ZFS did, regardless of whether or not the specific feature from ZFS was mediocre to begin with.
focusing on a release cycle and communicating would have helped it alot instead of vague things like "it is ready enough" and "some features arent ready"
> Time Machine can’t back up to an APFS-formatted disk. If you select a back up disk formatted as APFS, Time Machine offers to reformat it as HFS+.[1]
[1] https://support.apple.com/guide/mac-help/types-of-disks-you-...
It’s a big IF to depend solely on flash based error checking. I think Apple likely figured the cost of implementation and system overhead wasn’t worth it for Macs and iOS devices.
I’d be very surprised if they use APFS or a non checksumning file system on their iCloud server farm though.
Part of the problem is that the app might know what the proper checksum is and it would be nice to be able to write a document ahead of time and tell the FS that you already know the checksum.
Won't save a massive amount of CPU but it would be worth it if it was easy to implement.
Whether or not you have a RAID you should still have backups, and you don't want to replace a good backup by a corrupt one.
Windows is even worse with NTFS, which is a complete pile of garbage. I've fixed hundreds of computers with NTFS corruption over my lifetime and it's mind-boggling that Microsoft is fine with this shit.
It's fairly crash resistant, has well worn and battle-tested codepaths and generally seems to not loose data outside bitrot. So why pick btrfs with it's dataloss stories or ZFS with it's complex management?
It's an effort to switch to a new FS as default, especially if either itself or the integration is not as well tested or completed yet.
So everyone just sticks to the thing they know how it sucks.
Compared to just running gdisk and using raw partitions - sure, but I’ve been using LVM for over a decade now.
Even Synology and QNAP now have plug-and-play solutions to expand a RAID one disk at a time. You can even start plugging in higher capacity disks and once you have enough extra parity, the higher capacity becomes used magically.
No need for wasteful RAID10/50/60 setups. Just plug in more drives.
This also affects the RAID1 setup, any storage expansion requires 2 disk to pull up a new RAID1 into the pool.
I don't know where you read that, but don't be fooled. That functionality has been "in the works" for quite a long time. But it hasn't yet appeared.
In the past it has been said that something called "block pointer rewrite" was a prerequisite. Just google for that.
Here's some 10 year old information about it. At that time it was already a "frustratingly frequent request." I stopped searching when I found something 10 years old, but I'm sure you with some effort you could go back much further than that.
https://blogs.oracle.com/ahl/expand-o-matic-raid-z
Those RAID set ups are only "wasteful" if you don't need that level of redundancy (ie the number of disks that can fail at the same time). Most home users only really need 1 redundancy disk (actually most home users don't even need that) but a production database server is another matter entirely.
For clarity, the reason we have redundancy drives isn't to mitigate the need for backups; it's the mitigate the likelihood of downtime while you rebuild your storage array from a backup. Which is why most home users don't even need 1 redundancy disk - they just need to be disciplined in backing important stuff up.
Scenario 1: Replace smaller disc with bigger disc. Add new disc as mirror to vdev. Wait for it to resilver. Remove original disc from vdev. Pool will autoexpand to new disc size or can be manually expanded otherwise.
Scenario 2: Expand virtual storage. Use LVM lvresize, or expand VM virtual disc. Pool will autoexpand to new volume size; may need manual triggering if the system doesn't pick up the new device size automatically. Or you can do the same as for physical discs. Add new virtual storage as mirror, resilver, then drop the old storage.
You don't need a handful of tools just to do one simple task, just hack away at the zfs and zpool commands and everything gets done. No need for parted, mdadm, lvm or support of your needs compiled into the kernel. No matter on which operating system you are currently, you only need those two tools and they work the same everywhere.
Expanding a RAID vdev is not possible, instead the recommendation is to waste 50% of storage for a pool of mirrored devices to get the same protection as a RAID1 of two drives, which isn't much for some people.
For the avarage home user, such a scaling option is already no longer viable.
This is topped with ZFS' poor performance under low RAM conditions. FreeNAS barely managed 10MB/s when running with 2GB of RAM and an old Pentium 4 Dual Core. Not everyone can afford to throw hardware at the problem.
On the other hand, Linux performs under the same conditions with 8 times the speed for simply copying files and mdadm/lvm/unraid allow me to dynamically grow the pool of harddrive space a single drive at a time.
(plus mdadm/lvm is included in basically every larger distro by now, no need to install it or compile support into the kernel)
Or make a new vdev. It takes a bit of planning, but it's not some massive hurdle.
>This is topped with ZFS' poor performance under low RAM conditions. FreeNAS barely managed 10MB/s when running with 2GB of RAM and an old Pentium 4 Dual Core. Not everyone can afford to throw hardware at the problem.
FreeNAS is enterprise software, the default settings expect you to read and follow the system requirements. The developers aren't spending their time supporting it on less powerful systems, but ZFS (and likely FreeNAS) can run fine if you configure it for that hardware.
>Or make a new vdev. It takes a bit of planning, but it's not some massive hurdle.
I had a 3 disk, 6TB RAIDZ (3x 3TB drives). I bought a fourth 3TB disk when I started running low on space, not knowing that this limitation existed. I ended up using an intermediate, 8TB external drive to copy data off, re-create the RAIDZ with 4 disks, and then copy the data back.
This sucked. What should I have done instead?
RAIDZ expansion is coming though[0]. This is the right answer to your problem, but an answer at some point in the future doesn't help you with your problem today.
[0] https://www.bsdcan.org/2018/schedule/events/960.en.html
And, replacing disks one by one would require me to buy 3 disks instead of 1. I did that the last time I grew my array :-/
You should have read almost anything about ZFS before investing several hundred dollars and your data into it. It sucked for you, but seems like a reasonable ask.
I'm surprised you chose ZFS without knowing about it, it seems to be mentioned in every discussion about ZFS.
I was also making the point that "make a new vdev" doesn't solve that particular problem.
Overall, I'm quite happy with ZFS.
The hardest part on Linux if you're running a distro that doesn't officially support ZoL so you'd have to compile you're own kernel module. But even that isn't the problem it used be in the old days of Linux since we usually have bootstrapping scripts to handle the pain now (however if you can run FreeBSD then that's always a good choice as well. It's genuinely a really nice platform to use).
ZFS isn't quite easier to user for multiple disks or volumes, rather, there is a very tiny set of configurations that everyone recommends (RAID1 mirrors, leaving consumers unable to grow pools 1 disk at a time because ZFS is fundamentally unable to grow a RAID vdev). LVM and mdadm manage these jobs much better and are composable to your needs.
I've read people say that but I've never had any issues when I used to run ZFS on various systems with 2GB RAM. So I wonder how many of those people actually had practical experiences running ZFS on low memory systems and how much of it is people just looking at some of the recommended memory specifications (which are high) and start making assumptions about it's performance on low memory systems.
> RAID1 mirrors, leaving consumers unable to grow pools 1 disk at a time because ZFS is fundamentally unable to grow a RAID vdev
You actually can grow a ZFS vdev - exactly by replacing one disk at a time (albeit you don't get the gained capacity until all the disks in the vdev have been replaced).
What you cannot do is shrink it - but that's not a unique problem to ZFS and is a pretty uncommon use-case anyway. Also you cannot grow a ZFS "raidz" vdev by adding disks - but you cannot do that with any typical RAID5/6 array either. However if you want to add disks to a raidz then you can workaround the RAID5/6 striped parity problem by adding the additional disks into a new vdev and adding that new vdev into the existing ZFS volume (named "tank").
The kind of storage arrays that grow by adding solitary disks work quite different to your typical RAID. They'll often have designated parity drives instead. See solutions like "unraid".
> LVM and mdadm manage these jobs much better and are composable to your needs.
LVM and mdadm are not file systems though, so they only solve part of the problem. The fact ZFS handles volume management, software RAIDing, file system operations (and a bunch of other stuff that your typical fs doesn't yet support (eg snapshotting)) and does so from surprisingly intuitive command line tools is what edges ZFS for me in terms of usability.
Ultimately though, "easier" is such a broad term that we're essentially just arguing personal opinion. :)
I've ran it for real. Once a few more background services start running, ZFS bogs down and keels over. I've had to reboot the NAS dozens of times because the entire filesystem had become totally unresponsive.
>You actually can grow a ZFS vdev - exactly by replacing one disk at a time (albeit you don't get the gained capacity until all the disks in the vdev have been replaced).
Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time.
>The kind of storage arrays that grow by adding solitary disks work quite different to your typical RAID. They'll often have designated parity drives instead. See solutions like "unraid".
Not only unraid, there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too.
>surprisingly intuitive command line tools is what edges ZFS for me in terms of usability.
I found the commands anything but intuitive though I guess that's a difference of a few years of experience, I'd agree that would be personal opinion if it's easier.
Was that an off the shelf NAS? If so then I'm guessing it wasn't exactly high spec'ed even putting aside the RAM? eg did it have a 32bit CPU? I've heard that does a massive amount to dent ZFS's performance (I wouldn't know because I've only ever ran on it 64 bit systems).
As I said, I've not had any issues with raidz2 on 2GB RAM. I've even ran virtual machines on one OpenSolaris box with ZFS and 2GB RAM so these weren't always dedicated storage servers either.
edit: just read elsewhere you were running FreeNAS on a Pentium 4. Yeah that would do it. FreeNAS wasn't optimised for low end systems (in fact they outright tell you not to!) and the P4 is pretty low end CPU as well. I still think ZFS would have ran on that but you'd have been better off going with vanilla FreeBSD instead.
> Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time.
...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system. That process can literally take several days on some storage servers. So ZFS might require you to add vdevs to the pool (tank) but at least it can all be done on live systems without any downtime. That in itself is a massive boon for ease of use in my opinion.
To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway. And if you are the type of person that would need to do that then you wouldn't go for a traditional raid5/6 style pool either. Thus the argument becomes a comparison of apples and oranges as you'e comparing completely different types of software RAIDing.
> Not only unraid
I didn't say "exclusively unraid". I said "solutions like unraid". I'm fully aware there are other solutions out there.
> there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too.
When you said "there are other systems that provide almost this" and "once you have added enough [disks] to provide parity, the additional capacity will become used too." I really had to question what your actual point of this discussion was because just a paragraph earlier you were calling my points "unacceptable" because ZFS required you to add more than 1 extra disk to provide parity; and now you're saying your preferred solutions also does the same?
I hope I'm just being presumptuous here but it's starting to feel like you're just arguing for the sake of arguing. :-/
> I found the commands anything but intuitive though I guess that's a difference of a few years of experience
Maybe. I've been a Linux and UNIX sysadmin for a fair few years now so experience is not something I'm short of. :)
Honest question: what is your experience of ZFS outside of the troubles you had on your NAS? I could totally see why the problems you described would taint your opinion of ZFS but I definitely wouldn't say what you experienced is typical for ZFS.
In any case, I'm not going to argue that everyone should be running ZFS - it's just technology, you can run whatever you want to run.
OpenZFS is currently working on adding code to allow for vdev expansion[1]. I am a fan of ZFS as well, but I don't get not wanting to admit its limitations. Not being able to expand vdevs is a limitation and it's being addressed.
[1]: https://www.youtube.com/watch?v=Njt82e_3qVo
Or in other words: if growing the vdev is a hard requirement then a traditional raid might not be the best option regardless of ZFS. It’s about picking the right tool for the job rather than moaning that ZFS doesn’t do every job perfectly (which seems to be the argument the GP was making).
No it was a repurposed office computer running a 12 year old Pentium 4 Dual Core (about 2.8GHz per Core). 2GB of DDR2.
As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB.
>...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system.
If you use mdadm and ext4 you can do it online.
>To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway.
There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions.
The home desktop is possibly the largest filesystem market next to "disposable virtual machine".
>and now you're saying your preferred solutions also does the same?
They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage.
On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs.
>what is your experience of ZFS outside of the troubles you had on your NAS?
I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since.
That's pure conjecture. Plus I actually have ran ZFS on a Pentium 4 on FreeBSD. For a while that was my personal backup server.
> If you use mdadm and ext4 you can do it online.
Not according to any of the docs I've read in the past. Happy to be proven wrong if you have a citation though.
> There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions.
> The home desktop is possibly the largest filesystem market next to "disposable virtual machine".
I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al?
The issue I have with your complaints is that you're flip-flopping across a multitude of industries from desktop to enterprise, which seems grossly unfair for making a balanced opinion. I'm not going to deny that ZFS will lack some desirable features compared to some speciality tools but like I said before, it's about picking the right tool for the job rather than arguing that ZFS is garbage because it doesn't win every single argument in every specific speciality.
I mean you're not even willing to take on board any different configurations of ZFS nor alternative platforms it ships on - instead only basing your biases on one failed attempt at running FreeNAS (something which openly states they need beefy hardware to run) on equipment that is several generations old while then comparing that one specific ZFS instance against every alternative tool you've read about. That hardly seems fair, don't you think?
> They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage.
No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air.
But to assume you meant 2TB of extra storage when you add a 2TB HDD: in your situation you don't get redundancy so your data becomes at risk if you have a failure. You can actually expand your ZFS tank the same way if you really wanted (but no-one who cares about their data would want to) by adding a HDD vdev (unraided) to an existing tank.
The issue is when you want to expand an existing raidz while retaining redundancy. To do that on any of the existing FOSS tools I've used, you'd have to either buy more than one HDD or do an offline restripe (as is the case with mdadm). I guess at least mdadm allows you to do it - even if it takes the storage pool offline for several days. Personally I've always opted to buy multiple drives and expand the pool (tank) by adding an additional raidz vdev because that not only expands the storage array but also then gives me another failure option (eg I can then lose 2 disks simultaneously without losing data). As you start to expand the number of disks you have, you equally increase the likelihood that more than 1 disk might fail. So having that extra redundancy disk is desirable to me.
> On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs.
Actually no. You have to stand up two vdevs but a vdev can be a raidz, mirror, a solitary HDD or even a file on an existing file system. So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it).
> I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpoo...
https://raid.wiki.kernel.org/index.php/Growing
ext4 online resize works by invoking resize2fs.
>I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al?
QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs).
>That hardly seems fair, don't you think?
It's the real world experience I had with the software and I think it's totally fair to point out and complain about the shortcomings.
>No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air.
Reread what I wrote, I mentioned adding 2x2TB not 1x2TB.
> in your situation you don't get redundancy so your data becomes at risk if you have a failure
Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data.
And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array.
>So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it).
Again, what Syno offers (and what can be replicated on any Linux system) allows you to add a single drive to expand the storage without loosing parity and more drives to span any drive increases with parity. This allows adding a single drive at a time and at worst you loose any additional capacity without parity yet (in the 1TB/2TB example, that would be loosing 1TB of storage until the second drive is added)
That's resizing the filesystem not the pool (eg `mdadm --grow`). You're also missing the point about what I said. To run those commands you must first `umount` to pool. Hence it being an offline job.
> QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs).
That's completely irrelevant to the point I was making. How many home users do you know who configure their own home raids using LVM and mdadm?
> Reread what I wrote, I mentioned adding 2x2TB not 1x2TB.
You actually said neither. You said, and I quote, "2TB" (ie without any accumulation prefix).
> Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data
You can actually do that in ZFS. In fact what you've just described were is the exact solution that you called "unacceptable" in ZFS!
> And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array.
Indeed. Lots of commercial tools do support that. I thought we were comparing ext4, LVM and mdadm to ZFS though?
I'm guessing you just glossed other my post where I said you were flip-flopping all over the place? It's easy to pick any specific tool ("tool a")and compare it to a thousand other tools and argue that "tool a" is crap because it doesn't work as well as tool b, c and d in very specific edge cases but all you're doing is trying to win an argument via death from a thousand papercuts rather than forming a sane basis for analysis.
To be honest I think it's probably best we give up on this discourse because your points are all over the place. This feels very much like you've got an axe to grind and no practical experience to back up your prejudices.
No you don't need to unmount the pool under mdadm. You can do a live resize of the array. See the link I posted.
>How many home users do you know who configure their own home raids using LVM and mdadm?
Not many, hence me mentioning Syno and QNAP.
>You said, and I quote, "2TB" (ie without any accumulation prefix).
I said, quote, "if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage."
>In fact what you've just described were is the exact solution that you called "unacceptable" in ZFS!
Again, this is not what ZFS does, the mdadm solution stripes multiple RAID arrays over the same drive, which to my recollection is not a recommended configuration in ZFS by a long shot.
>Lots of commercial tools do support that. I thought we were comparing ext4, LVM and mdadm to ZFS though?
The commercial tools I mentioned previously use ext4, LVM and mdadm native functionality.
By default, sure. There is a FreeBSD (I think) guide to tuning ZFS that lets you tune it for low memory. It's a long time since I've done this so I can't remember which settings, but I ran an old machine tuned for its low max memory, with 4 drives in RaidZ, and it was just fine.
The FreeNAS forum doesn't help much with this as the default opinion there seems to be 16GB ECC. :)
As you mentioned, the bitrot. Also, my understanding is that a COW filesystem can be more reliable than a traditional journaling filesystem in a system crash since nothing is overwritten. Plus there is compression and all of the features provided by a COW filesystem.
The risk of bitrot becomes interesting when you talk about long timeframes like a decade.
Considering the capacity and price of a 2TB disk, compression isn't the immediate need of the consumer either, unless they have a NAS, where Syno and QNAP already provide integrated solutions with btrfs and ext4 both.
The modern ext4 journaling is also very battle tested, it's a very simple and resilient physical journal, data loss in ext4 happens largely when the disk is already failing or has been cheap enough to lie about the state of data being written or not.
For the average consumer that sums up to "it doesn't matter enough". Something like bcachefs could change the game by being btrfs+ZFS but without the dataloss risk.
Of course I’ve been using btrfs RAID1, so perhaps bit rot has been silently corrected.
> The risk of bitrot becomes interesting when you talk about long timeframes like a decade.
I think you are making a lot of assumptions here about frequency of bitrot, especially in the face of failing hardware. Also, consumer desktops are not the only use. There are bunch usages where the risk of bitrot is much more concerning. Or were you intending your original post to be only about consumer usage?
>Considering the capacity and price of a 2TB disk, compression isn't the immediate need of the consumer either, unless they have a NAS, where Syno and QNAP already provide integrated solutions with btrfs and ext4 both.
This kind of feels like you are saying that it isn't needed because it is already being used when it is needed. Am I misunderstanding? Consumers may want compression for smaller hard drives they already own in order to get more out of them. Also, compression can speed up disk IO in some workloads.
> The modern ext4 journaling is also very battle tested, it's a very simple and resilient physical journal, data loss in ext4 happens largely when the disk is already failing or has been cheap enough to lie about the state of data being written or not.
The issue with ext4 resilience isn't about quality of the implementation. It is a fundamental limitation of no COW filesystems. Since things are overwritten, if the system crashes mid write you can end up with partial data. COW filesystems always give you the old data or the new data. This can be worked around but that requires that all of your applications do this workaround.
> For the average consumer that sums up to "it doesn't matter enough". Something like bcachefs could change the game by being btrfs+ZFS but without the dataloss risk.
What makes you think bcachefs is going to be any better? A lot of functionality still has to be implemented. What makes you think that the implementation of this functionality will be any less error prone?
You also haven't addressed the large pile of new features provided by BTRFS/ZFS. Snapshotting alone can make it worth the price of admission for consumers. With snapshotting, you can set up a system where rollbacks after any upgrade can occur seamlessly.
You do know how journalling works, yes? Unless the drive lies about the state of data being written, in which case not even COW will save you, a journal is just as good as COW. Exceptions being coding errors but those can occur on COW with the same devastating result.
Maybe you don't know as much about filesystems as you want to appear to?
Yes, I'm aware of how journaling works. The problem with your description is that it requires both data and metadata journaling. This is rarely the case because data journaling requires that you write all of your data to the disk twice (which also requires additional seeks as well). So, nearly all in-place filesystems default to only metadata journaling for performance reasons (this is the case for ext4). This means that, in practice, if you get a system crash mid file write, you will end up with a partially updated file. On a COW filesystem you are always guaranteed the old file or the new file.
> Maybe you don't know as much about filesystems as you want to appear to?
That is possible, although I'm not sure how much I want to appear to know :P. More seriously, this is extremely condescending. Why is this necessary? If you think I'm wrong, just tell me I'm wrong. There is a pretty decent chance I'm wrong about most things. So, condescension shouldn't be required.
I've had a single issue across many EBS drives partitioned with only XFS, in several mdadm configurations. The data-loss occured during a maintenance window, so I'm convinced it was an EBS issue and not XFS.
I've also used it on physical hardware and never experienced dataloss, even losing drives in RAID configurations (I will admit I was lucky enough to never lose 2 drives in my normal RAID10 setup).
I've had several filesystems become corrupted, but the XFS community is extremely helpful, and the tooling that comes with it has never failed me in at least letting me figure out what's actually wrong.
Basically all of them.
> what are the guides like for setting it up?
Choose "xfs" instead of "ext4" in the installer.
> Doesn't seem like the next-gen filesystem we should be hoping for.
XFS was an early journaling file system. It wasn't designed to be next-gen, it was designed to be fast, service soft-realtime I/O demands and handle large volumes well.
We have seen host kernel panics which damage the filesystem such that xfs_repair cannot fix it. It could be mounted read-only, though I am not sure I would trust the content. We just reformatted and rebuilt content from external sources.
What I believe caused this panic was a habit of creating sparse VM disk images. Later, under heavy write loads from multiple VMs, the host OS filesystem seemed to choke under all the out-of-order allocation needed to place new data into those sparse disk images. When we instituted a policy of always preallocating the VM disk files, we never saw this problem repeated.
That said, it should only cause memory allocation deadlocks, not corruptions.
Is the underlying code terrible? I haven't heard any particular concerns about code quality (aside from the original implementation of RAID 5/6). Most of the issues I've heard about were due to people using features that weren't ready yet. I've heard a lot fewer of these issue since BTRFS started providing clear statuses for individual features[1]. Do you have a different impression?
[1] - https://btrfs.wiki.kernel.org/index.php/Status
Its totally anecdotal to say just because my ~10 hard drives with it being stable for two years means its improved but I'd consider putting some faith in it now. A bit. With backups.
in addition, we know that there are patent landmines out there because NetApp and Oracle have a (undisclosed) settlement on the issue. Using ZFS is a probably good way to get a visit from your legendary friendly Oracle sales agent and auditor. Oracle could fix all of this with a quick re-license or patent indemnification, but they haven't and almost certainly won't.
My Synology (and all modern Synologies) uses BTRFS, and it seems pretty solid so far. I would have preferred that ZFS had matured and been dominate in FS, but there are some really risky things that come with ZFS.
The CDDL also includes a patent agreement, Oracle can’t retroactively waive it.
Go look at what the Java community is going through right now.
You did not understand what happened with the latest Oracle Java news and you are spreading FUD.
People have a choice what Java implementation to use, they can pay for latest Oracle implementations, use Open JDK (it is included in Debian and most distributions), you can get support from RedHat and other big vendors. There is no risk of getting in trouble by using Open JDK, MS and Google had trouble because they "forked" Java
I've been using it on a 6 drive NAS since 2012, and on two laptops with SSDs.
In my experience "crashes" are symptoms of people who didn't read the documentation and don't understand how to recover.
For example, on the laptops I've often run out of disk space. And btrfs can run into a situation where it can't clean up partly used blocks without more disk space. To recover you have to add more space. I usually used the swap partition, but a USB stick also works. Or a RAM drive, but that's risky.
Some other issues I've had to help people with is snapshot deletion. Deleting a btrfs snapshot appears instant but none of the used space comes back. That's because it is doing it in the background and it can take a really long time. OTOH, ext4 does exactly the same thing with background delete of very large files.
aka most users. Filesystems must be reasonably resilient to incorrect usage. This is a problem across the board; if you're not careful setting up a ZFS pool, it's easy to accidentally create a scenario where you're unwittingly striping across multiple devices, greatly increasing the likelihood of a catastrophic data loss event without ensuring the user is fully aware.
Yeah that's every filesystem ever. I helped a guy out just a couple of months ago because his Debian virtual machine had plenty of disk space but he couldn't install anything.
Because it was out of inodes. What "regular user" aka Nodejs / React developer knows about inodes? Not that guy.
So I guess ext4 is junk too.
I run exclusively with mirrored vdev’s and have never run into issues growing my pool.
Perhaps you're thinking of shrinking, which it still cannot do without rebuilding the whole FS infrastructure.
My opinion after trying both parity-checked redundancy and true mirrored redundancy is that any serious data storage should be strictly on mirrored volumes (preferably with disks of substantially different ages so you don't risk them dying within a couple weeks of each other, possibly before a replacement gets installed).
I once read "50% storage efficiency is plenty, don't be greedy". Although painful for the home user, that seems like the right attitude to take for important stuff, especially as volume sizes increase. It can be a very slow and very fraught process to recover several TBs from parity data.
I had a VM running OpenSuse Tumbleweed - I know, rolling release, don't expect stability, etc - and it would consistently freeze after being up and unused for a few days. Rebuilt the VM but used XFS instead of Btrfs, it was rock solid. This happened within the last four months.
At work, we have a number of machines using Btrfs. At some point they get into a state where the weekly Btrfs scrub process hangs the machine for hours at a time. The solution in every case was to move that filesystem onto a non-Btrfs volume. This is an ongoing issue, but I'm not in the team responsible so I don't know what they're doing with this. Anecdotally, this seems to be a problem when there is a lot of writing going on with the volumes. Machines that use Btrfs root but do most of their work on NFS volumes don't seem to experience this issue.
I also have a laptop running OpenSuse Tumbleweed with Btrfs. One day it somehow got itself into a state where it would hang when mounting the filesystem read/write, but it would be fine if it was mounted read-only. I can't recall exactly what happened to cause it, but I'm pretty sure I didn't do anything exceptional with the computer - I think I rebooted it from the GUI while it was doing a scrub in the background, but that's just a guess. I think this happened about a year ago.
This is all disappointing because some of the things Btrfs enables are awesome. In particular, OpenSuse's snapper has saved me countless hour of work. I haven't played with the send/receive stuff, but it seems like a great solution for backups.
In the meantime, I'm using ZFS on a variety of FreeBSD boxes with a variety of workloads (including one running more VMs than its RAM should support) and it's been rock solid.
At this point it will be a long time before I give Btrfs another look. It has the potential to be really good, and I want it to be good, but I don't believe it's there yet.
A great example is that if you ever try to run a database or a large virtual machine on a btrfs mount, you'll quickly become acquainted with chattr +C for "nocow". Any write-intensive applications grinds the filesystem to a halt until you disable the copy-on-write functionality for that file specifically, making it so snapshots, etc., are ineffective and the FS stops tracking the changes that would make them possible.
Whereas on ZFS, like any application that uses fixed block sizes, virtualization and databases _may_ require some alignment tuning to avoid a read-modify-write cycle, and ZFS works without fuss from that moment on.
I know that Facebook is investing heavily in btrfs right now, but to be frank, I don't see the point. btrfs doesn't have its youth as an excuse anymore and the core issues that exist seem like they're pretty fundamental.
It's a gargantuan waste of effort when people could just take the tack that Ubuntu has taken and decide all that FUD around the CDDL, kicked up by fears that the commercial Unices were plotting to take their revenge on Linux, that maybe all that isn't super relevant anymore, and they can just start shipping ZoL.
1. https://code.fb.com/open-source/linux/
Today I'm running ext4 because maintaining ZFS setups is exhausting since it is not upstreamed.
Still... I hope there's a future in one of these volume-manager-meets-filesystem filesystems. Maybe bcachefs will end up being the answer some day?
People who have been running btrfs in production (reckless, imho) on kernels that are 2.5+ years old or worse. And then they're surprised and disappointed when it loses data.
For everyone who's posted here with a btrfs data loss story, it would be helpful to quantify your anecdote with what kernel you were running.
As far as why, https://code.fb.com/open-source/linux/ has a nice quote:
"Btrfs has played a role in increasing efficiency and resource utilization in Facebook’s data centers in a number of different applications. Recently, Btrfs helped eliminate priority inversions caused by the journaling behavior of the previous filesystem, when used for I/O control with cgroup2 (described below). Btrfs is the only filesystem implementation that currently works with resource isolation, and it’s now deployed on millions of servers, driving significant efficiency gains."
(I don't know though)
It's very common for MegaTechCos to hire maintainers like this, but us in the hoi polloi need to be cautious not to jump headfirst into a situation where we can't call upon the experts/inventors to get us out of any jam. We have to rely on our own abilities, the mailing list archives, and hopefully some relevant random blog posts that aren't too outdated.
This should be taken into consideration much more often than it is. If anything, maintainer employment by a MegaTechCo should nix out their implied endorsement of the technology rather than taken as proof that it's viable.
Btrfs was a known bug, and the kind folks on #btrfs were able to help me recover (though it was a quite involved process).
For ZFS, the consensus in #zfs and #zfsonlinux was that with the error message I was getting "I hope you have backups."[1]
I see XFS mentioned here, but in the admittedly unusaly use-case of editing a file that when compiled and modprobed may hang the kernel, I found that vim was insufficiently fsync-ing to ensure I had valid data (either old or new copy would have been acceptible).
I don't know what my point was with this comment except "All filesystems are buggy, so backup your data"
1: It should be noted that in the case of ZFS, I was on vacation and it was my laptop that had no critical data, but I absolutely needed it working for other things, so I only spent a couple hours trying to recover; The btrfs issue happened at home where I had plenty other systems to use and where I was more motivated to recover the data.
My current backup server would need ~50GB of RAM just for the in-RAM deduplication tables.
I've been thinking of trying btrfs for the backup server, but the deduplication story is not something that's obvious, it all seems to be third-party, which makes me skeptical. But it worth trying. I might just try a test Dragonfly setup, because my simple testing of HAMMER seemed very promising, years ago.
I've had pretty bad experiences with btrfs in the distant past. Around a decade ago I had all my company laptops using btrfs and it worked great, as long as we didn't get >80% usage or something. It worked well for around a year. But then the reliability of btrfs dropped, and we had repeated data loss and switched back to ext.
btrfs wasn't considered stable until 2012. Using an experimental filesystem for all your company laptops seems to be asking for trouble.
There are some other types of dedup that don't guarantee 100% dedup, but then the dedup levels would very depending on data sets -- this may be fine for VMs and user data. They work by being using data locality, predicting which dedup table would needed to dedup the next N bytes. But it is not likely we are going to see this any time soon because the patents on this sort of dedup is fairly recent.
This is only true for real-time, "live" dedup, which is an essential feature for almost nothing.
btrfs' offline dedup (which, despite the name, can be done on a mounted filesystem) is far more rational. You fill the disk with data, and then dedup it later, at a schedule that works for you. No extra RAM needed.
You still have to process it, and you are still going to use a lot of ram -- and you are going to want to use a lot of ram once you get into larger data sets otherwise the process will take days.
Those customers can't redistribute Solaris any more than they can redistribute Linux with a CDDL-licensed ZFS. If they really wanted to switch to Linux they could do so today and be in the exact same legal position license-wise. Not to mention they could switch to FreeBSD and be able to redistribute, but again, I doubt this is something very many Solaris users care about. Perhaps the cost of porting their applications away from Solaris is too high, or their middle managers are Oracle fanboys, whatever the reason is, a GPL-compatible ZFS does not threaten that business model.
I think a more likely reason why they pour money into the Btrfs death march while having a turnkey solution just sitting around in the next office over waiting to be singed off by the legal department as CDDL+GPL or Apache or whatever is that each project is the pet of a different pointy-haired boss and they refuse to cooperate internally.