As far as I can tell, building a similar system using NVMe is vastly more complicated. If you can fit everything into M.2 slots, it’s easy. Otherwise you need barely-standardized PCIe cards, backplanes, connectors ordered off a menu of enterprise things with bizarre model numbers, and possibly “RAID” cards even though you don’t actually want hardware RAID.
Nowadays some recent consumer platforms supports PCIe x16 to x4/x4/x4/x4 bifurcation that should be enough. For RAID, switching (dynamically shares bandwidth between SSDs) is nonsense, isn't it?
22TB of NVMe drives is going to be a bit more expensive than the system in the article however.
I do wonder what the power consumption figures would be though. His system was drawing an annoyingly large amount of power and I suspect that was mostly those HDDs.
I also run a home NAS in a Node 304. I went with a supermicro mainboard for ECC support which means I had to swap the three fans that come with the case because the mainboard only supports PWM fans. Non-PWM fans would only spin at full speed otherwise.
Regarding the SLOG device, you probably don't need it for a file server, but if you do you can definitely free a drive bay for an HDD by just using double sided tape somewhere like on the PSU. I'm sure it's also possible to put three more HDDs above the CPU, right in front of the exhaust fan. If I had a 3D printer I would try to build something bringing the total to nine HDDs.
If you need more SATA ports but are running out of PCIe slots, you may be able to reuse an empty M.2 slot. An M.2 with two lanes of PCIe 3 gives you 5 SATA ports with an adapter[0].
I occasionally look at the PCIe-to-SATA market and find it confusing. It appears polarized into either cards from a reputable brand and very expensive, even with few SATA ports, or cards from an unknown brand and relatively affordable. What's your experience with this and what can you recommend (2-port or 4-port)? Are the cheap cards safe & reliable or are they to be avoided?
I've been running older SAS cards for years and it's been doing just fine. They go for cheap on eBay. Each SAS port serve four SATA drives, using SAS-to-SATA cables.
Just make sure to get one that runs in IT mode or you have to mess with the firmware.
> Just make sure to get one that runs in IT mode or you have to mess with the firmware.
In case some people wonder what "IT mode" is, as I used to some years ago, what you basically want is a card that will expose the drives directly to the OS, as opposed to "volumes".
In other terms, if the card is a RAID controller, it may insist on you creating arrays and only expose those. You can circumvent it by creating single-drive arrays, but it's a pain.
Some cards can do both, but it's usually not advertised. Non-RAID cards also tend to be cheaper. Others (usually LSI) can be flashed with a non-RAID firmware, but again, it's less of a hassle to not have to do it.
Basically: they can be buggy. Either they work fine for you, or you hit an edge case and have trouble. They can also have worse performance as they only have one SATA controller and split it amongst the drives.
If you're doing this at home, you can get used enterprise gear on eBay (like an LSI SAS HBA) for the same price or cheaper than brand-new consumer gear, and it will probably still be more reliable (I built a 130 TB NAS for my friend's video production business and literally everything aside from the drives and the cables was bought used on online auction, and it's been humming along fine for a while now - the only part that was bad was one stick of RAM, but the ECC errors told me that before I even got around to running my tests on the sticks)
> I see people talking about snapshotting, but I haven’t found a need for it. I already have snapshots in my restic backup solution. They’re not especially convenient, but I’ve been using restic for two years, and I only recall needing to recover data from a snapshot once.
The ease at which you can revert mistakes using ZFS snapshots is much better compared to restic. You can pretty much navigate to the correct snapshot on your live filesystem and restore whatever you need to restore.
It also makes backups easier as you can just send the snapshots to the backup device (another server or external storage device).
I never do "rm prefix" in a dir. I always do "rm ./aDir/prefix" for example. This assures I'm not globbing outside a directory (or just a directory) and tries to help assure I'm not shooting my own foot.
Yea, i love up 1 directory before I delete anything.
This. I'm always going into my backup server and looking in the ".zfs/snapshots" directory to look at the history of how files have changed over the backups. Love restic, but the ZFS snapshots are fantastic.
If you setup SMB right (TrueNAS configures this out of the box, SCALE is great if you need a Linux NAS), you can use Windows Shadow Copies to access ZFS snapshots and browse or restore file contents from them.
Also possible with BTRFS. I set this up once for a small business with hourly snapshots during working hours. This way users could just restore older versions of files they accidentally deleted, overwrote or messed up in some other way.
Another benefit: Those snapshots were read-only, so they also served as a protection against ransomware.
I don't think BTRFS supports NFSv4 ACLs yet (ie, Windows ACLs are natively supported on ZFS, there is patchset so Linux also supports it but BTRFS obviously has no integration for a patchset that only exists for ZFS).
Having NFSv4 ACL access is a huge plus since you can configure permissions natively from windows and have them enforced even on the shell.
They likely use XATTRs to store the ACL (that is an option in Samba), but it's not native like it's on the TrueNAS systems with the kernel. I bet if you log into the Syno's via SSH you don't get the ACLs enforced on the shell. With the NFSv4 ACL patchseries, they would and you could benefit from the better options that the NFSv4 ACLs give you.
Storing them in metadata is not the same as having them natively.
They maintain their own kernel module to handle ACLs (synoacl_vfs) and they are indeed enforced locally as well. They can be read and modified by using the `synoacltool` cli.
Is that kernel module open source? One of the advantages of what TrueNAS did is that I can patch it into my own kernels if I'd need to. Plus being compatible to the NFSv4 ACL binary format, so it works via NFS too. Also handling Active Directory would be important there.
Not sure about the module being open-source, but running custom kernels is not really a thing on Synology. They seem to integrate SoC BSPs into kernels for each specific model, and they do not seem to port them across versions. Different models use different kernel versions in the same DSM (Synology distro) version.
The ACLs do work via NFS and it also works with Active Directory. They ship an AD implementation too, if you are interested in that (it is actually Samba in AD mode).
Well, TrueNAS Scale isn't exactly designed to be run on a SoC, it's more of a normal linux distro for NAS/SAN Servers. Hence most of it is open source and there is active upstreaming efforts. Plus the entire ACL thing is native to ZFS already, it's just the glue layer that's missing in Linux. For Linux it's presented in an XAttr for compatibility, for Solaris it's part of the proper permission bit fields.
> I chose raidz1. With only a handful of disks, the odds of two drives failing simultaneously is fairly low.
Which is not really the case of you bought x amount of the same disks and always use them together. I had that happen to me just a few months ago. 4 identical discs bought at the same time. Raidz1 reported one dead/dying disk so I replaced it and started resilvering, which can take days and leaves the disks at 100% utilization.
So after 12 hours or so a second one failed and the data was gone.
In my case even mixing up the disks might not help but I agree it’s still helpful.
I bought 4x Seagate Ironwolf Pro 12TB drives from different vendors, one failed after a year, then when I got the replacement another drive failed during the rebuild, and then 6 months later the replacement failed. Now another one of the original drives is also reporting reallocated sectors.
Same system has 4x WD Red drives which have been running fine with 0 reallocated sectors for almost 7 years.
I'm okay with claims that snapshots are much better than backups for many uses. But in this case the GP was explaining that they only used their backups once in several years, so they did not need to change their backup system.
I'm in the same boat. I configured remote backup systems on a handful of computers. I think I reached for backups only twice over the last ten years. Of course I need something, backups or snapshots, but for my use case snapshots (with a network copy) would need work to set up. And if the remote storage is worse, that would be more of a problem than the changes in the restore process.
Get LSI 2000 or 3000 series SATA cards. Several manufacturers make them approximately to the reference spec. The drivers are in the Linux kernel. If they don't come flashed to the IT spec firmware (no RAID capabilities), do that, but the cheap ones usually do. The 4i models sometimes come with ordinary individual SATA connectors; the 8i will have one of two kinds of combo connectors that can accept cables that go to a backplace or breakout cables to ordinary SATA connectors.
Nice build. I recently built my second NAS (from a used R720 from ebay). The total (without disks) is pretty similar to the build documented in this article.
Having a large NAS has had an interesting (though predictable) impact on all the computers around it: Pretty much every bit of data lives on the NAS (accessed either by CIFS, NFS, or iSCSI). When I had to reinstall Windows, it was mostly painless because all my important data and Steam games library was on a remote iSCSI disk. When I replaced the drives on my linux servers, I didn't have to backup hardly anything, as I worked almost exclusivly on NFS-mounted directories. When bringing up a new raspberry pi for projects, it also has instant access to more terabytes of storage than it could ever need.
Also, for a homelab, getting 10GBe fiber between two machines is surpringly cheap and easy. For certain workloads, it can be a noticable speed boost over 1GBe.
How is performance, especially with regards to load times, if your steam library is mounted remotely?
I ask because the difference between a SSD and a hard drive can be massive in this regards, so I'd be really interested to know if the network latency is also a comparable hit.
I'm not a hardcore gamer by any means, but I really wonder how much influence drives actually have on games.
My gaming computer had an old SATA SSD (Samsung 840 Evo IIRC). Some games took ages to load (particularly Fallout 4). I switched to a much faster NVME drive, and subjectively, it's not any faster loading games. I'd say this was a very underwhelming purchase.
There's certainly diminishing returns between "an SSD" and "a faster SSD" (unless the slower one is DRAMless or QLC), but hard drive to SSD is still a big gulf
The performance is fine. It's been years since I ran my steam library on HDD, so I don't have anything really to compare to (except my own expectations and impatience). The NAS is running 7 SSD drives in zfs raid, and exports a 1TB volume over iSCSI, via a dedicated 10GBe fiber link. Anecdotally, I will often boot into games faster than friends who have a local SSD (so I think this means that I've gotten disk perf to be "good enough" that other hardware components start to dominate things like load times)
Nice article - though one misstatement is that ZFS dies not allow you add disks to a pool. It does [1] [2], by adding new vdevs. The linked issue is about adding support for expanding existing vdevs instead.
It's a hard pill to swallow, adding two more drives in a vdev to get one more drive worth of storage (authors case maxes out at 6 drives, currently has 4). So often you will bite the bullet and just completely rebuild.
True RAIDz expansion is something that's supposed to be coming, possibly in Q3 2022, so it may be that by the time one needs to expand a volume, that ability will have landed. That'll be a game changer.
However, such a drive is getting heavily into diminishing returns territory.
e.g. a 20TB drive from Seagate is $500. A 4TB drive is $70, 8TB is $140. Getting the same spend in smaller capacity drives would give you 28TB in the 4TB drives and 24TB/32TB in the 8TB drives (for $80 under/$60 over).
Add in a second to rotate and you're spending $1000 in drives, assuming these 26TB drives replace the 20TB drives at a similar price when they trickle down to consumer hands.
You have to factor in the power usage of having multiple drives spinning. Though I’d agree that smaller drives are better when you have a drive failure, as resilvering is quicker.
"CephFS supports asynchronous replication of snapshots to a remote CephFS file system via cephfs-mirror tool. Snapshots are synchronized by mirroring snapshot data followed by creating a snapshot with the same name (for a given directory on the remote file system) as the snapshot being synchronized." ( https://docs.ceph.com/en/latest/dev/cephfs-mirroring/ )
We found ZFS led to maintenance issues, but it was probably unrelated to the filesystem per say. i.e. culling a rack storage node is easier than fiddling with degraded raids.
Buy another, use ZFS send/receive. It's only double the price! Better yet, put it elsewhere (georedundancy). With ZFS encryption, the target system need not know about the data.
For critical data though I use Borg and a Hetzner StorageBox.
I have a nightly restic backup from my main workstation to buckets on Backblaze and Wasabi. It backs up the few local folders I have on my workstation and all the files I care about on my NAS, which the workstation accesses over Samba. I've published my scripts on Github.[0]
I don't back up my Blu-Rays or DVDs, so I'm backing up <1 TB of data. The current backups are the original discs themselves, which I keep, but at this point, it would be hundreds of hours of work to re-rip them and thousands of hours of processing time to re-encode them, so I've been considering ways to back them up affordably. It's 11 TiB of data, so it's not easy to find a good host for it.
Yes but there’s a price/reliability/performance trade off.
Also, with disks that big failures become qualitatively different. For example, when a disk fails in a mirror, the bigger the disk size the higher the chance the 2nd disk will have unreadable blocks.
It's almost entirely because people really like technology that not only promises to reward you with excellent features and stability, it follows through on those promises.
Because people don't know about cephfs yet, and the silliness of degraded raid setups. i.e. trusting a zfs community edition in a production environment can be a painful lesson.
;-)
I'm sure it'd be painful, but let's throw "infrequent" onto your description of the lesson. :-)
I've run ZFS for home storage and work backups for ~15 years, across Nexenta, ZFS-fuse, FreeBSD, and OpenZFS, backing up hundreds of machines, and have never lost data on one of them.
I know about CephFS, but performance was abysmal compared to ZFS for a home server. On a single box with 4-8 drives I didn't come close to saturating a 10G link, which ZFS managed just fine.
It was also very complex to manage compare to ZFS, with many different layers to consider.
I'm sure it shines in a data center, for which it has been designed. But unless something radical has changed in the last year, it's not for a budget homelab NAS.
The cephfs per-machine redundancy mode is usually the preferred configuration.
i.e. usually avoids cramming everything into a single point of failure, buying specialty SAS cards, and poking at live raid arrays to do maintenance.
Seen too many people's TrueNAS/FreeNAS installs glitch up over the years to trust the zfs community edition as a sane production choice. ZFS certainly has improved, but Oracle is not generally known for their goodwill toward the opensource community. ;-)
I've never run TrueNAS/FreeNAS in proper production, but I have run it at home for over a decade and never lost data, despite generally running on old hardware, multiple drive failures, motherboards dying and power outages/lightning strikes.
Overall been very little fuzz for my home NAS system.
Thanks for including energy usage in the article. I carry USB-C SSDs around the house for backups and storage of archived files. Of course this is a bit of a hassle and I played with the idea of either buying or building a NAS. My current standby consumption for all computer stuff (access points, router, switches, a dozen or so microcontrollers and various SmartHome stuff, but not TVs, running computers or gaming consoles) is already above 100w and I would really like to bring this number down. An extra 30-60w makes it really hard to justify the purchase of a NAS ( that I don’t really need). I thought at least the synonogies would use way less power when not in use, so thanks for making me aware of this.
I agree and plan to buy a Kill A Watt P4460 meter. My HPE Gen 9 servers were free, but I still would like to know the operating cost of a single server.
Yeah, I've never thought much about power consumption, but I've done a few write-ups of previous builds, and I received a lot of questions about power draw, so I decided to measure it on this one. I was surprised at how much power the system consumed, and it will be something I think about more up-front on future builds.
If you are not after speed, then you can do a redundant array of cheap nodes. Instead of using raid, just shove in an 8-12tb disk in a number of thin clients.
The key is that they spend most of the time turned off.
I think using 1 disk redundancy is a mistake. It’s not only physical failure you’re worried about, it’s an error upon rebuild when you lose a drive. Bit rot on your remaining drives can occur which wouldn’t be detected until rebuild time when you lose a drive, and that could cause you to lose your entire volume. Bit rot can be checked for but you can’t always be sure and with larger and larger sets of data it gets slower to do.
I use raid 6 and also backup my data externally to another nas as well as backup to a static usb drive. Backup requires multiple different types since failures are so catastrophic and can occur in ways you don’t expect.
> I chose raidz1. With only a handful of disks, the odds of two drives failing simultaneously is fairly low.
Only if you buy different hard drives or at least from different production batches. I had a lot of trouble on the same premise and I won't make that mistake again.
Edit: He mentioned it though ( a bit later in the article)
> The problem is that disks aren’t statistically independent. If one disk fails, its neighbor has a substantially higher risk of dying. This is especially true if the disks are the same model, from the same manufacturing batch, and processed the same workloads. Given this, I did what I could to reduce the risk of concurrent disk failures.
> I chose two different models of disk from two different manufacturers. To reduce the chances of getting disks from the same manufacturing batch, I bought them from different vendors. I can’t say how much this matters, but it didn’t increase costs significantly, so why not?
> I had a lot of trouble on the same premise and I won't make that mistake again.
Please elaborate, I'd love to hear your story!
I hear a lot of advice around raid/z-levels and it often seems backed up by shaky math that doesn't seem to be backed up by reality (like the blog posts that claim that a rebuild of an array of 8 TB drives will absolutely have hard read errors, no exceptions, and yet monthly ZFS scrubs pass with flying colors?)
I've gone the TrueNAS route, but I'm running it on a QNAP TS-451. I'm running TrueNAS off of a USB stick hanging off the back, so I didn't have do to anything with the hardware and reverting back to QTS is just a matter of setting the boot order in the BIOS.
I really like seeing other people's builds, but I know that building my own computer isn't something I want to do. I was happy to see the comparison between the DIY model and the roughly-equivalent commercial units. I'll likely buy another QNAP (to run TrueNAS on) when the time comes, and the comparison tells me that I won't get screwed too badly by doing so.
>I've gone the TrueNAS route, but I'm running it on a QNAP TS-451. I'm running TrueNAS off of a USB stick hanging off the back
Oh, I didn't realize that QNAP allows that. Synology makes it pretty hard to boot any other OS, and I assumed the other vendors were similar. I'll keep that in mind for my next build because I do have fun building servers, but I also really appreciate systems like Synology and QNAP where the hardware and case is optimized for the NAS use-case.
>If you’re new to the homelab world or have no experience building PCs, I recommend that you don’t build your own NAS.
>Before building this system, I had zero experience with ZFS, so I was excited to try it out.
Sorry, but this is amusing to me. ZFS on TrueNAS is probably fine, but you're building your production NAS, to replace the Synology device you've become "so dependent on". Don't become dependent on ZFS without knowing the implications!
I was facing this choice recently, and I agreed with the other tech savy person in the household that we should just use good old LVM + Btrfs. Not only does it run like a charm, but it also allowed us to switch the LV from single (during the data move) to RAID 1 and eventually to RAID 5/6 with zero issues. It will also be much easier to recover from than ZFS.
On another note, it's a bad market to buy NAS drives, especially from Seagate. Seagate Exos drives are at this point in time often cheaper than IronWolf, even non Pro IronWolf. They're slightly more noisy and don't come with the free data recovery, but otherwise they're a straight upgrade over the IronWolf drives.
My tiny NAS is still using ext4 + md raid1. It’s the third incarnation of essentially the same design (previously used raid10 when drives were smaller).
When it fills up, I delete some files rather than adding disks.
Has someone yet created open source patches for LVM + Btrfs like what Synology does to pierce the layers and use the btrfs checksums as a tie-breaker to tell lvm what disk to trust to repair errors?
I'm not sure. We use the raidintegrity option, which layers dm-integrity on top of the raid. We then run btrfs scrubs periodically to trigger reads on the device.
What is the technical upside of using TrueNAS instead of samba? If you want to optimize for control, it seems a bit weird to me to settle for an "all in one" software stack.
I see, so I assume the upside is that it's a time saver. Thanks! I personally wen't with samba on Linux and with btrfs. I was wondering if there's something non-obvious in TrueNAS that I'm missing out on.
And to my account, I think my upsides are that:
- ability to choose the kernel
- no need for SSD for base OS since running off of RAM is
rather easy on Linux
- samba can run in a container thus a bit more control security-wise
- server may run something else as well
Of course, this comes with a lot more technical hurdles. More like a side-project than utility really. That's why I was wondering does TrueNAS provide non-obvious upsides that would be lacking in self-rolled one.
There are two flavors of TrueNAS - Core and Scale. Core is basically a FreeBSD distro and Scale is basically a Linux distro. They're both a base OS with the typical packages anyone would need for a NAS, with sane defaults + a user-friendly web-based management system.
The upsides are that it's plug-and-play for anyone who doesn't want to research all the options available and figure out the various pitfalls on their own.
> no need for SSD for base OS since running off of RAM is rather easy on Linux
I don't understand this sentence. You're running off a RAM disk with no boot drive? What if you have a power outage?
> samba can run in a container thus a bit more control security-wise
Core supports FreeBSD jails and Scale supports Docker so you could run samba in a container on either if you're willing to do set it up yourself.
> server may run something else as well
As before, both have jail/container functionality. I haven't used Scale myself but Core comes with a bunch of "click to install" jail options for stuff like Plex, ZoneMinder, etc. Our machine also runs a Windows VM (ew) and a Wordpress install in a Jail
Thanks, this is a great explanation! I wish the blog post would have described the TrueNAS like this.
> You're running off a RAM disk with no boot drive? What if you have a power outage?
Yes, the server only has the HDDs which contain the NAS data. The server bootloops until it gets an image from the router (ipxe boot). The disk images have systemd scripts which install everything from 0 on each boot. Coincidentally, this means system restart is how I upgrade my software.
> Core supports FreeBSD jails and Scale supports Docker
This clarifies the situation -- TrueNAS seems like an option that I would recommend for anyone who wants a quick OSS NAS setup.
I consider myself an intermediate homelabber and a TrueNAS beginner. I just built my first NAS server, so wrote this to capture everything I wish I'd known at the start. I hope it's helpful for anyone else thinking about building their first NAS server.
Any questions or feedback about the post are more than welcome.
Just wanted to share my appreciation for not just this post but all your work in recent times! Been following your trail since your post about Google promos and the set of useful projects you've been working on since then.
Some suggestions for anyone else looking to do the same:
i3 runs a bit cooler than ryzen, still 8 threads. 8tb WD blues (they're SMR at 8 and up). You can find Atx boards with 8 sata ports and dual nvme slots for caching / fast pools.
I'd be really careful about SMRs in a RAID. You can end up with no end of performance issues. It's all the downsides of a single SMR drive multiplied by however many drives are in the pools.
You actually have to be careful. There's disk sizes where the manufacturer will do say 8TB CMR and essentially the same drive with different firmware as a 10TB SMR. They'll also have a 10TB CMR model. You have to pay close attention to the model numbers. It's even more of a crapshoot if you shuck drives. You have to carefully research what externals are known to have CMRs.
A few points from someone with years managing raid and ZFS in arrays all the way up to 50 disks:
RAID-Z1 is something I never consider without a solid backup to restore from and a plan to execute that process at least once in the lifecycle of an array.
If you suffer a total disk failure of one of those disks in the array, you have likely lost some data. The good news is that ZFS will tell you exactly which files you have lost data for and cannot rebuild. If you have those files, you can overwrite them with the backups to get your integrity back.
The reason is, with a total loss of a single disk, any read error on any of the remaining disks is a lost/corrupted file.
For this reason, you need a strong(easily accessible, consistent, current) backup strategy and an acceptance of downtime with Z1.
As for ECC, it's better, but your absolute worse case scenario is that you get a bit flip before the sync and hash happens, and now that bit flipped data is committed to disk and you think it's OK. I prefer ECC to avoid this, but you are still reaping a multitude of benefits from ZFS without ECC.
The only valid rule for RAM and ZFS is that more RAM = more caching of recently read data. Single, or very few user appliances will see little benefit past 8GB even with 100TB unless you happen to be reading the same data over and over. Where ZFS shines is having hundreds of gigabytes of RAM and tens or more concurrent users mostly accessing the same data. That way the vast majority of reads are from RAM and the overall disk IOPS remain mostly idle.
Most of the ZFS RAM myths come from Deduplication, which should be disregarded as a ZFS feature until they allow storing of the DDT on a Optane-like latency device. Even better would be offline deduplication, but I doubt that will be a thing in ZFS this decade.
> If you suffer a total disk failure of one of those disks in the array, you have likely lost some data. [...] The reason is, with a total loss of a single disk, any read error on any of the remaining disks is a lost/corrupted file.
Wait, what? If a RAID-(z)1 ZFS array loses one disk, there's data loss? I've ran so many RAID-1 and RAID-10 arrays with mdadm that I can't even being to count them, and I had many drive failures. If any of those arrays would have corrupted data, I would have been mad as hell.
What I am missing here? How is this even remotely acceptable?
> any read error on any of the remaining disks is a lost/corrupted file.
That is the meat of it. With traditional RAID it is the same issue, except you never know it happens because as long as the controller reads something, it's happy to replicate that corruption to the other disks. At least with ZFS, you know exactly what was corrupted and can fix it, with traditional RAID you won't know it happened at all until you one day notice a corrupted file when you go to use it.
RAID-Z1 is better than traditional RAID-5 in pretty much every conceivable dimension, it just doesn't hide problems from you.
I have encountered this literal scenario where someone ran ZFS on top of a RAID-6(don't do this, use Z2 instead). Two failed drives, RAID-6 rebuilt and said everything was 100% good to go. A ZFS scrub revealed a few hundred corrupted files across 50TB of data. Overwrote the corrupted files from backups, re-scrubbed, file system was now clean.
ZFS automatically self-heals an inconsistent array (for example if one mirrored drive does not agree with the other, or if a parity drive disagrees with the data stripe.)
ZFS does not suffer data loss if you "suffer a total disk failure."
I have no idea where you're getting any of this from.
If the data on disk (with no redundant copies) is bad, you’ve (usually) lost data with ZFS. It isn’t ZFS’s fault, it’s the nature of the game.
The poster built a (non redundant) zfs pool on top of a hardware raid6 device. The underlying hardware device had some failed drives, and when rebuilt, some of the underlying data was lost.
ZFS helped by detecting it instead of letting the bad data though like would normally have happened.
The parity cannot be used in the degraded scenario that was under discussion.
See eg here where the increasing disk size vs specified unrecoverable read error rate is explored in relation to the question at hand: https://queue.acm.org/detail.cfm?id=1670144 (in the article Adam Leventhal from Sun, the makers of ZFS, talks about the need for triple parity).
Also, the conclusion "ensure your backups are really working" is an important point irrespective of this question, since you'll also risk losing data due to buggy software, human errors, ransomware, etc.
You're not missing anything. They're completely wrong.
In RAID-Z, you can lose one drive or have one drive with 'bit rot' (corruption of either the parity or data) and ZFS will still be able to return valid data (and in the case of bit rot, self-heal. ZFS "plays out" both scenarios, checking against the separate file checksum. If trusting one drive over another yields a valid checksum, it overwrites the untrusted drive's data.)
Regular RAID controllers cannot resolve a situation where on-disk data doesn't match parity because there's no way to tell which is correct: the data or parity.
The situation I laid out was a degraded Z1 array with the total loss of a single disk(not recognized at all by the system), plus bitrot on at least one remaining disk during resilver. Pairity is gone, you have checksum to tell you that the read was invalid, but even multiple re-reads don't give valid checksum.
How does Z1 recover the data in this case other than alerting you of which files it cannot repair so that you can overwrite them?
Why do you have bitrot to begin with? That's what scheduled scrubbing is for. You could of course by very unlucky and have a drive fail and the other get corruption on the same day, but check how often you find issues with scrubbing and tell me how likely that scenario is.
I've had hundreds of drives in hundreds of terabytes of appliances over years. URE and resilver is a common occurrence, as in every monthly scrub across 200+ drives. This isn't 200 drives in a single array, this is over 4 appliances geographically distributed.
The drives have been champs overall, they're approaching an average runtime of about 8 years. During that 8 years we've lost about 20% of the drives in various ways.
It is almost guaranteed that when a drive fails, another drive will have a URE during the resilver process. This is a non-issue as we run RAID-Z3 with multiple online hotspares.
We could do weekly. The volume of data is large enough that even sequential scrubbing when idle is about an 18 hour operation. As it is, we're happy with monthly scrubbing on the Z3 arrays. We don't bother pulling drives until they run out of reallocatable sectors, this extends the service lifetime by a year in most cases.
I intentionally provisioned one of the long term archive only appliances with 12 hot spares. This was to prevent the need for a site visit again before we lifecycle the appliance. Currently down to seven hot spares.
That replacement will probably happen later this year. Should reduce the colo cost by power requirement reduction enough that the replacement 200TB appliance pays for itself in 18 months.
- "RAID is not a backup" primarily because "you could rm -rf". ZFS snapshots cover that failure mode to the same extent that synchronization with offsite does, but cheaper. ZFS snapshots obviously don't cover other failure modes like natural disasters or a break in, so RAID is still not a backup.
- for ZIL to do its work properly, you need the disks not to lie when they claim that the data has been truly saved. This can be tricky to check, so perhaps think about a UPS
- if you have two M.2 slots you could use them to mirror two partitions from two different disks for your data pool's SLOG. The same could be done to form a new mirrored ZFS pool for the OS. In my case I even prefer the performance that a single-copy SLOG gives me at the risk of losing the most recent data before it's moved from the SLOG to the pool.
> - "RAID is not a backup" primarily because "you could rm -rf".
or your house could burn down
or somebody could steal the computer while you're away on vacation
or lightning could strike your electrical grid service entrance or a nearby pole/transformer, causing catastrophic damage
or your house could flood
lots of other things.. if you really have important data it's important to plan to for the total destruction of the storage media and server holding it.
> - "RAID is not a backup" primarily because "you could rm -rf". ZFS snapshots cover that failure mode to the same extent that synchronization with offsite does
Not really. You need to be synchronizing to a _write-only_ backup archive. A local ZFS snapshot can be deleted locally.
The slog is only used for synchronous writes, which most writes are not (as I understand it). Most workloads (ie non-db server) won't see much improvement with one.
> Performance topped out at 111 MiB/s (931 Mbps), which is suspiciously close to 1 Gbps.
That's because of overhead in TCP over IPv4. You're testing the payload throughput, not the physical throughput. The theoretical maximum performance without jumbo frames is around 95%.
I have a couple of home built TrueNAS systems for many years (since FreeNAS, ~ 10 years ago), here is some feedback:
- with same disk size, but just 3 disks, I get around 240 MB/sec read speed for large files (with 10 Gbps NIC). I guess the biggest difference is the CPU power, your NAS seems very slow. On 1 Gbps NIC I get 120 MB/sec transfer speed. My system is even virtualized, on bare metal may be a little bit faster.
- you cannot expand your pool, if you add one more disk there is no way to cleanly migrate to a 5 disk raidz1. There is some new development that kind of does something, but it is not what is needed
- unless esthetics is a big deal for you, there are still $30 cases around. The extra $70 can be used for something else *
- * with a small percentage cost increase, an investment in CPU and RAM can give you the capability to run some VMs on that hardware, so that CPU will not sit at idle 99.9% of the time and be underpowered when you do use it. Using a dedicated computer just for a NAS is not very cost and power efficient, but if you group multiple functionalities it becomes a great tool. For example I run 3-4 VMs at all times, up to ~ 12 when I need it.
- that motherboard and the comparison to a B450 is wrong. The MB restricts you to 4 SATA, while the B450 I bought for ~ $120 has 6 SATA ports
- TrueNAS does not *require* a HBA firmware change, that is needed if you want to convert a RAID controller to plain HBA mode or with certain old HBA that need newer firmware. However for your setup a HBA is not needed. If you want to add many disks and have a good performance (like more than 500-1000 MB/sec) then you need the HBA
- your math is wrong. You calculate available space using ~ 3.8TB disks and divide to 4 TB. The 4TB disks don't have 4TB, but 4x10^12 bytes, so the percentages in your table are exactly 80%, 60% and 40%.
- that CPU does not work with 32GB DIMMs. This works only with newer Ryzen generations, not with Zen+ in this CPU.
- GPU is not missing. TrueNAS does not render anything on a GPU, there is no need for one. I did ran TrueNAS for a couple of years on a computer with no video capability at all (a Ryzen 2700) without any problem, I just used a GPU for the initial installation and then removed it.
- unless you store a database for a SQL server or similar, there is no benefit in a SLOG; it is not a tiered cache, so it does not speed up file transfers in any way. You can have a disk dedicated as a read cache, but the cache content is currently wiped at every restart (a documented limitation) and not needed if you don't want very good performance with small files over the network
Oh, cool! I was referring to SHR. I thought it was a proprietary format and didn't realize you could access it from non-Synology systems. I've updated the post:
Back around 2004 there was a technology called VRRP then OpenBSD got a similar thing called CARP - a quick google suggests these still exist but i never see mention of them in my filter bubble for some reason.
I was obsessed with making an instant failover cluster. I never managed to get it working exactly how i wanted and it relied on two old UPSs with dead batteries to operate as STONITH devices (they had well supported rs232 interfaces).
I sometimes think about investigating that idea again but maybe with raspberry pis and cheap iot plugs.
I set up my most recent NAS using a TerraMaster unit. It's basically a nifty case (9x5x5 inches) around a low power Intel board with a USB stick for a boot device (which I replaced with a mini 100gb USB SSD).
I don't know and don't care about TerraMaster's software (it might be awesome - I have no idea). I just rolled my own NixOS install with ZFS so that I could have a deterministic installation (I've heard good things about the TrueNAS OS as well, but I'm a control freak and like being able to rebuild the entire server with a single command and a config file, so I stick with NixOS).
The nice thing is that I essentially got a motherboard, CPU, PSU, and compact case for $350 (for the F2-422). All I had to do was upgrade the RAM (SO-DIMM) and add the drives.
I've long since reduced to only two drives for my NAS. At one point I was up to 7 drives before I realized my madness. It's cheap enough to get the storage I need with two mirrored drives, is quieter and uses less energy (I can keep it in the same room), and when I finally outgrow them in 5 years or so, the old drives will be re-purposed as backup via an external USB enclosure I keep around.
I don't know, but if I could install NixOS without difficulty, it should be possible. I installed Ubuntu server on it at first and that also worked fine. No tweaking necessary at all. You just flash the standard x64 installer on a USB stick, plug it in, and install like you would on any PC (because it basically is a PC - it even has a working HDMI port).
Yes, but it’s usually not recommended, because the receiver side doesn’t verify that the data is identical to that at the sender side, and a small error could corrupt the file system.
ZFS send and receive is a good way to do it, but there is no ZFS send and btrfs receive!
I have to say, I'm put off by the power benchmark. I have a way older system with way more stuff (6 3.5 inch HDD, 2 2.5 inch HDD, 2.5 inch SSD, 8-port SAS controller, Intel NIC card) and it idles (all drives spun down) at ~30 watts.
When I first bought the system over 10 years ago, ZFS on Linux wasn't really a thing, so I used FreeBSD. I later switched and with the switch came substantial power savings.
Oh, that's interesting. TrueNAS is available on Debian now, so I wonder if there would be a big drop in power consumption.
Lawrence Systems just ran benchmarks[0] between TrueNAS Core (FreeBSD) and TrueNAS Scale (Debian), but they didn't include power consumption, unfortunately.
315 comments
[ 2.3 ms ] story [ 293 ms ] threadI do wonder what the power consumption figures would be though. His system was drawing an annoyingly large amount of power and I suspect that was mostly those HDDs.
Regarding the SLOG device, you probably don't need it for a file server, but if you do you can definitely free a drive bay for an HDD by just using double sided tape somewhere like on the PSU. I'm sure it's also possible to put three more HDDs above the CPU, right in front of the exhaust fan. If I had a 3D printer I would try to build something bringing the total to nine HDDs.
If you need more SATA ports but are running out of PCIe slots, you may be able to reuse an empty M.2 slot. An M.2 with two lanes of PCIe 3 gives you 5 SATA ports with an adapter[0].
[0] https://www.aliexpress.com/item/1005004223466933.html
Just make sure to get one that runs in IT mode or you have to mess with the firmware.
In case some people wonder what "IT mode" is, as I used to some years ago, what you basically want is a card that will expose the drives directly to the OS, as opposed to "volumes".
In other terms, if the card is a RAID controller, it may insist on you creating arrays and only expose those. You can circumvent it by creating single-drive arrays, but it's a pain.
Some cards can do both, but it's usually not advertised. Non-RAID cards also tend to be cheaper. Others (usually LSI) can be flashed with a non-RAID firmware, but again, it's less of a hassle to not have to do it.
The fancier more expensive ones are typically referred to as HBAs instead of "SATA cards" https://unraid-guides.com/2020/12/07/dont-ever-use-cheap-pci...
If you're doing this at home, you can get used enterprise gear on eBay (like an LSI SAS HBA) for the same price or cheaper than brand-new consumer gear, and it will probably still be more reliable (I built a 130 TB NAS for my friend's video production business and literally everything aside from the drives and the cables was bought used on online auction, and it's been humming along fine for a while now - the only part that was bad was one stick of RAM, but the ECC errors told me that before I even got around to running my tests on the sticks)
The ease at which you can revert mistakes using ZFS snapshots is much better compared to restic. You can pretty much navigate to the correct snapshot on your live filesystem and restore whatever you need to restore.
It also makes backups easier as you can just send the snapshots to the backup device (another server or external storage device).
Yea, i love up 1 directory before I delete anything.
It took me a minute, but I assume this should be "move up". This seems like a good habit.
Having NFSv4 ACL access is a huge plus since you can configure permissions natively from windows and have them enforced even on the shell.
Storing them in metadata is not the same as having them natively.
The ACLs do work via NFS and it also works with Active Directory. They ship an AD implementation too, if you are interested in that (it is actually Samba in AD mode).
> I chose raidz1. With only a handful of disks, the odds of two drives failing simultaneously is fairly low.
Which is not really the case of you bought x amount of the same disks and always use them together. I had that happen to me just a few months ago. 4 identical discs bought at the same time. Raidz1 reported one dead/dying disk so I replaced it and started resilvering, which can take days and leaves the disks at 100% utilization.
So after 12 hours or so a second one failed and the data was gone.
Lesson learned: mix up your disks
I bought 4x Seagate Ironwolf Pro 12TB drives from different vendors, one failed after a year, then when I got the replacement another drive failed during the rebuild, and then 6 months later the replacement failed. Now another one of the original drives is also reporting reallocated sectors.
Same system has 4x WD Red drives which have been running fine with 0 reallocated sectors for almost 7 years.
I'm in the same boat. I configured remote backup systems on a handful of computers. I think I reached for backups only twice over the last ten years. Of course I need something, backups or snapshots, but for my use case snapshots (with a network copy) would need work to set up. And if the remote storage is worse, that would be more of a problem than the changes in the restore process.
https://forums.serverbuilds.net/t/official-recommended-sas2-...
There you go.
Having a large NAS has had an interesting (though predictable) impact on all the computers around it: Pretty much every bit of data lives on the NAS (accessed either by CIFS, NFS, or iSCSI). When I had to reinstall Windows, it was mostly painless because all my important data and Steam games library was on a remote iSCSI disk. When I replaced the drives on my linux servers, I didn't have to backup hardly anything, as I worked almost exclusivly on NFS-mounted directories. When bringing up a new raspberry pi for projects, it also has instant access to more terabytes of storage than it could ever need.
Also, for a homelab, getting 10GBe fiber between two machines is surpringly cheap and easy. For certain workloads, it can be a noticable speed boost over 1GBe.
I ask because the difference between a SSD and a hard drive can be massive in this regards, so I'd be really interested to know if the network latency is also a comparable hit.
My gaming computer had an old SATA SSD (Samsung 840 Evo IIRC). Some games took ages to load (particularly Fallout 4). I switched to a much faster NVME drive, and subjectively, it's not any faster loading games. I'd say this was a very underwhelming purchase.
[1]: https://openzfs.github.io/openzfs-docs/man/8/zpool-add.8.htm...
[2]: https://docs.oracle.com/cd/E53394_01/html/E54801/gayrd.html
True RAIDz expansion is something that's supposed to be coming, possibly in Q3 2022, so it may be that by the time one needs to expand a volume, that ability will have landed. That'll be a game changer.
* https://www.techradar.com/news/larger-than-30tb-hard-drives-...
Buy multiple drives and a docking station and you can rotate them:
* https://www.startech.com/en-us/hdd/docking
ZFS send/recv allows for easy snapshotting and replication, even to the cloud:
* https://www.rsync.net/products/zfs.html
* https://arstechnica.com/information-technology/2015/12/rsync...
e.g. a 20TB drive from Seagate is $500. A 4TB drive is $70, 8TB is $140. Getting the same spend in smaller capacity drives would give you 28TB in the 4TB drives and 24TB/32TB in the 8TB drives (for $80 under/$60 over).
Add in a second to rotate and you're spending $1000 in drives, assuming these 26TB drives replace the 20TB drives at a similar price when they trickle down to consumer hands.
* https://github.com/openzfs/zfs/pull/10349
* https://github.com/openzfs/zfs/releases/tag/zfs-2.0.0
We found ZFS led to maintenance issues, but it was probably unrelated to the filesystem per say. i.e. culling a rack storage node is easier than fiddling with degraded raids.
For critical data though I use Borg and a Hetzner StorageBox.
I don't back up my Blu-Rays or DVDs, so I'm backing up <1 TB of data. The current backups are the original discs themselves, which I keep, but at this point, it would be hundreds of hours of work to re-rip them and thousands of hours of processing time to re-encode them, so I've been considering ways to back them up affordably. It's 11 TiB of data, so it's not easy to find a good host for it.
[0] https://github.com/mtlynch/mtlynch-backup
* https://www.techradar.com/news/larger-than-30tb-hard-drives-...
Crazy.
I've run ZFS for home storage and work backups for ~15 years, across Nexenta, ZFS-fuse, FreeBSD, and OpenZFS, backing up hundreds of machines, and have never lost data on one of them.
It was also very complex to manage compare to ZFS, with many different layers to consider.
I'm sure it shines in a data center, for which it has been designed. But unless something radical has changed in the last year, it's not for a budget homelab NAS.
Seen too many people's TrueNAS/FreeNAS installs glitch up over the years to trust the zfs community edition as a sane production choice. ZFS certainly has improved, but Oracle is not generally known for their goodwill toward the opensource community. ;-)
Overall been very little fuzz for my home NAS system.
BTRFS seems to be maturing nicely, hopefully we can start using it for these types of workloads in the next few years.
Energy use is very low.
Yeah, I've never thought much about power consumption, but I've done a few write-ups of previous builds, and I received a lot of questions about power draw, so I decided to measure it on this one. I was surprised at how much power the system consumed, and it will be something I think about more up-front on future builds.
If you are not after speed, then you can do a redundant array of cheap nodes. Instead of using raid, just shove in an 8-12tb disk in a number of thin clients.
The key is that they spend most of the time turned off.
I use raid 6 and also backup my data externally to another nas as well as backup to a static usb drive. Backup requires multiple different types since failures are so catastrophic and can occur in ways you don’t expect.
ZFS can perform periodic scrubs to detect and repair bit rot, and I'm pretty sure TrueNAS is configured to do this by default
Only if you buy different hard drives or at least from different production batches. I had a lot of trouble on the same premise and I won't make that mistake again.
Edit: He mentioned it though ( a bit later in the article)
> The problem is that disks aren’t statistically independent. If one disk fails, its neighbor has a substantially higher risk of dying. This is especially true if the disks are the same model, from the same manufacturing batch, and processed the same workloads. Given this, I did what I could to reduce the risk of concurrent disk failures.
> I chose two different models of disk from two different manufacturers. To reduce the chances of getting disks from the same manufacturing batch, I bought them from different vendors. I can’t say how much this matters, but it didn’t increase costs significantly, so why not?
Please elaborate, I'd love to hear your story!
I hear a lot of advice around raid/z-levels and it often seems backed up by shaky math that doesn't seem to be backed up by reality (like the blog posts that claim that a rebuild of an array of 8 TB drives will absolutely have hard read errors, no exceptions, and yet monthly ZFS scrubs pass with flying colors?)
Not much to tell otherwise though. I was really annoyed with it at the time.
I really like seeing other people's builds, but I know that building my own computer isn't something I want to do. I was happy to see the comparison between the DIY model and the roughly-equivalent commercial units. I'll likely buy another QNAP (to run TrueNAS on) when the time comes, and the comparison tells me that I won't get screwed too badly by doing so.
>I've gone the TrueNAS route, but I'm running it on a QNAP TS-451. I'm running TrueNAS off of a USB stick hanging off the back
Oh, I didn't realize that QNAP allows that. Synology makes it pretty hard to boot any other OS, and I assumed the other vendors were similar. I'll keep that in mind for my next build because I do have fun building servers, but I also really appreciate systems like Synology and QNAP where the hardware and case is optimized for the NAS use-case.
I run two offline Nas (I unpower them and do scrabing every month) and have one with raidz2 for all critical things like my photos.
To resilver 8tb takes ages and while he wrote his thoughts on it, I was missing the repair risk calculation
>Before building this system, I had zero experience with ZFS, so I was excited to try it out.
Sorry, but this is amusing to me. ZFS on TrueNAS is probably fine, but you're building your production NAS, to replace the Synology device you've become "so dependent on". Don't become dependent on ZFS without knowing the implications!
I was facing this choice recently, and I agreed with the other tech savy person in the household that we should just use good old LVM + Btrfs. Not only does it run like a charm, but it also allowed us to switch the LV from single (during the data move) to RAID 1 and eventually to RAID 5/6 with zero issues. It will also be much easier to recover from than ZFS.
On another note, it's a bad market to buy NAS drives, especially from Seagate. Seagate Exos drives are at this point in time often cheaper than IronWolf, even non Pro IronWolf. They're slightly more noisy and don't come with the free data recovery, but otherwise they're a straight upgrade over the IronWolf drives.
When it fills up, I delete some files rather than adding disks.
https://man7.org/linux/man-pages/man7/lvmraid.7.html#DATA_IN...
On the other hand, I can’t stand people who say “homelab”. Ugh.
And to my account, I think my upsides are that:
- ability to choose the kernel
- no need for SSD for base OS since running off of RAM is rather easy on Linux
- samba can run in a container thus a bit more control security-wise
- server may run something else as well
Of course, this comes with a lot more technical hurdles. More like a side-project than utility really. That's why I was wondering does TrueNAS provide non-obvious upsides that would be lacking in self-rolled one.
The upsides are that it's plug-and-play for anyone who doesn't want to research all the options available and figure out the various pitfalls on their own.
> no need for SSD for base OS since running off of RAM is rather easy on Linux
I don't understand this sentence. You're running off a RAM disk with no boot drive? What if you have a power outage?
> samba can run in a container thus a bit more control security-wise
Core supports FreeBSD jails and Scale supports Docker so you could run samba in a container on either if you're willing to do set it up yourself.
> server may run something else as well
As before, both have jail/container functionality. I haven't used Scale myself but Core comes with a bunch of "click to install" jail options for stuff like Plex, ZoneMinder, etc. Our machine also runs a Windows VM (ew) and a Wordpress install in a Jail
> You're running off a RAM disk with no boot drive? What if you have a power outage?
Yes, the server only has the HDDs which contain the NAS data. The server bootloops until it gets an image from the router (ipxe boot). The disk images have systemd scripts which install everything from 0 on each boot. Coincidentally, this means system restart is how I upgrade my software.
> Core supports FreeBSD jails and Scale supports Docker
This clarifies the situation -- TrueNAS seems like an option that I would recommend for anyone who wants a quick OSS NAS setup.
I consider myself an intermediate homelabber and a TrueNAS beginner. I just built my first NAS server, so wrote this to capture everything I wish I'd known at the start. I hope it's helpful for anyone else thinking about building their first NAS server.
Any questions or feedback about the post are more than welcome.
Some suggestions for anyone else looking to do the same:
i3 runs a bit cooler than ryzen, still 8 threads. 8tb WD blues (they're SMR at 8 and up). You can find Atx boards with 8 sata ports and dual nvme slots for caching / fast pools.
SMRs are a fucking blight.
RAID-Z1 is something I never consider without a solid backup to restore from and a plan to execute that process at least once in the lifecycle of an array.
If you suffer a total disk failure of one of those disks in the array, you have likely lost some data. The good news is that ZFS will tell you exactly which files you have lost data for and cannot rebuild. If you have those files, you can overwrite them with the backups to get your integrity back.
The reason is, with a total loss of a single disk, any read error on any of the remaining disks is a lost/corrupted file.
For this reason, you need a strong(easily accessible, consistent, current) backup strategy and an acceptance of downtime with Z1.
As for ECC, it's better, but your absolute worse case scenario is that you get a bit flip before the sync and hash happens, and now that bit flipped data is committed to disk and you think it's OK. I prefer ECC to avoid this, but you are still reaping a multitude of benefits from ZFS without ECC.
The only valid rule for RAM and ZFS is that more RAM = more caching of recently read data. Single, or very few user appliances will see little benefit past 8GB even with 100TB unless you happen to be reading the same data over and over. Where ZFS shines is having hundreds of gigabytes of RAM and tens or more concurrent users mostly accessing the same data. That way the vast majority of reads are from RAM and the overall disk IOPS remain mostly idle.
Most of the ZFS RAM myths come from Deduplication, which should be disregarded as a ZFS feature until they allow storing of the DDT on a Optane-like latency device. Even better would be offline deduplication, but I doubt that will be a thing in ZFS this decade.
Wait, what? If a RAID-(z)1 ZFS array loses one disk, there's data loss? I've ran so many RAID-1 and RAID-10 arrays with mdadm that I can't even being to count them, and I had many drive failures. If any of those arrays would have corrupted data, I would have been mad as hell.
What I am missing here? How is this even remotely acceptable?
That is the meat of it. With traditional RAID it is the same issue, except you never know it happens because as long as the controller reads something, it's happy to replicate that corruption to the other disks. At least with ZFS, you know exactly what was corrupted and can fix it, with traditional RAID you won't know it happened at all until you one day notice a corrupted file when you go to use it.
RAID-Z1 is better than traditional RAID-5 in pretty much every conceivable dimension, it just doesn't hide problems from you.
I have encountered this literal scenario where someone ran ZFS on top of a RAID-6(don't do this, use Z2 instead). Two failed drives, RAID-6 rebuilt and said everything was 100% good to go. A ZFS scrub revealed a few hundred corrupted files across 50TB of data. Overwrote the corrupted files from backups, re-scrubbed, file system was now clean.
ZFS automatically self-heals an inconsistent array (for example if one mirrored drive does not agree with the other, or if a parity drive disagrees with the data stripe.)
ZFS does not suffer data loss if you "suffer a total disk failure."
I have no idea where you're getting any of this from.
The poster built a (non redundant) zfs pool on top of a hardware raid6 device. The underlying hardware device had some failed drives, and when rebuilt, some of the underlying data was lost.
ZFS helped by detecting it instead of letting the bad data though like would normally have happened.
See eg here where the increasing disk size vs specified unrecoverable read error rate is explored in relation to the question at hand: https://queue.acm.org/detail.cfm?id=1670144 (in the article Adam Leventhal from Sun, the makers of ZFS, talks about the need for triple parity).
Also, the conclusion "ensure your backups are really working" is an important point irrespective of this question, since you'll also risk losing data due to buggy software, human errors, ransomware, etc.
In RAID-Z, you can lose one drive or have one drive with 'bit rot' (corruption of either the parity or data) and ZFS will still be able to return valid data (and in the case of bit rot, self-heal. ZFS "plays out" both scenarios, checking against the separate file checksum. If trusting one drive over another yields a valid checksum, it overwrites the untrusted drive's data.)
Regular RAID controllers cannot resolve a situation where on-disk data doesn't match parity because there's no way to tell which is correct: the data or parity.
thanks for the clarification.
in that sense, yes, of course, if you have bit rot and another disk failing, things go south with just two disk. ZFS is not magic.
How does Z1 recover the data in this case other than alerting you of which files it cannot repair so that you can overwrite them?
The drives have been champs overall, they're approaching an average runtime of about 8 years. During that 8 years we've lost about 20% of the drives in various ways.
It is almost guaranteed that when a drive fails, another drive will have a URE during the resilver process. This is a non-issue as we run RAID-Z3 with multiple online hotspares.
Are they used 24/7 at high iops? Why not nightly scrub?
I intentionally provisioned one of the long term archive only appliances with 12 hot spares. This was to prevent the need for a site visit again before we lifecycle the appliance. Currently down to seven hot spares.
That replacement will probably happen later this year. Should reduce the colo cost by power requirement reduction enough that the replacement 200TB appliance pays for itself in 18 months.
- for ZIL to do its work properly, you need the disks not to lie when they claim that the data has been truly saved. This can be tricky to check, so perhaps think about a UPS
- if you have two M.2 slots you could use them to mirror two partitions from two different disks for your data pool's SLOG. The same could be done to form a new mirrored ZFS pool for the OS. In my case I even prefer the performance that a single-copy SLOG gives me at the risk of losing the most recent data before it's moved from the SLOG to the pool.
or your house could burn down
or somebody could steal the computer while you're away on vacation
or lightning could strike your electrical grid service entrance or a nearby pole/transformer, causing catastrophic damage
or your house could flood
lots of other things.. if you really have important data it's important to plan to for the total destruction of the storage media and server holding it.
Not really. You need to be synchronizing to a _write-only_ backup archive. A local ZFS snapshot can be deleted locally.
(Also fire, compromise, police confiscation, etc.)
For my write-only backup needs, this free service known as S4 works wonderfully:
http://www.supersimplestorageservice.com/
(This is a joke.)
That's because of overhead in TCP over IPv4. You're testing the payload throughput, not the physical throughput. The theoretical maximum performance without jumbo frames is around 95%.
https://en.wikipedia.org/wiki/Jumbo_frame#Bandwidth_efficien...
- with same disk size, but just 3 disks, I get around 240 MB/sec read speed for large files (with 10 Gbps NIC). I guess the biggest difference is the CPU power, your NAS seems very slow. On 1 Gbps NIC I get 120 MB/sec transfer speed. My system is even virtualized, on bare metal may be a little bit faster.
- you cannot expand your pool, if you add one more disk there is no way to cleanly migrate to a 5 disk raidz1. There is some new development that kind of does something, but it is not what is needed
- unless esthetics is a big deal for you, there are still $30 cases around. The extra $70 can be used for something else *
- * with a small percentage cost increase, an investment in CPU and RAM can give you the capability to run some VMs on that hardware, so that CPU will not sit at idle 99.9% of the time and be underpowered when you do use it. Using a dedicated computer just for a NAS is not very cost and power efficient, but if you group multiple functionalities it becomes a great tool. For example I run 3-4 VMs at all times, up to ~ 12 when I need it.
- that motherboard and the comparison to a B450 is wrong. The MB restricts you to 4 SATA, while the B450 I bought for ~ $120 has 6 SATA ports
- TrueNAS does not *require* a HBA firmware change, that is needed if you want to convert a RAID controller to plain HBA mode or with certain old HBA that need newer firmware. However for your setup a HBA is not needed. If you want to add many disks and have a good performance (like more than 500-1000 MB/sec) then you need the HBA
- your math is wrong. You calculate available space using ~ 3.8TB disks and divide to 4 TB. The 4TB disks don't have 4TB, but 4x10^12 bytes, so the percentages in your table are exactly 80%, 60% and 40%.
- that CPU does not work with 32GB DIMMs. This works only with newer Ryzen generations, not with Zen+ in this CPU.
- GPU is not missing. TrueNAS does not render anything on a GPU, there is no need for one. I did ran TrueNAS for a couple of years on a computer with no video capability at all (a Ryzen 2700) without any problem, I just used a GPU for the initial installation and then removed it.
- unless you store a database for a SQL server or similar, there is no benefit in a SLOG; it is not a tiered cache, so it does not speed up file transfers in any way. You can have a disk dedicated as a read cache, but the cache content is currently wiped at every restart (a documented limitation) and not needed if you don't want very good performance with small files over the network
I wonder what he means by this. If he's referring to SHR, then it's just standard mdraid and Synology themselves have instructions on how to mount the volume in Ubuntu https://kb.synology.com/en-us/DSM/tutorial/How_can_I_recover...
edit: He later mentions encrypted volumes but those are also just using standard encryptfs https://www.impedancemismatch.io/posts/decrypt-synology-back...
This is one of the reasons I feel comfortable recommending Synology devices - there's not a lot of lock-in
https://github.com/mtlynch/mtlynch.io/pull/920
I was obsessed with making an instant failover cluster. I never managed to get it working exactly how i wanted and it relied on two old UPSs with dead batteries to operate as STONITH devices (they had well supported rs232 interfaces).
I sometimes think about investigating that idea again but maybe with raspberry pis and cheap iot plugs.
I don't know and don't care about TerraMaster's software (it might be awesome - I have no idea). I just rolled my own NixOS install with ZFS so that I could have a deterministic installation (I've heard good things about the TrueNAS OS as well, but I'm a control freak and like being able to rebuild the entire server with a single command and a config file, so I stick with NixOS).
The nice thing is that I essentially got a motherboard, CPU, PSU, and compact case for $350 (for the F2-422). All I had to do was upgrade the RAM (SO-DIMM) and add the drives.
I've long since reduced to only two drives for my NAS. At one point I was up to 7 drives before I realized my madness. It's cheap enough to get the storage I need with two mirrored drives, is quieter and uses less energy (I can keep it in the same room), and when I finally outgrow them in 5 years or so, the old drives will be re-purposed as backup via an external USB enclosure I keep around.
It’s unclear if one could install TrueNAS on a Synology, QNAP, Teramaster etc. Sometimes hardware is not supported.
Edit: Looks like someone did a writeup for TrueNAS on a TerraMaster: https://joelduncan.io/freenas-on-terramaster-f2-221/
Also: https://mightygadget.co.uk/how-to-upgrade-the-terramaster-f4...
You get an affordable TrueNAS server.
I mean backing up the file system (as with ZFS send) not scanning all files (using rsync of restic).
ZFS send and receive is a good way to do it, but there is no ZFS send and btrfs receive!
When I first bought the system over 10 years ago, ZFS on Linux wasn't really a thing, so I used FreeBSD. I later switched and with the switch came substantial power savings.
Lawrence Systems just ran benchmarks[0] between TrueNAS Core (FreeBSD) and TrueNAS Scale (Debian), but they didn't include power consumption, unfortunately.
[0] https://www.youtube.com/watch?v=BoiHHnBDg0E