I still pay for snooty, and the reason for that is that when a disk goes bad (not if; when) I pop its tray out, replace the disk, pop the tray with the disk back in, click a couple of widgets, and that’s it. I know it will be rebuilt properly.
(And I know I have to do that, because when the disk fails it beeps and lights a led near the bad disk)
It’s easy to build a NAS such as the one described in this article, but in the long run, data loss is significantly more likely.
Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.
This is what I'm hoping to do. I haven't done much research yet, so I look forward to reading about your learnings. (Also to the person who responded to you with their experience and link.)
Thanks for saving me probably some hours...
EDIT: I'm also coming from an old Synology NAS from the ~2015 era.
Not a great time price-wise to be building a NAS, but I have been doing so for the last two weeks. Inside a Jonsbo N6 case, which is pretty nice with an 8x SATA backplane and drive bays (unlike the earlier Jonsbo variants).
I ended up on shucking 4x the 14 TB WD Elements Desktop. They contain helium drives, the WD140EDGZ in my case, and are about a third cheaper than 4x the 12 TB WD Red Plus drives (which are air-filled). The shucking was easier than I expected too, and the performance seems very comparable. The warranty is a definite downside (European, so no Magnuson-Moss), but I think I can even get them back in their enclosure should they fail during the 2-year warranty period.
I've put some second hand 256 GB M.2 SSDs in there as boot drives. It was a bit of a struggle to get it to work in a way that failure of one of the drives doesn't hold up booting, combined with LUKS, TPM keys and ZFS on root. Learned a lot about systemd-boot which I have never used before, but feels a lot saner to me than grub ever was. So now I have a large script which debootstraps a Debian based NAS into being.
I noticed that there are a lot of ZFS myths and cargo culting. For example TFA mentions ECC RAM, which in some circles is a must-have because ZFS would wreck your pool during a scrub otherwise, which is a myth. It's also very expensive, especially this year. You also don't need much RAM for ZFS, L2ARC doesn't use much RAM at all, to name a few others.
Still doubting about setting `dnodesize=auto` (which is the default), because there are some horror stories about that [1]. And it seems impossible to find a cloud storage provider with reasonable prices that supports `zfs send`. Rsync.net upped their minimum order to 10 TiB recently, which is far too much for my use case.
I've been looking at setting something up with mkosi/systemd for my home server/NAS - it's interesting seeing how different setups are solving the problem for inspiration.
Yeah no. When it comes to backups and data storage, I would rather use a proven reliable system that’s been used and tested by millions of other users, keep these hacky stuff for your hyperland set up.
this is really cool. I've been dealing with an aging Synology nas and this is something I can pick up, evaluate performance and how safe it is to serve as home for my data.
> I am creating a RAIDZ1 (RAID 5) zpool. That means 1 drive redundancy in-case of failure
A friend once told me that RAID5 has a high latency cost, because every Write requires a Read to update the stripes across all drives, and while this made sense when drives were expensive, nowadays you might as well do a RAID10 instead, and trade space for latency.
To this day I still use a ZFS array for my critical backup storage. I've migrated over to using snap raid + mergerfs for my larger Linux iso storage array. Simple enough and I can pull a drive on it's own without any other stuff.
Just yesterday I was brainstorming with ChatGPT about this. I have an ancient QNAP plus a slightly less ancient NUC running PiHole, Wireguard and other services. Both need replaced, so why not combine them?
I don't know much about ZFS, but it sounds like I need to learn. Docker may have conquered the world, but I plan to stay with LXD for services.
The one thing I take issue with: an appliance like this runs 24/7. It should be low power and fanless. A processor like the N100 seems like the obvious choice.
I've been thinking about setting up something like this for a while. I have a Broadwell dual socket Xeon workstation that I'm going to upgrade to Proxmox. Would it be reasonable to run something like this as an LXC or VM or would you put it in the base kernel?
> I have a Broadwell dual socket Xeon workstation that I'm going to upgrade to Proxmox. Would it be reasonable to run something like this as an LXC or VM or would you put it in the base kernel?
I've got a small HP Z440 workstation (small in that it's 10 years old and only single-CPU) and a 14 cores / 28 threads Xeon e5-2680 v4 in it. It's a little workhorse.
I run Proxmox (basically "Debian + ZFS out of the box + easy to spawn VMs / LXCs") on it.
And you can totally put something like FreeNAS in a VM. If you plan to really serve lots of files and need lots of perfs, you should get a tiny bit fancy and passthrough to the FreeNAS VM an entire disk controller (device passthrough to a VM in this day and age of LLMs that are there to find the relevant parts of docs for you in a split second is easy). IIRC the FreeNAS documentation itself explains how to run FreeNAS in a VM so I don't think it's "unsupported".
There's nothing wrong with running it bare either.
One alternative for those who don't want any of the major NAS vendors, just use RHEL10. It's free up to 16 licenses, it's ultra stable, cockpit is a very mature gui for a lot of maintenance tasks.
It's a set and forget OS that will run for years without requiring your attention. But these days it has decent container support for hosting services on.
I found cockpit to be incredibly useful for managing NAS and VPS. Even though any configuration is not an issue with LLMs, having a dashboard with toggles, formatted logs and such is helpful, and I can even run shell commands on phone without ssh. In the NAS case, you can easily see the results of backup/maintenance cronjobs, manage samba (which is a huge pain in the ass to get to work properly on iPhone), and monitor disk usage
I killed 2x 4tb Nvme drives on Pciexpress with ZFS raid1 under proxmox, got them replaced, tried again on both pci-express and usb adapter kept getting io problems there too with zfs, drive gets disconnected during scrub or high load, tried already kernel or power management options for pciexpress and usb without sucess, managed to reproduce with high load.
or proxmox ( especially community edition) is fucked or zfs is still unstable, surprisingly with different drives on different interfaces I obtained same results, didnt swap proxmox for clean os, that might being some changes since modules and options would be different i guess. ( drives firmware etc are fine and performing well when inspected with proprietary windows tools)
just search zfs nvme pcie or usb problema or disconnect and you see similar stability problem for different users in different cases (os, drives etc), but also unraid and others, maybe somewhere is rithe right combo of options/glitches but didnt find it yet
I really, do not want to have a big server rack lying around at my house. I have a couple of SSDs that are not being used and 3 USB HDD drives. I also have a PN50 that I am not using as well.
I am thinking of buying a USB bay with 5 SSD slots in it and then 3 HDD drive.
My use case is very, very cold. It is mostly just readonly data with some rclone sync every week.
Does anyone have a suggestion? I am pretty much relying on ZFS to do all the redundancy for me.
I'm sure people have reasons for taking these things as far as they do with ZFS, and everything else that goes into what is commonly considered a "NAS." But I've found great convenience in a tight NFSv3 config running from a single high capacity HDD with ext4 on a Linux machine (primary system that's always on anyway while I'm around/awake) making things available to my other devices for the electricity and overhead cost of one machine instead of multiple.
I try to keep my network configuration restrictive by default, so I'm not too concerned about possible security arguments running it from my main machine. I've probably committed some great sin here, but is plain NFSv3 and secondarily Samba (for compatibility) really not enough?
Once you add multiple machines reading and writing simultaneously, you'll hit a limit. But that's fine, build what you need now, no need to over-optimize.
I think it's appropriate to link to my blog post about building such a NAS but with very different technologies: dm-integrity, mdadm and XFS. The reason being that as a C/C++ dev following OpenZFS development closely, I found the focus on ginormous features over general stability and the constant problems due to the SPL/separate page cache too worrying.
Sure, I traded the convenience of kitchen sink (or Swiss army knife for the more charitable souls) ZFS for some initial pain, but I'm very happy with my choice today.
A friendly reminder since it's probably relevant: when's the last time you tested disaster recovery for your own setup? If you haven't verified recovering it, you don't have an offsite backup.
I had a cloud backup (Backblaze B2, using rclone and encryption) for my home NAS. Some unlucky drive failures later, I was getting ready to recover from my Backblaze backup... and then I couldn't find/remember where I saved the rclone encryption backup.
I lost all data in that NAS, including irreplaceable personal/family photos, due to that mistake. My lesson to share: please verify you can indeed recover from backups.
Can you explain a bit more about this. Would this help in a single server to facilitate backups? Or does this need the the NAS and the Proxmox Backup Server on a different system?
I'm running PBS on a separate mini pc, it is the perfect forever near instant incremental backups that sold me. I did perfectly recover in 20 minutes after a full hardware failiure. Best recovery experience ever.
Building a NAS from scratch is really fun! A small hinderance, but definitely part of the fun as well, is the lack of a "complete resource" on the topic covering how to do every single thing you need to do. Part of the point of my blog post is actually to bring to the internet yet another opinionated NAS setup "guide" (eventhough I would hesitate to call it a guide, but if I ever had to do the same thing again I would definitely read my own post first).
56 comments
[ 3.0 ms ] story [ 77.9 ms ] thread(And I know I have to do that, because when the disk fails it beeps and lights a led near the bad disk)
It’s easy to build a NAS such as the one described in this article, but in the long run, data loss is significantly more likely.
Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.
Thanks for saving me probably some hours...
EDIT: I'm also coming from an old Synology NAS from the ~2015 era.
I ended up on shucking 4x the 14 TB WD Elements Desktop. They contain helium drives, the WD140EDGZ in my case, and are about a third cheaper than 4x the 12 TB WD Red Plus drives (which are air-filled). The shucking was easier than I expected too, and the performance seems very comparable. The warranty is a definite downside (European, so no Magnuson-Moss), but I think I can even get them back in their enclosure should they fail during the 2-year warranty period.
I've put some second hand 256 GB M.2 SSDs in there as boot drives. It was a bit of a struggle to get it to work in a way that failure of one of the drives doesn't hold up booting, combined with LUKS, TPM keys and ZFS on root. Learned a lot about systemd-boot which I have never used before, but feels a lot saner to me than grub ever was. So now I have a large script which debootstraps a Debian based NAS into being.
I noticed that there are a lot of ZFS myths and cargo culting. For example TFA mentions ECC RAM, which in some circles is a must-have because ZFS would wreck your pool during a scrub otherwise, which is a myth. It's also very expensive, especially this year. You also don't need much RAM for ZFS, L2ARC doesn't use much RAM at all, to name a few others.
Still doubting about setting `dnodesize=auto` (which is the default), because there are some horror stories about that [1]. And it seems impossible to find a cloud storage provider with reasonable prices that supports `zfs send`. Rsync.net upped their minimum order to 10 TiB recently, which is far too much for my use case.
[1] https://github.com/openzfs/zfs/issues/11353
[2] https://www.rsync.net/products/zfsintro.html
https://www.samsung.com/us/memory-storage/nvme-ssd/990-pro-p...
Is part of this publicly available? After seeing, - https://words.filippo.io/frood/ - https://0pointer.net/blog/fitting-everything-together.html
I've been looking at setting something up with mkosi/systemd for my home server/NAS - it's interesting seeing how different setups are solving the problem for inspiration.
> I am creating a RAIDZ1 (RAID 5) zpool. That means 1 drive redundancy in-case of failure
A friend once told me that RAID5 has a high latency cost, because every Write requires a Read to update the stripes across all drives, and while this made sense when drives were expensive, nowadays you might as well do a RAID10 instead, and trade space for latency.
Is this still true with ZFS RAIDZ1?
I don't know much about ZFS, but it sounds like I need to learn. Docker may have conquered the world, but I plan to stay with LXD for services.
The one thing I take issue with: an appliance like this runs 24/7. It should be low power and fanless. A processor like the N100 seems like the obvious choice.
I've got a small HP Z440 workstation (small in that it's 10 years old and only single-CPU) and a 14 cores / 28 threads Xeon e5-2680 v4 in it. It's a little workhorse.
I run Proxmox (basically "Debian + ZFS out of the box + easy to spawn VMs / LXCs") on it.
And you can totally put something like FreeNAS in a VM. If you plan to really serve lots of files and need lots of perfs, you should get a tiny bit fancy and passthrough to the FreeNAS VM an entire disk controller (device passthrough to a VM in this day and age of LLMs that are there to find the relevant parts of docs for you in a split second is easy). IIRC the FreeNAS documentation itself explains how to run FreeNAS in a VM so I don't think it's "unsupported".
There's nothing wrong with running it bare either.
It's a set and forget OS that will run for years without requiring your attention. But these days it has decent container support for hosting services on.
For some reason people insist on doing truenas on top of proxmox and then introducing a networking layer between everything they do. Noooo…
or proxmox ( especially community edition) is fucked or zfs is still unstable, surprisingly with different drives on different interfaces I obtained same results, didnt swap proxmox for clean os, that might being some changes since modules and options would be different i guess. ( drives firmware etc are fine and performing well when inspected with proprietary windows tools)
just search zfs nvme pcie or usb problema or disconnect and you see similar stability problem for different users in different cases (os, drives etc), but also unraid and others, maybe somewhere is rithe right combo of options/glitches but didnt find it yet
I am thinking of buying a USB bay with 5 SSD slots in it and then 3 HDD drive. My use case is very, very cold. It is mostly just readonly data with some rclone sync every week.
Does anyone have a suggestion? I am pretty much relying on ZFS to do all the redundancy for me.
I try to keep my network configuration restrictive by default, so I'm not too concerned about possible security arguments running it from my main machine. I've probably committed some great sin here, but is plain NFSv3 and secondarily Samba (for compatibility) really not enough?
Sure, I traded the convenience of kitchen sink (or Swiss army knife for the more charitable souls) ZFS for some initial pain, but I'm very happy with my choice today.
https://world-playground-deceit.net/blog/2025/06/nas-setup-l...
I had a cloud backup (Backblaze B2, using rclone and encryption) for my home NAS. Some unlucky drive failures later, I was getting ready to recover from my Backblaze backup... and then I couldn't find/remember where I saved the rclone encryption backup.
I lost all data in that NAS, including irreplaceable personal/family photos, due to that mistake. My lesson to share: please verify you can indeed recover from backups.
Building a NAS from scratch is really fun! A small hinderance, but definitely part of the fun as well, is the lack of a "complete resource" on the topic covering how to do every single thing you need to do. Part of the point of my blog post is actually to bring to the internet yet another opinionated NAS setup "guide" (eventhough I would hesitate to call it a guide, but if I ever had to do the same thing again I would definitely read my own post first).