in my defense, i wasn’t dismissing his work. i was wondering why it merits HN attention. it’s completely ordinary stuff. the world needs ordinary stuff but this isn’t buzznews “the 3 surprising tricks about zfs snapshots”.
Sanoid seems like a great tool. It was at the top of my list when I was looking for this kind of solution.
I ended up going with zfs_autobackup - https://github.com/psy0rz/zfs_autobackup because it doesn't need anything set up on the computer to backup. Just a script on the backup server that does pull backups.
I'm not trying to hijack the thread but I've often appreciated links to similar software in threads like this.
sanoid works great, used it for quite some time and the creator is actively involved in the zfs community
in the spirit of link sharing for other tools, I've been moving many of my personal infrastructure & tools to golang-based apps, and I started using this for zfs replication
The issue I have with that is that if everything is done by the backup server, it has the ability to both create and destroy (prune) datasets (snapshots), which means it has the ability to erase all of the data and all of the backups.
One thing I like about Sanoid/Syncoid is that if you set up Sanoid on both sides and use Syncoid to pull, the backup server only requires the ability to send snapshots (which you can configure with "zfs allow"). The backup server also doesn't require root in this case. This means neither server can delete the other's data.
And you still get all the automatic snapshot creation (done on the primary server) and pruning (done on both servers).
This looks pretty solid. On one of my machines I'm using "zfs-auto-snapshot", which works great for something I can just install and not have to manage, for getting some history. For replication of snapshots, I haven't done that recently, but in the past I just used some scripts I had written to do one-time migrations. That was a decade ago. This looks much more capable than those scripts.
Remote access to a ZFS gives you much different guarantees from your average S3-lookalike, so an integer price difference does not seem unreasonable to me.
"The last addition is the ability to do raw, encrypted sends and receives. The idea here is to send raw encrypted and compressed data and receive it exactly as is on a backup system. This means that the dataset on the receiving system is protected using the same
user key that is in use on the sending side. By doing so, datasets can be efficiently backed up to an untrusted system without fear of data being compromised."
Unfortunately it's taking a long time for the feature to propagate through the ZFS implementations. I don't know if rsync.net's servers support it.
"I've been interested into rsync.net. How does it work if I want my remote ZFS dataset to be encrypted without leaving the key on the remote server?"
If you have multiple terabytes of data, zfs-send is a very efficient (and elegant) way to transfer them to rsync.net.
We can give you your own zpool, with root, running the latest production/stable ZoL code - so you will have encryption and raw send, etc.
However ...
If you have hundreds of GB up to ... one or two TB ... we paste this suggestion to you instead:
I think you should look into the 'borg' backup tool - it has become the de facto standard for remote backups because it does everything that rsync does (efficient, changes only
backups) but also produces strongly encrypted remote backup sets that only you have a key to ... rsync.net has no access to the data.
I have an Ubuntu server with a ZFS pool that I want to rebuild as a FreeNAS/TrueNAS server. Can I use something like rsync.net to push my datasets up as a backup, rebuild my server, and then pull them back down into a reorganised/renamed tree of datasets? Are there compatibility issues between ZFS on Linux and ZFS on FreeNAS that I need to be concerned about? (that come into play when pushing/pulling datasets)
I should also mention that this is actually an Ubuntu VM running in Hyper-V on a Windows Server 2012 host... don’t ask! It needs to be rebuilt one way or another (and it is running on an HP Microserver N54L!)
As for FreeNAS, I feel it gives the closest thing to a Synology NAS in terms of manageability through a GUI, etc, while also supporting ZFS.
Ahh, I don't have the hardware to run zfs in a VM, for home I run a big zfs array on bare metal and also host some vms next to it. If I were running zfs in a VM itself I would be more open to freenas.
Maintaining freenas and trying to run some other virtualized apps next to it was not good for me.
> Can I use something like rsync.net to
> push my datasets up as a backup
Yes, but depending on the size/value of the dataset, rsync might not be the best option.
Even after recent price reduction (Yay!), and with a hn discount (Yay!) - I still consider rsync.net as a personal backup service a little overkill/expensive past 100 GB or so. And likely you could just handle anything on the order of 1 to 2 tb locally anyway (spare disks).
You technically could use it for 10+tb of data... But uploading and downloading is going to be slow unless you have a real uplink (ie at least 1gbps).
And if you did, unless you need filesystem level Metadata - rsync might be a better option than zfs send/receive (even though zfs is cool).
>, rebuild my server, and then pull them back down into a reorganised/renamed tree of datasets?
Again, if you just need the data back - you might be well served with rsync or restic (both choices should be orthogonal to using rsync).
> Are there compatibility issues between ZFS on Linux and ZFS on FreeNAS that I need to be concerned about?
The world is divided in Oracle and not-Oracle - AFAIK there shouldn't be major differences between those two - that I believe now both follow upstream zfsonlinux.
However - I believe rsync.net still doesn't support zfsonlinux/openzfs encryption (zfsonlinux 0.8+). While locally it's possible to encrypt the disk "under" zfs... Native encryption is tempting.
And you would of course never send anyone data without encrypting it?
The switch of FreeBSD's ZFS master being ZoL is still work in progress, and is not the default ZFS in FreeBSD yet. (FreeNAS being FreeBSD + Configuration + Management Software).
A fair number of the ZoL 0.8 features (encryption, allocation classes being two prominent examples) are not available currently for FreeBSD
Your pools may have incompatible features in use that would prevent this. I recently set up an Ubuntu (19.10 desktop) box and cannot recv it's datasets on FreeNAS.
For now I'm just doing rsync of the snapshot directories, not upgrading the Ubuntu pool, and waiting on FreeNAS feature support to match.
[edit, of course moments after posting this I found this link
https://www.rsync.net/products/zfsintro.html in the Arstechnica article. Still can't find it linked from the site though...]
> Now, if only there were a cloud storage provider that you could directly zfs-send to ...
Is there something on rsync.net's site that describes its support for zfs send?
I've found this old thread on HN referring to an Arstechnica article:
Only tangentially related, but seems like a good thread to ask:
Does ZFS (or any other FOSS file system) have a setup like Isilon or NetApp, where you could have any 2 disks in a box of 8 or any box out of say 6 die, and still not lose anything other than bandwidth?
If you want something that deals with multiple tiers of fault tolerance, you should be looking at distributed filesystems (eg. Ceph) rather than single-system filesystems like ZFS and btrfs. Also keep an eye out for erasure coding as a keyword, since that usually implies more control over the degree of redundancy than a handful of RAID levels.
For Isilon, a better comparison would be Ceph. With Ceph you can have a cluster of 10s or 100s of servers, and you can lose 1 disk, or 1 server, or 1 rack or servers and it will recover very quickly.
As long as your minimum requirements for your type of replication or erasure coding profile are satisfied, your data stays safe and available.
sanoid is great.
I asked how to stop sanoid from snapshotting docker created datasets in rpool/var/lib/docker when rpool has the recursive flag, and they added a feature to do it the next day.
For those that actually use perl - I know perl has had (fairly sane?) package management for a while... Is it reasonable to use system perl version in anger?
I would typically reach for a version manager/virtual env with python or ruby - do you, with perl as well?
I love using Sanoid+Syncoid for keeping my laptop (with Xubuntu on ZFS) backed up to my NAS (running FreeNAS). In addition to the periodic snapshots Sanoid manages, I also have a small script (shameless plug [0]) that hooks into APT to take snapshots right before upgrades occur. It's already saved me decent chunk of time that would've been spent rebuilding from a botched upgrade.
I wanted to use this tool for backing up my laptop to my NAS, however it only allows SSH to be used for data transfer, which is a real bottleneck for speed. It basically means you can only transfer as fast as one core can decrypt or encrypt data on either end of the pipe.
I ended up using https://github.com/oetiker/znapzend instead, because it can use mbuffer for the actual data transfer, which is much faster.
Is SSH speed a real limitation you’ve measured? I was able to easily saturate a 1 Gbps link for days using ZFS send over SSH with an Intel Atom when initializing my remote backups.
Yes. I have an Atom on one end of the pipe as well, but it's over 10Gbe and to a 6 drive mirrored and striped vdev. Using SSH as the transport it would only approach 100MB/s, using mbuffer it can max out the storage at about 250-275MB/s.
I've used a few different ZFS snapshot management tools, the one feature I'm missing is something to assist with data retrieval. I'd like to be able to specify a dataset, path and date/time and the tool would mount the closest snapshot to the specified time and pull the appropriate data for me.
55 comments
[ 4.4 ms ] story [ 120 ms ] thread"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
I ended up going with zfs_autobackup - https://github.com/psy0rz/zfs_autobackup because it doesn't need anything set up on the computer to backup. Just a script on the backup server that does pull backups.
I'm not trying to hijack the thread but I've often appreciated links to similar software in threads like this.
in the spirit of link sharing for other tools, I've been moving many of my personal infrastructure & tools to golang-based apps, and I started using this for zfs replication
https://github.com/zrepl/zrepl
No issues thus far in about 6mo of usage for local-and-remote snapshots
One thing I like about Sanoid/Syncoid is that if you set up Sanoid on both sides and use Syncoid to pull, the backup server only requires the ability to send snapshots (which you can configure with "zfs allow"). The backup server also doesn't require root in this case. This means neither server can delete the other's data.
And you still get all the automatic snapshot creation (done on the primary server) and pruning (done on both servers).
"Which would push-replicate the specified ZFS filesystem from the local host to remotehost over an SSH tunnel ..."
Now, if only there were a cloud storage provider that you could directly zfs-send to ...If only ...
rsync.net pricing is 2.5c per GB, per month with no other charges possible, regardless of usage.
I feel like we're in the ballpark there ...
As a bonus, you can do things like this:
... which is detailed here[2].[1] https://aws.amazon.com/s3/pricing/
[2] https://www.rsync.net/products/universal.html
"The last addition is the ability to do raw, encrypted sends and receives. The idea here is to send raw encrypted and compressed data and receive it exactly as is on a backup system. This means that the dataset on the receiving system is protected using the same user key that is in use on the sending side. By doing so, datasets can be efficiently backed up to an untrusted system without fear of data being compromised."
Unfortunately it's taking a long time for the feature to propagate through the ZFS implementations. I don't know if rsync.net's servers support it.
If you have multiple terabytes of data, zfs-send is a very efficient (and elegant) way to transfer them to rsync.net.
We can give you your own zpool, with root, running the latest production/stable ZoL code - so you will have encryption and raw send, etc.
However ...
If you have hundreds of GB up to ... one or two TB ... we paste this suggestion to you instead:
I think you should look into the 'borg' backup tool - it has become the de facto standard for remote backups because it does everything that rsync does (efficient, changes only backups) but also produces strongly encrypted remote backup sets that only you have a key to ... rsync.net has no access to the data.
The borg website is here:
https://borgbackup.readthedocs.io/en/stable/
and a good description of how it works and why you should use it is here:
https://www.stavros.io/posts/holy-grail-backups/
... and our own page with details of our attic/borg support, along with a deeply discounted signup link, is here:
http://rsync.net/products/borg.html
As for FreeNAS, I feel it gives the closest thing to a Synology NAS in terms of manageability through a GUI, etc, while also supporting ZFS.
Maintaining freenas and trying to run some other virtualized apps next to it was not good for me.
Yes, but depending on the size/value of the dataset, rsync might not be the best option.
Even after recent price reduction (Yay!), and with a hn discount (Yay!) - I still consider rsync.net as a personal backup service a little overkill/expensive past 100 GB or so. And likely you could just handle anything on the order of 1 to 2 tb locally anyway (spare disks).
You technically could use it for 10+tb of data... But uploading and downloading is going to be slow unless you have a real uplink (ie at least 1gbps).
And if you did, unless you need filesystem level Metadata - rsync might be a better option than zfs send/receive (even though zfs is cool).
>, rebuild my server, and then pull them back down into a reorganised/renamed tree of datasets?
Again, if you just need the data back - you might be well served with rsync or restic (both choices should be orthogonal to using rsync).
> Are there compatibility issues between ZFS on Linux and ZFS on FreeNAS that I need to be concerned about?
The world is divided in Oracle and not-Oracle - AFAIK there shouldn't be major differences between those two - that I believe now both follow upstream zfsonlinux.
However - I believe rsync.net still doesn't support zfsonlinux/openzfs encryption (zfsonlinux 0.8+). While locally it's possible to encrypt the disk "under" zfs... Native encryption is tempting.
And you would of course never send anyone data without encrypting it?
A fair number of the ZoL 0.8 features (encryption, allocation classes being two prominent examples) are not available currently for FreeBSD
I tried to directly import the Linux created pools into freenas, but they are read only due to a feature property that’s only supported by ZoL.
In read only mode I sent everything to the backup Linux server, destroyed the main pool and sent the whole thing back to the Freenas setup.
For now I'm just doing rsync of the snapshot directories, not upgrading the Ubuntu pool, and waiting on FreeNAS feature support to match.
> Now, if only there were a cloud storage provider that you could directly zfs-send to ...
Is there something on rsync.net's site that describes its support for zfs send?
I've found this old thread on HN referring to an Arstechnica article:
https://news.ycombinator.com/item?id=10751269
and a reference to it on IXSystem's site:
https://www.ixsystems.com/community/threads/rsync-net-now-su...
but they're both back in 2015.
I can't find anything on the site, https://www.rsync.net/index.html, but I may be reading right past it.
Does ZFS (or any other FOSS file system) have a setup like Isilon or NetApp, where you could have any 2 disks in a box of 8 or any box out of say 6 die, and still not lose anything other than bandwidth?
For Isilon, a better comparison would be Ceph. With Ceph you can have a cluster of 10s or 100s of servers, and you can lose 1 disk, or 1 server, or 1 rack or servers and it will recover very quickly.
As long as your minimum requirements for your type of replication or erasure coding profile are satisfied, your data stays safe and available.
https://www.freshports.org/sysutils/sanoid/
(I'm the port maintainer, open to improvements)
For those that actually use perl - I know perl has had (fairly sane?) package management for a while... Is it reasonable to use system perl version in anger?
I would typically reach for a version manager/virtual env with python or ruby - do you, with perl as well?
I'm fairly happy with asdf day to day.
https://asdf-vm.com/
0: https://github.com/paxswill/zfs-apt-snapshot
I ended up using https://github.com/oetiker/znapzend instead, because it can use mbuffer for the actual data transfer, which is much faster.
You can also use SAMBA to expose the snapshots as previous versions in windows.
SAMBA general documentation: https://www.samba.org/samba/docs/current/man-html/vfs_shadow...
Randomly found ZFS specific: https://blog.chaospixel.com/linux/2017/09/zfs-auto-snapshots...