No workflow, we just use BackupPC (http://backuppc.sourceforge.net/) - cant recommend it enough. Restores are easy, monitoring and automation on different schedules is all built in. It's really great.
Love backuppc, especially 4.X where they replace the somewhat ugly hack of using hardlinks with checksums. Recommended if you have a collection of systems, if it's just a single system I'd use rdiff-backup (if you have remote storage) or duplicity if you want to pay for remote storage (amazon, rackspace, or similar).
Put all personal data on a zfs z2 RAID system (FreeNAS). Take regular snapshots.
Someday I'm going to get a second offsite system to do ZFS backups to, but so far the above has served well. Then again I've been lucky enough to never have a hard drive fail, so the fact that I can lose 2 without losing data is pretty good. I'm vulnerable to fire and theft, but the most likely data loss scenarios are covered.
I do something very similar, my oldest build gets synched once a year and is in cold storage. I hope I never need it. The hard drives from my previous build hold cold copies of the most valuable volumes (monthly sync), and the current system has plenty of redundancy and snapshots (insurance, not really backups). I use zerotier to stay in sync when away from home.
Your raid array hasn't failed you until it does one day. Make backups, they don't have to be off-site. Don't wait, speaking from irreparable experience.
Even in 2017 you still can't sneakernet sometimes. I have a ZFS file server similar to yours, and it acts as the backup destination for all my other devices. Then, I use two USB HDDs to back up the server, then bring one to work. If either is attached a nightly script bring it up to date. I just keep rotation the two between home & work.
Backup to external hard drive with btrfs. Rsync is used to copy the full source file system, with short exception lists for stuff I don't want backupped. After the sync, a btrfs snapshot is taken to get history. These napshots are removed with an exponential strategy (many snapshots are recent, few are old, the oldest is always kept), keeping ~30 snapshots a year.
Backup takes ~10min for searching 1TB of disk space. The daily diff is typically 6..15 GB, mostly due to braindead mail storage format...
I want to keep it simple but still have full history and diff backup: no dedicated backup tool, but rsync + btrfs. A file-by-file copy is easy to check and access (and the history also looks that way).
If the source had btrfs, I would use btrfs send/receive to speed it up and make it atomic.
I have two such backup disks in different places. One uses an automatic backup trigger during my lunch break, the other is triggered manually (and thus not often enough).
The sources are diverse (servers, laptops, ...). The most valued one uses 2 x 1 TB SSDs in RAID1 for robustness.
> If the source had btrfs, I would use btrfs send/receive to speed it up and make it atomic.
btrfs subvolume snapshot / send are not atomic, for various definitions of atomic.
Unlike zfs, subvolume snapshots are not atomic recursively. That is, if you have subvol/subsubvol, there's no way to take an atomic snapshot of both. At least this one is obvious, since there's no command for taking recursive snapshots, so it tips you off that this is the case. Not having an easy way to take recursive snapshots, atomic or not, is a different pain point...
What's more insidious is that after taking the snapshot, you must sync(1)[0] before sending said snapshot, otherwise the stream would be incomplete! I'm invoking Cunningham's Law here and saying for the record that this is fucking retarded. I have lost files due to this...design choice.
Moreover, though is is probably a fixed bug, I used to have issues where subvolumes get wedged when I run multiple subvolume snapshot / send in quick succession. I'd get a random unreadable file, and it's not corruption (btrfs scrub doesn't flag it). Usually re-mounting the subvolume will fix it, and at worst re-mounting the while filesystem would fix it so far. I haven't had it happen for a while, but it's either due to my workaround -- good old sleep(60) mutex -- or because I'm running a newer kernel.
I can't wait until xfs reflink support is more mature: that'll get me 90% what I use btrfs for.
"rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership (if it is running as root), modification times, acls, eas, resource forks, etc. Finally, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted."
With rsync, using the --link-dest option to make a complete file tree using hard links to a remote server.
Cron runs it, on @reboot schedule. If the backup is successful, some (but not all) old backups are deleted. I delete some oldest preserved backups manually, if disk space runs low.
I've long used rsnapshot for automated incremental backups, and manually run a script to do a full rsync backup with excludes for /tmp, /sys and the like to an external drive.
http://www.snebu.com -- something I wrote because I wanted snapshot style backups, but without the link farm that rsync-based snapshot backups produce. Snebu does file-level deduplication, compression, multi-host support, files are stored in regular lzo-compressed files, and metadata is in a catalog stored in a sqlite database.
Only real things missing is encryption support (working on that), and backing up KVM virtual machines from the host (working on that too).
My installation of Ubuntu is so standard that I'm one apt install and rsync (of the home directory) away from the same result and don't have to manually configure LVM, encryption, etc.
My Arch and Gentoo installs have gone somewhat further than that 8)
Admittedly I'm not sure that you can consider one of them that went from i386 to amd64 as the same install simply because /var/lib/portage/world and /etc/portage/ (plus a few other bits) are the same. I still use it and it's on its fifth or sixth incarnation as my current laptop.
>I also rsync my entire home directory to an external drive every other week or so.
This is what I do as well, just not quite as often. Sometimes I wonder if I should switch to something like rdiff-backup to get snapshots, but that would only really be useful if I accidentally deleted a file and didn't notice for a while, for instance, which in practice is not a serious problem.
If I had more time and inclination, I might set up a small 2-drive RAIDed NAS box, and do automated regular backups to that. But for my laptop PC, just doing regular syncs to an external HD seems to be fine for now.
Drive failures are really rare until they happen when your data is personal experience of a few devices. When you worry about a data centre or n and or rather a lot of other systems with storage then you realise that it happens with monotonous regularity.
Now keeping a few short copies is also fine provided you don't make mistakes. Have you ever wanted to recover from a cock up you did six months ago or three years ago?
You do offsite (Tarsnap), so you have covered off local failures - cool.
Everyone's needs are different and the value they place on their data is different but I would respectfully suggest that you think really hard about how important some bits of your data are and protect them appropriately.
My Tarsnap backups are versioned so I can recover dotfiles and projects from virtually any week several years back. And the projects themselves are usually under git.
I sync data using git-annex to a home server (atom with a couple of drives) and a hosted dedicated server. Technically not "backup" since I don't store old versions.
My work files are kept in git repos which I push to the same servers.
I use Ansible to configure my machine, so that I don't have to backup system files, just the playbooks.
I have a script that checks which external drives are connected and when it finds the correct ones it backups to them using rsync. (Only /home is backed up but it varies depending on which external drive is connected)
Crashplan for docs/media. As for the system, I've been meaning to set up proper automatic btrfs snapshots/rsync but haven't gotten around to it yet. Worst case scenario all the docs/media are on their own RAIDZ array, so if some weird system corruption ever happens I can just reformat/reinstall.
rsync /home, /etc, and a text file of the currently installed package list to my homebuilt ZFS RAIDZ2 NAS running Arch Linux. This has been enough to recover when my laptop drive fails.
Each device gets its own ZFS filesystem and is snapshotted after rsync.
FolderSync on Android does this automatically when I'm on home wifi. AcroSync for Windows. Both FolderSync and AcroSync are worth the small purchase price. Cronjobs for nix machines. iPad syncs to Mac which has a cronjob.
Stuff I really don't want to lose (photos, music, other art) are on multiple machines + cloud.
Just keeping my home folder safe, everything else is up and running in less than one hour if I had to start over. Daily backups with borg backup plus a couple of git repos for the dotfiles. All the important stuff is small enough for this, my backup is like 25 gb (a lot of random crap included), and all the photos and videos we used to worry about a few years ago is up in some unlimited sized google cloud for free. Times are pretty good :)
85 comments
[ 3.3 ms ] story [ 144 ms ] threadIt has an ugly looking interface, but the core of the product is super reliable.
Someday I'm going to get a second offsite system to do ZFS backups to, but so far the above has served well. Then again I've been lucky enough to never have a hard drive fail, so the fact that I can lose 2 without losing data is pretty good. I'm vulnerable to fire and theft, but the most likely data loss scenarios are covered.
I've got incremental snapshots sent to a box in a family member's house (and vice versa).
2. Clone said drive to an external drive. Detach and lock it in a water/fireproof box when not in use.
3. Swap external drive with another that is stored off site every week or two.
4. Swap with yet another off site external drive less often (a few months).
Backup takes ~10min for searching 1TB of disk space. The daily diff is typically 6..15 GB, mostly due to braindead mail storage format...
I want to keep it simple but still have full history and diff backup: no dedicated backup tool, but rsync + btrfs. A file-by-file copy is easy to check and access (and the history also looks that way).
If the source had btrfs, I would use btrfs send/receive to speed it up and make it atomic.
I have two such backup disks in different places. One uses an automatic backup trigger during my lunch break, the other is triggered manually (and thus not often enough).
The sources are diverse (servers, laptops, ...). The most valued one uses 2 x 1 TB SSDs in RAID1 for robustness.
All disks are fully encrypted.
btrfs subvolume snapshot / send are not atomic, for various definitions of atomic.
Unlike zfs, subvolume snapshots are not atomic recursively. That is, if you have subvol/subsubvol, there's no way to take an atomic snapshot of both. At least this one is obvious, since there's no command for taking recursive snapshots, so it tips you off that this is the case. Not having an easy way to take recursive snapshots, atomic or not, is a different pain point...
What's more insidious is that after taking the snapshot, you must sync(1)[0] before sending said snapshot, otherwise the stream would be incomplete! I'm invoking Cunningham's Law here and saying for the record that this is fucking retarded. I have lost files due to this...design choice.
Moreover, though is is probably a fixed bug, I used to have issues where subvolumes get wedged when I run multiple subvolume snapshot / send in quick succession. I'd get a random unreadable file, and it's not corruption (btrfs scrub doesn't flag it). Usually re-mounting the subvolume will fix it, and at worst re-mounting the while filesystem would fix it so far. I haven't had it happen for a while, but it's either due to my workaround -- good old sleep(60) mutex -- or because I'm running a newer kernel.
I can't wait until xfs reflink support is more mature: that'll get me 90% what I use btrfs for.
tl;dr: btrfs: here be dragons!
[0] https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
"rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership (if it is running as root), modification times, acls, eas, resource forks, etc. Finally, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted."
[1] - https://github.com/sol1/rdiff-backup
Cron runs it, on @reboot schedule. If the backup is successful, some (but not all) old backups are deleted. I delete some oldest preserved backups manually, if disk space runs low.
http://rsnapshot.org/
Only real things missing is encryption support (working on that), and backing up KVM virtual machines from the host (working on that too).
Plain old btrfs snapshot + rsync to local usb drive and offsite host for /etc, /var, /root
My (two) servers: dump of db, rsync of dumps and files to another server.
It's ok only because I've got little data.
The important stuff (projects, dotfiles) I keep on Tarsnap. I also rsync my entire home directory to an external drive every other week or so.
Similar for servers but I do back up /etc as well.
Admittedly I'm not sure that you can consider one of them that went from i386 to amd64 as the same install simply because /var/lib/portage/world and /etc/portage/ (plus a few other bits) are the same. I still use it and it's on its fifth or sixth incarnation as my current laptop.
Trigger's broom?
(OK https://en.wikipedia.org/wiki/List_of_Ship_of_Theseus_exampl...)
This is what I do as well, just not quite as often. Sometimes I wonder if I should switch to something like rdiff-backup to get snapshots, but that would only really be useful if I accidentally deleted a file and didn't notice for a while, for instance, which in practice is not a serious problem.
If I had more time and inclination, I might set up a small 2-drive RAIDed NAS box, and do automated regular backups to that. But for my laptop PC, just doing regular syncs to an external HD seems to be fine for now.
Now keeping a few short copies is also fine provided you don't make mistakes. Have you ever wanted to recover from a cock up you did six months ago or three years ago?
You do offsite (Tarsnap), so you have covered off local failures - cool.
Everyone's needs are different and the value they place on their data is different but I would respectfully suggest that you think really hard about how important some bits of your data are and protect them appropriately.
Fuck ups are hard to recover from 8)
2 HD failures in the last 4 years. Not rare for me :-(
My work files are kept in git repos which I push to the same servers.
I use Ansible to configure my machine, so that I don't have to backup system files, just the playbooks.
Each device gets its own ZFS filesystem and is snapshotted after rsync.
FolderSync on Android does this automatically when I'm on home wifi. AcroSync for Windows. Both FolderSync and AcroSync are worth the small purchase price. Cronjobs for nix machines. iPad syncs to Mac which has a cronjob.
Stuff I really don't want to lose (photos, music, other art) are on multiple machines + cloud.
https://news.ycombinator.com/item?id=12999934
https://news.ycombinator.com/item?id=13694079