Ask HN: How do you backup your files without depending on a third party service?
I'm using Github, Dropbox and iCloud, but this makes me nervous about my data's privacy and longevity for a number of reasons. Among the three services, I have ~700GB of data.
But I got to thinking, aren't USB thumb sticks reliable and big enough nowadays to fit this much data? I could just buy two 512GB sticks and use rsync to backup to them a few times per week. This way I wouldn't have to lug around an awkward hard drive if I'm ever traveling.
What strategy do you use to keep your data safe and private for the long term? Do you have a portable solution, or do you recommend something else?
111 comments
[ 2.8 ms ] story [ 148 ms ] threadHere's a blog post showing everything from which drive I use to the backup script: https://nickjanetakis.com/blog/automatic-offline-file-backup...
I've been using this strategy for about 20 years and it hasn't failed yet. Although in the older days I had a slaved HD and also used CDs on occasion.
I would never trust a consumer grade USB flash stick. I've had so many fail on me over the years.
Synology in particular is a fantastic platform which does way more than a traditional NAS. It's quite polished personal cloud platform (way more so than FreeNAS) and includes hundreds of apps, even the backups into the cloud can be done in different ways.
I also place a premium on the fact that Synology is a piece of hardware. The software comes built-in and taken for granted. What this means is that it's been sitting somewhere in the dark corner of my house since 2012, self-updating and not bothering me with upgrades or expired credit card / subscription, etc. It doesn't care if I run Linux/MacOS/Windows, it's just dumb storage that all of these OSes can backup to using built-in tools.
Another tool I like is Resilio Sync [1], it's a cloudless Dropbox and it's perfect to sync /Documents and /Desktop across computers at home and at work. It's not a backup tool, but very much related, i.e. if I put a file on my desktop at work I know it will end up on Synology at home and eventually will get encrypted and backed up to Glacier, and yes, without any 3rd party servers involved.
[1] https://www.resilio.com/individuals/
And I too like that it's a hardware business model: I have the sense that my purchase(s) fund development of useful features, not creepy upsell opportunities.
They had a scare several years back about a remote exploit. I think it has caused them to take security and updates seriously ever since.
I think it helps if anyone's concerned with avoiding vendor lock-in.
My friend has one of the smaller Synology boxes with 20TB and it is pretty nice. You can think of it as the Apple of the NAS systems out there, everything just works.
However, if you like to tinker with your NAS a bit more, FreeNAS is excellent. It's pretty comforting running zfs over standard RAID, and you have a FreeBSD system where you can run anything you want in native jails.
I haven't tried Synology in a long time. I'm looking to get my few TBs off of the QNAP and sell it in favour of something custom built.
Another advantage is that they're reasonably open (modules, plugins, etc), and the base operating system is GPL (but unfortunately not the UI / modules), so you can play with it when needed.
I had to play with the source code a few times over the years, and having the ability to understand what the operating system is doing [1][2], or be able to do some hardcore recovery through the serial console [3] is pretty remarkable for a consumer storage product.
(note: I wrote these posts 5-6 years ago, and the software has evolved a lot since them. I haven't checked the source code or used the serial console recently, so would love to hear thoughts if anyone has done it recently)
[1] https://wrgms.com/reverse-engineering-synology-openssh/
[2] https://wrgms.com/synologys-secret-telnet-password/
[3] https://wrgms.com/recovering-a-failed-synology-diskstation-d...
Can you imagine it? NAS'es usually are targeted for business users, yet they dont have full disk encryption...
For now i'm using QNAP, in terms of possible features i believe QNAP is little ahead (x86 cpu, virtualization, containers, fde). It has all what synology has + more.
However their platform is badly constructed, most processes are esentially running as root user... You get like firmware update every 2 weeks. It doesn't mean that there are bugs tho. I'm running it for 2 years and never had a problem.
I also use Jottacloud [1] who provide unlimited (yes unlimited) storage for 7.5 EUR a month. Based in Norway, they keep an up to date warrant canary.
Client side, I backup to both of these services using Duplicati [2] which offers client side encryption, block level deduplication and excellent integration with Jottacloud and SFTP. For synchronisation, I use Syncthing [3] since I don't want to expose plaintext documents to Jottacloud. All in all, I'm very pleased and it has required no maintainence since I set it up a year ago.
[1] https://www.jottacloud.com/en/
[2] https://www.duplicati.com/
[3] https://syncthing.net/
So far I'm pretty happy and impressed, it's a decentralized piece of software that Just Works (TM). My plan for backups is to spin up a VPS somewhere and have them back it up.
I also love that it can use LAN; I can finally sync up my giant mp3 collection.
I have one on my computer, one one our Raspberry Mediabox (with a dedicated drive) and one on a dedicated VS.
New version using inotify really helped the Pi !
1. It is storage location agnostic (backup to a usb, external HD, network HD, dropbox, amazon, etc)
2. your backup data is encrypted locally before being backed up to any medium
What's great is that i can back up one set of data to an external hard dive manually as needed, another set of data to say dropbox on a nightly basis, and another set to a B2 bucket on a monthly basis (all of which are encrypted so i retain my privacy/data security).
Edit: i suppose that it is a "third party service" but i think it reliably solves for the lock-in and privacy problems you reference
1. There's no information about what it is doing (which file are currently being backing up, what's the current upload rate, what's the estimated completion time, how much data is left, etc)
2. Large files (like VM images, or crypto containers) block small files from being backed up. Essentially, your backup frequency can be no more than largest_file/upload_rate (well, it's more complicated than that because of deduplication)
#2 is an issue I've encountered as well
I have a SpiderOak subscription for 2TB that is private enough for my needs, and that's where I backup pretty much any machine i have. I also keep a low-power Linux server at home (a cubox) with two 1TB disks in raid-1, that acts as a Time Machine for the occasional quick restore. Most of my code lives in Gitlab or Bitbucket, and some business stuff in OneDrive. Family pics and videos, that are massive, get pushed every few months to Amazon Glacier.
Until one day I got robbed and I lost both my laptop AND my backups... (and USB sticks, etc). That sucked.
If you’re tech savvy and want to save a few bucks by doing it yourself, check out borg[2]. You’ll need to set up an external server for it but at scale (TB) it’s an order of magnitude cheaper than Tarsnap.
Regardless of off site storage I’d suggest also buying two large USB drives and formatting them with LUKS. Keep them offline and only sync them occasionally as live storage isn’t really a backup.
If you’re not tech savvy I’m convinced what you’re asking for is impossible. The best a layman can do is pick a reputable provider (like the ones you’ve listed).
[1]: https://www.tarsnap.com
[2]: https://borgbackup.readthedocs.io/en/stable/
E.g.: Borg to local RAID every 2 hours, sync RAID with cloud every 24 hours.
> E.g.: Borg to local RAID every 2 hours, sync RAID with cloud every 24 hours.
The problem with a fully online (non-append only) backup system is that if everything is working properly then an accidental 'rm -rf /' will propagate throughout all your supposed backups and destroy them as well. The USB drives are for that type of "Oh shit!" moment as a (possibly stale but better than nothing) backup of last resort
Unless you're going to run your own servers in multiple locations (or convince you friends to let you host a NUC at their homes), you're going to rely on a third party. All that changes is the type of service that you're receiving, i.e. whether it's an object store (S3, B2, etc), a virtualized server, or even a rack in a data center.
Looks promising. Though I wish apps like Borg/Restic had functional GUIs, even though minimal. In fact preferably minimal. Just notifications, list of archives, info about data backed up and actual space used, scheduling etc.
[0] https://restic.net
For the personal backups, I keep one at home (second location) and another in the office. In both cases, I rely on rsnapshot (http://rsnapshot.org/). It's an oldie but a goodie - I've been using it for over 15 years. It solves the rotating backup problem - and thanks to the glory of symlinks reduces my disk usage.
I also use duplicity to gpg-encrypt files that I save into SpiderOak - but feel free to use s3 buckets. I very much like having encrypted backups just work. In both cases I have my scripts notify my via a webhook when things fail - and every 7 days regardless, just so that I don't have a heart-attack wondering if anything happened.
SpiderOak is new - about 8 years. The webhook is newer, about 4. The rest is ancient and continues to work across everything, always.
For this reason, I use rsync to a btrfs volume (both local, on a USB stick, and remote over SSH).
Then I take a btrfs snapshot, so I have versioned and unduplicated backups.
Furthermore, rsync supports gitignore-like files, so you can control what gets and what doesn't get backuped in a very fine-grained way.
Tarsnap is also a great option.
For your particular use case, I'd go for both a remote server and a local pair of USB sticks to have duplicated backups in different locations all the time.
If btrfs were simple software redhat/fedora wouldn't have abandoned it in favor of xfs.
My point is to use utilities that do one thing, and compose well.
Plan9's Venti/Fossil immediately come to mind as an example of simplicity in this vein.
> For this reason, I use rsync to a btrfs volume (both local, on a USB stick, and remote over SSH).
I love simple software too: that's why I use lmnop and throw in a little xyz.
I use rclone to sync with Google and odrive to sync with Amazon. Stuff on my phone (including photos) gets synced to Amazon and Google using the respective photos apps for photos and foldersync for other stuff.
Then use SMB, NFS, Time Machine, rsync, Syncthing, sftp or anything else you like to push data to one of the FreeNAS machines.
Periodic ZFS snapshots prevent common cases of PEBCAK and ransomware.
Fully redundant, encrypted & open-source solution that can saturate 10GbE!
ECC is highly recommended but even a low powered CPU will do.
https://github.com/zrepl/zrepl
* "Back in Time" for keeping older file versions in external hard drives.
The two were chosen mainly because they have decent GUIs, both support diffs, and the backups can be viewed using nothing but a file explorer.
Edit: Granted, time machine provides a MUCH more 'slick' interface
For a bulk backup, once per month I bring one of the 3 portable HDDs from another place, attach it to my server and start my custom backup script that saves encrypted zips of sensitive directories and rsync-backup of everything else to it. At any time 2 or 3 HDDs are outside the house.
Then for large archival files I use: https://www.backblaze.com/b2/cloud-storage.html
And just use Cyberduck to move the files over. Super cheap and gets the job done. Beauty of B2 is you can use a plethora of clients to move files: https://www.backblaze.com/b2/integrations.html
This still requires the user to remember this process but it is about as minimal as I can imagine for a local backup that is mostly kept offsite.
I'm thinking about this in particular for a local non-profit that has privacy concerns with storing their data with a 3rd party.
I'm considering putting something together using a Raspberry PI or other SBC. Having the result be under a $100 USD with 128GiB-256GiB of storage would probably be necessary for it to be acceptable.
Under the hood it's basically doing that, but everything's SHA-256 checksummed so you can check its validity, and all copies of your data also sync metadata telling them who else has copies, it's easy to enforce policies that some content must have >N copies etc.
With this method, a virus or something like a fire that only affected one location would not be able to take out all copies of my data and, at most, I'd lose the most recently backed-up information.
It's not 100% bullet-proof, but it's good enough for my personal purposes.
Every night this system also reaches out to a remote system in my office that has an 8TB RAID-5 array. It "pushes" any changes, again using rsync, to this remote host. This time it uses the --link-dest argument to create nightly "snapshots." If a file hasn't changed, it simply gets hardlinked. This backup volume is also encrypted, and only the core file server has the key to decrypt it.
For the remote system, I have a script that runs periodically that purges backups according to my own specification. Right now it keeps unlimited yearly backups, three years of monthly backups, six months of weekly backups, and 14 days of daily backups.
Once a month I also run a script to back up the core file server to an external 8TB hard drive, again with --link-dest, creating snapshots. This hard drive is stored in a fireproof safe. It is also encrypted.
I have various levels of redundancy elsewhere, too. (Yes, RAID/redundancy is not a backup). For example, my desktop system has a RAID1 array in it.
In 2000, I installed Windows ME on my desktop system. It corrupted my hard drive somehow, and I lost all of my work prior to then. In case you can't tell, this scarred me for life, and I have worked hard to make sure it cannot happen again.
I'll add, every single storage device that I use is encrypted. Sometimes it's just with a key stored on a separate boot disk (usually an SSD). The rationale being if I ever have to RMA a disk, I don't want the manufacturer to be able to see any of my data.
On another tangent, the remote system mentioned above is actually a rather old GX260 with a SATA expansion card and 3 4TB disks. The boot disk is an ancient EIDE 40GB drive. I keep it around mostly for entertainment at this point. Here's why:
That's right, the drive has been spinning for a little over 14 years.Unfortunately I've recently hit a problem where e2fsck runs out of memory (it's a 32-bit CPU, so only 2GiB per process) checking the file system. I tried setting a scratch file, but that bombed for some reason too. So, I may finally have to upgrade the thing to a 64-bit system.
Oh well.
I had a friend in college that backed up all of his files on a flash drive but he used git rather than rsync.
O_o ...For some reason, my mental model of SSD rates was stuck in 2014 at $0.50/GB. As I type this, my laptop is busy moving a VM image over to the NAS to free up some space on my 256 GB SSD.
But no, according to https://pcpartpicker.com/trends/price/internal-hard-drive/ and https://pcpartpicker.com/products/internal-hard-drive/, you can indeed find SSDs at $0.20/GB (the low end being currently dominated by Crucial MX500 drives at $0.18/GB). Looks like it's long past time to upgrade!
It was more the way it failed - I had one or two boots where a few errors started to pop up, and then the next boot, nothing - the disk had disappeared to the BIOS' eyes and any other tools - just dead as a doornail with no easy way of getting any of the data off it.
I assume they've improved in that regard, or are spinning platters still a safer proposition over the long term?
> I assume they've improved in that regard, or are spinning platters still a safer proposition over the long term?
I don't think you need to assume..just check out what statistics are out there. Sadly, I don't know of anyone publishing numbers for consumer SSDs like Backblaze does for spinning disks.