Ask HN: How should organize and back up 23 TiB of personal files?
I take a lot of photos, so a lot of these are files that I would actually want to back up, but many of them are old operating system installs and other "useless" files that I don't need archival storage for.
The actual size of the data that I need backed up I would estimate at around 6 TiB.
A few of my requirements:
1. I don't need the files to be accessible online, in fact, I would prefer if they were not.
2. If anything is backed up to the cloud, I want pre-internet-encryption with keys that only I know and control.
3. I want something simple, that could be recovered using a pragmatic approach and open source software in case of a disaster.
4. I'd like a system where I can easily test my recovery strategy.
Open questions:
1. What local filesystem setup should I use? Number of drives? Local backup approach?
2. If you've done this before, is there a strategy that you used for the actual aggregation of the data? Are there any particularly convenient IDE to USB docks? Any good software that you would recommend for locating duplicate files?
3. What remote backup software should I use?
[ edits ]
Answering some questions from the comments:
Cost: Given a quick look at the cost of archival cloud storage, I guess I would be willing to spend up to $60 per month on a remote copy. (Noting the estimate of 6 TiB of "acutal" data)
How often: I would expect to access the remote files rarely (maybe once a month), and need a complete recovery very rarely (with a low requirement for recovery speed). Local backups I would like to occur at least weekly, with a verification or access frequently (daily?).
Risk tolerance: For the local-encryption-for-remote-storage aspect, I would like something with a high level of confidence in the cryptography and the implementation surrounding it. I would also like a high degree of confidence that I can recover my files in case of a natural disaster or similar that wipes out my local copies.
Local security: I live in a relatively secure home in a relatively low crime area. I could store a copy at a relative's house, although I may move far enough away soon that it would not be practical to deliver or access such a backup.
101 comments
[ 4.6 ms ] story [ 172 ms ] threadCollecting and arranging the archive is the big job. you're just gonna have to bite down and start doing that, yourself: no one else is likely to know what needs saving or not. Set up a NAS or file share with a big HDD and start collecting files there.
You may find the old stuff fun to scrape up; for example how difficult is it to find a PATA interface today? that problem only gets harder. Motivation to get on the job now, rather than later; and to make the "archive maintenance" more of your everyday task list than to let it pile up.
It's not exactly matching OP's requirements, but my solution is to regularly rsync important stuff to a small home server, on that homeserver do a borg-backup of the backup folder once a week onto a second drive. First one is ssd, second spinning rust.
In ops case I think the biggest problem is sifting through all the data. I did this a few years ago with only 4 or 5 old drives and a dozen dvds and it was exhausting, but also occasionally pretty nostalgic when I found photos from college or old programs I wrote as a teen.
As a first low-budget bandaid I got a single 16TB enterprise HDD (Toshiba MG08ACA16TE) in an external dock that is plugged in to a Prodesk 400 G6.
I regularly snapshot and sync all my btrfs drives with btrbk (all my linux devices and several of my external drives are btrfs). The Toshiba is being deduped with bees to not make this too inefficient.
For important files I have syncthing folders that continuously synchronize between my devices, as well as above solution.
For some dead media I use blurays. Still looking for a solution for Windows, though there I at least have OneDrive.
It's a mess to be honest.
Eventually I want to have big enough RAID for file centralization, restic cloud backups, network shares and so on.
A crucial piece I'm still missing is a synchronization, hosting and backup classification/policy for the various file types I have... This may well be the more difficult thing compared to just getting any redundancy.
With 4 bays you can definitely fill them up and go for striping to increase capacity and redundancy but high TB disc's are $$
1: https://www.tarsnap.com/
Still, that's $375/month. Nothing to shake a stick at.
I think the intended use case of tarsnap is for <100 GB raw data that is really important to you, for which you must have an offsite backup that is (according to them) insanely backed up and safe. I could see myself using them for a curated folder of personal documents I can't afford to lose in the event of a house fire. Then again, I could probably fit all my most important documents into my 1Password vault.
You could just make four copies of all your data every month, and throw hard drives in all directions.
My current combination is just Backblaze + TimeMachine for local backup. I also mirror it to a Synology with CarbonCopyCloner.
All the encrypted, cli tools I have used in the past I abandoned again as it was always annoying to maintain and monitor and impossible to explain to anyone.
The pricing looks too good to be true. What’s the catch?
1. They will delete all backups of your external hard drive if you do not plug them in and fully sync them for at least ~12 hours every 30 days. You can pay an extra fee each month to have your external drives stored longer to mitigate this problem [1]. The fee is per GB so it does materially change the Backblaze pricing if you have a lot of external data.
2. Their client is extremely extremely slow, even with performance settings set to their recommendations it just obliterates CPU (irrespective of the size of the actual changeset being backed up). I've always been able to feel my computers slow way down when the Backblaze client spins up. [This issue may be resolved on Apple Silicon, but it's been a nuisance on every Intel Mac I've ever had.]
There are some other weird quirks in their macOS client, like you can set your backup schedule to "Once Per Day" overnight and it will still, multiple times per day, just start running unexpectedly and then you have to deal with the slowness or hit "Pause Backup". Somehow this happens 2–3 times every single day. I've tweaked the settings, reinstalled the client, and all of the things, but this issue has persisted for 5+ years on every machine I've used their software on. Super annoying.
Sidebar: They have a native mobile app but it is really really bad. Slow, clunky, somehow always getting logged out. The mobile client sucks so bad that I usually just wait until I'm at my desktop to do a restore.
[1]: https://www.backblaze.com/blog/backblaze-7-0-version-history....
Some would say there is a third. Decryption during restore is handled on the server, not on the client. They plan to support doing all encryption/decryption client side, but if that is something you need now Backblaze's client backup service may not be the right choice.
Here's a comment with more details and references [1].
[1] https://news.ycombinator.com/item?id=38862215
Most users don’t use much in terms of HD space for their backups so they can absorb the extra cost for people who back up large amounts and since they are using their own lean-focused servers with consumer drives their cost per TB is much lower than a company backed by S3.
They are also the company that puts out drive stats every quarter.
And they open-source the hardware for their rack-mount “pods”.
They also added another revenue stream: B2. It’s an S3-compatible storage service with a decent profit margin that still manages to be 1/4 the cost of S3.
TL;DR: They have a solid business model built around providing that low price.
Do you just run a periodic task to sync to the SMB folder?
You can set up zpool scrub every week and email yourself the results to check the files are surviving
ZFS:
- ensures you don't get bit rot
- manages both disks (raid/mirrors &c) and the filesystem, it's an all-in-one solution
- supports block level replication to local and remote systems, after the first backup it's fast
- can create dynamic partitions to group files together and build replication strategies around
Choose either RAID or mirrored drives (https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...) I've gone mirrored but more for flexibility and performance. Use a calculator to see what options of disks you have https://jro.io/capacity/ (and google 'ZFS calculator' for others)
For backup get a second machine somewhere else in your house with a smaller setup and use ZFS replication to keep it up to date with everything on the main box you need backed up. Currently I use a raspberry pi with a USB disk but this is perhaps cutting it fine. You wanna keep this online so ZFS can periodically check the health of the data on the disks. Fully offline backups can be a risk.
Finally for a 3rd backup use some of those external drives, format to ZFS and use replication. Plug them in on a schedule and take a backup.
If you want to backup to remote systems (cloud/a box in your parents house) it also supports filesystem encryption. With the right options you can stream incremental backups over SSH only passing encrypted blocks. The system at the other end never needs to see the raw data.
* disastrous * very upsetting * disappointing * meh
(It might also help to consider when you last needed to access any of it?)
Because honestly, my bet (without judgement) is that there's likely a significant amount of data in there that simply doesn't warrant keeping. I base this on my own habits (I have to actively fight a hoarding tendency digitally and in real life) and also knoweldge of friends who (while otherwise very well adjusted) seem to find digital hoarding easy to fall into - maybe because it has less of a visible life impact than physical belongings.
The latter works well for larger datasets, outputs a TXT which you can analyse and decide what to do with.
https://en.wikipedia.org/wiki/Borg_(backup_software)
https://news.ycombinator.com/item?id=21642364 ("BorgBackup: Deduplicating Archiver", 103 comments)
https://news.ycombinator.com/item?id=34152369 ("BorgBackup: Deduplicating archiver with compression and encryption", 177 comments)
*You definitely don't want your private filenames leaked to data brokers, like Backblaze's clients experienced.
https://news.ycombinator.com/item?id=26536019 ("Backblaze submitting names and sizes of files in B2 buckets to Facebook", 517 comments)
I had the same type of thing happen to me with the LastPass leak, which made me very wary of closed-source supposed-e2e encrypted services.
Looking at Borg, it doesn't seem to have native support for the actual transfer of the remote backup. It seems like if I used Borg it would be for a local backup, and then I would need an additional layer to sync the backup to remote storage.
It looks like rclone does, mentioned in other comments here, any others?
They provide some of the best cost-per-GiB storage. I've been using them for 3 years, and I prepay a year in advance. I've been super happy. They're cheap; maybe they'll suffer data loss, but I practice the 3-2-1 backup strategy[2]; if they lose data, I have a local backup; if by some weird chance both InterServer goes down and my house burns down at the same time, I've got another backup drive at my parents' house.
If you're more risk-averse, there's rsync.net[3], but it is substantially more expensive. However, it has really good data backup practices on its side.
[1] https://www.interserver.net/storage/
[2] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
[3] https://rsync.net/
1. How can you tell if they’ve lost data that requires pulling from the local backup
2. What happens if they silently corrupt the data?
[1] https://borgbackup.readthedocs.io/en/stable/usage/check.html
Its very simple to have a "dumb" drive that simply stores the files and use annex to remember which drive it is. Also, to track and remember that you only have it in one place in case you want more copies. It can also push files to s3, glacier, and some other backup repos if needed with client side encryption (code between either symmetric or gpg)
For example, if your house burns down, it doesn't matter if you have 10 mirrored copies of the same 8 TiB drive in the same room. If your parents get a new housekeeper who goes on a cleaning spree, it doesn't matter if you have 10 mirrored copies of the same 8 TiB drive in the same shoebox.
How often: I would expect to access the remote files rarely (maybe once a month), and need a complete recovery very rarely (with a low requirement for recovery speed). Local backups I would like to occur at least weekly, with a verification or access frequently (daily?).
Risk tolerance: For the local-encryption-for-remote-storage aspect, I would like something with a high level of confidence in the cryptography and the implementation surrounding it. I would also like a high degree of confidence that I can recovery my files in case of a natural disaster or similar that wipes out my local copies.
Local security: I live in a relatively secure home in a relatively low crime area. I could store a copy at a relative's house, although I may move far enough away soon that it would not be practical to access.
For the long-term backups that sound more sentimental, the "put lots of 6 TiB HDDs from different manufacturers in safe places" becomes more feasible when you don't have to update it weekly and when it is OK to wait a few days until so you can drive to a relative's house after a disaster. But you also have to test those drives every now and then.
I have a bunch of long-term drives at my parents' house a 5 hour flight away, but I visit them every winter and do a test every year. If I needed those today, it might cost me $1000 to drop everything and get on a plane. But if I'm visiting every winter and it isn't really urgent, I can wait. You should think of those kinds of costs too.
So I've been enjoying b2 (as an example, you can Fuse mount to browse the files without downloading). But just for backup don't Amazon glacier or Google cloud archive is so so cheap. If you wanted to be paranoid you could do both separately
I haven't independently audited rclone's encryption layer
I used to use external hard drives for backing up large amounts of data, but nearly all of them failed (4 out of 5 of my external hard drives broke. I can't get data of them because there are multiple I/O errors, even though RAID says the drive is fine. Other devices even fail to show up at all)
So I actually decided to get a Synology NAS and use it exclusively as a backup target.
Glacier shouldn't even be your redundancy solution, it should be the redundancy of your redundancy :)
Like you're alluding, if you actually need to pull the data out of glacier, you've probably screwed up in the design/testing of your primary backup systems, but we're all fallible so it's an insurance policy.
[0] When you pay for 6TB you get 18TB free egress making it functionally free since you’d need to to >3 full restores a month to incur a charge
I think all the recommended backup SW (restic, duplicity) encrypt before storing. I use restic but haven't exactly been hugely exercising it, so can't place much weight on my experience . But it should be okay with this quantity of data.
Generally I'd guess that diversity of backups wins over using a single expensive one in terms of reliability, but at the expense of more overhead managing them.
See: - https://rclone.org/crypt/ - https://rclone.org/b2/
> When rclone uploads a new version of a file it creates a new version of it. Likewise when you delete a file, the old version will be marked hidden and still be available. Conversely, you may opt in to a "hard delete" of files with the --b2-hard-delete flag which would permanently remove the file instead of hiding it.
https://rclone.org/b2/#versions
The limitation is that it's going to be somewhat more tricky to retrieve a consistent snapshot, because each file is versioned individually - rather like using CVS instead of git.
However S3/B2 versions are still a nice feature because they allow a simple way to provide the property of 'non-fate-sharing' which is a great property to have in backups but is otherwise difficult to arrange without multiple machines. Essentially, two systems fate-share if someone who hacks one can delete both. So Ideally the machine being backed up should not be able to delete the backups, and the machine (or service) hosting the backups should not be able to delete the original. But, dumb storage does not have this property (if you keep it online) and using an online service as dumb storage doesn't either. But if you use a credential which isn't allowed to delete versions in your backup tool, enable versions, and keep your backblaze admin credential elsewhere, then you have non-fate-sharing. So, it's a good idea to enable versioning even if you use restic which does its own versioning (and I don't think this will eat more space, because restic names files by content)
Another limitation of using rclone as a backup tool is that different rclone backends support different metadata (because the underlying systems do). So, it's not suitable to back up the whole of your OS, or something where you care about not losing the metadata. Eg, the b2 backend doesn't support metadata, so if you back up your OS using it you won't know what file was owned by who.
I just copy everything to HDDs using a Plugable USB-C/SATA dock (nicer and far more reliable IMO than those $9 dongles you see around). I then put the drives in a Turtle HDD case for padding against environmental factors. That protects me against everything except house fire/theft/tornado sucking up the case and dropping it in another state.
My backup needs are beyond any single drive, but at 23 TiB you don't have to purge much to fit on a single hard drive...there are 20 and 22 TB models for sale.
I'd buy a drive WAY larger than 6 TiB just in case you underestimated how much you actually want to save. Having the extra space would also allow you to incorporate error-correction techniques like generating PAR2 files (I did that with some emotionally-important personal files).
I did this once with a buffalo raid nas. It died and I was left with 10 years of my life inaccessible. Much sadness ensued.
Distribute the metal cards as widely as you can, at home, potentially in safety deposit boxes or buried somewhere (potentially with an additional layer of protection from a brain memory password.)
This protects against a class of physical data destruction from house fires, theft, floods etc.
However I'd withdraw this recommendation if you live in a jurisdiction where you can be compelled to hand over passwords, e.g England
Does it mean there's more than one right answer, or that no one solution is ideal?
Each has tradeoffs and “lock-in” (which cause people invested in an answer to avoid others)
If money is no object, just buy a synology NAS drive and be done with it. They do everything you want and more, and are incredibly user friendly.
Use their BTRFS filesystem with SHR Disk groups to get multiple disk redundancy alongside data scrubbing for bitrot protection.
It also contains software to connect to any cloud provider for remote backups if that is what you want.
EDIT: Something like a synology DS1621+ would do you well.