I put everything in a Resilio Sync folder, and keep a full sync on at least two devices (a home NAS and a cloud seedbox). Resilio Sync handles pretty much everything. You instantly get hot backup, have files immediately available on every device you have, and if you have a phone you can download any file on demand, etc. Unlike other file synchronization methods such as `rsync --delete`, it keeps a version for every file modification and moves the file to Archive when it is deleted, so you can't lose data. Also, you get encryption without the headache by using the "encrypted folder key". This syncs an encrypted copy so other devices can sync from that device.
I use resilio sync together with a NAS (which is still large enough to hold all my files, I wonder how long that'll stay given how much 6k BRAW footage I'm shooting), and I also sync to a Google Cloud Storage bucket from my NAS (moving items to Archive storage class if they haven't been touched in a while, significantly saving cost).
I use borg backup to sync all our files (from ~5 machines) to an old PC which synchronizes everything to the cloud. Borg has deduplication, compression, encryption, and saves several versions of the files which I think is crucial. Also all text-based data are on Git like software, papers, etc.
Good point. I am indeed synchronizing the borg file. I have however my script run borg check before it synchronizes to the cloud. It would be better to synchronize the source files but the encryption and deduplication are what's keeping me from doing that.
It's a good discussion. I've taken a different path but with the same ideals.
1. All my live, working data is in Dropbox. I've been using my (paid) Dropbox account for over a decade. This step isn't really intended to be a backup per se, but you get the backup behavior "for free" because this is how I keep my secondary machine in sync with my primary. My Dropbox folder is also replicated to a 3rd computer in my home as another level of redundancy.
2. Everything in my home dir, especially photography, is ALSO covered by Backblaze. Having some backups elsewhere is mandatory; not enough people really understand how important this is. Should my house burn down, I still have data.
3. My primary system is a Mac, so I use Time Machine. TM is the only backup on this list I've ever actually used as a backup. When our home was robbed a number of years ago in a quickie smash-window-and-grab affair, they got my laptop. I went out and bought a new computer, plugged it into the TM drive, and in an hour or so I was right back where I left off. Hard to beat that. Even my app windows were in the same place.
4. Periodically, I take a full clone of my main machine's drive using a drive imaging tool (Mac specific; I use SuperDuper) and **store that drive at a friend's house**. This probably only happens a few times a year a this point. I should do it more often.
That tertiary computer I mentioned in step 1 is also the home NAS server / home media server. It holds the photo archive in a large outboard disk. Backblaze covers that disk, and Time Machine on that computer keeps the outboard disk backed up as well. This data is mostly static, so the images I've taken of it and stored elsewhere don't need to be updated all that often (ie, just when I migrate prior year photo data to that drive).
I currently trust Restic with basically all of my long-term backups, which, according to the author, really isn't a thing I should do.
However, I'm still somewhat confident in my strategy as I backup my all of my data to two entirely different repositories, one of them backed by Google Cloud, and another by the server sitting in my pantry. So one of these repositories could get irrecoverably corrupted and I still wouldn't lose any of my data. With cloud storage becoming so cheap I've also thought about adding a third repo.
Of course this would not protect me from a hypothetical bug in Restic that corrupts all my repositories before I notice, so maybe I should also add another auto-backup solution into the mix.
Doing manual things like moving data to external storage seems like a robust strategy, but I really don't trust myself to do something like that nearly often enough for it to be useful.
I have used restic for quite a while. Once in a while, I test that I can restore my backups. That's an important step that lots of people miss.
I had a client that asked me to setup their system. I setup the system, they got a tape drive and I had them rotate tapes daily. There was a cronjob to tar everything to tapes.
It was great until their hard drive failed and I found I had a typo where it was only backing up the current folder, not the whole drive. Needless to say, they found a new IT provider and I learned an important lesson. If you haven't tested your backups, you have no backups.
Being an Apple chap, I have found that Time Machine works really well. I use a Synology 5-disk NAS. It has a ton of capacity, and allows me to increase capacity, if needed.
My important stuff is in the cloud.
I have a nightly "hot backup," that is a CC clone of my main and dev drives. It used to be every 4 hours, but I found that to be overkill.
I almost never need anything more than the "hot backup."
I'm near a very similar setup. Syncthing for file replication, storage/backup to ZFS TrueNAS, then a secondary backup to another ZFS system that's usually offline.
Darn, I was hoping this would be an article about organizing one's files. I was really in the mood for reading about that, then spending the rest of the morning reworking my own system.
Same. I get lost in researching everyone else's file organization and other workflow methods. I always feel like mine is a dog's breakfast, scattered across many incongruent, un-coordinated drives, clouds, shares, etc.
Can someone point me to the flaws of my method? I simply use Google Drive.
Me and my partner have multiple devices (tablet, iPhone, Android phone, multiple Mac and Windows laptops) and we just sync folders to our desktop. We just store everything we don't want to lose in Drive. We share folders we both need. Photo's we take on our phones are automagically backed up in the cloud, music and movies are streamed.
My house can burn down overnight and we won't lose any valuable data.
Although I'm not sure of the rate, people have reported getting banned from google services without notice for seemingly no apparent reason. And there is little recourse to unlocking their account.
It's something I personally don't worry about, but something to consider when relying on their services for backing up data.
The flaw is that you depend on Google. You can not trust them at all, for anything. You can use Drive as a redundant fifth backup either for data that you’ve encrypted yourself or for things that you don’t mind being public, but not as the only place that has your data. Things that could very well happen: you get locked out with no recourse; people at Google browse your data; your data gets mined; your data gets lost; Google messes up and other people get access to your data; Google gets a subpoena and those photos that you thought were perfectly normal are now evidence of a manufactured crime.
I do the same with Dropbox. I've been doing this for years and it's saved me a number of times. I don't know if there are flaws with this but it's worked better than anything I've cobbled together myself.
Makes you solely dependent on Google. That's not really a problem, but as we've seen many times on HN and elsewhere, there's always the random chance that Google's automated systems can decide that your account meets some unknown criterion to be locked, deleted, or otherwise preventing you access.
It's rather unlikely, but so is the chance of your house burning down.
That's generally the reason people recommend the 3-2-1 rule. By having multiple separate backup solutions, you're hedging your bets. But as is commonly the case in computing, you have to play the convenience vs security/reliability game here and decide where the line between your time and convenience lies against the reliability of your backups and what risks your willing to take.
Something tricky is how to bootstrap restoration of backup. If you have lost "everything", how do you get it back?
For example, if you use borg to backup remotely via ssh, you will need ssh keys as well as a passphrase for the encrypted backup. Where do you store those to make sure you have them if your computer is gone? What I did was create a self-extracting restoration script, which embedded everything needed. This is also encrypted, and synced to many places. The idea is, as long as I have the passphrase for that, it takes care of the rest.
Check with your bank. If you have enough accounts, or the right kind of accounts, they may give you one for free.
It's usually listed included in the long list of benefits that almost nobody ever reads.
Each year I get a bill from a major American bank in the amount of $0 for a safe deposit box I don't use because I have a certain amount of money in one account.
I keep the most important pieces on printout in a firesafe lockbox. I don’t have any eventuality for if the firesafe goes at the same time as online failsafes, but I feel like the best approach for those kinds of situations is nihilism.
Every good backup practitioner has an origin story. That one time they lost data to a clicking HDD, or to a scratched CD, or to a typo. The pain of never getting that data back. Galvanizing pain that catalyzed a resolution never to lose data again.
Never trust someone (including yourself) to truly look after your data unless they know this pain first hand.
I also have a ZFS NAS and a separate backup server for the NAS that only gets turned on once a week. I disagree about his recommendation to avoid encryption. I keep the NAS and backup server drives encrypted. It’s not a big deal to unlock the drives at boot with a password and I’ve never had any issues with it. If someone breaks in and steals one of the servers, at least they won’t have access to years of all of my documents backed up.
This author’s setup looks good, but I think most people would get burned out by the complexity unless you really like managing servers, automating things with scripts, and navigating the little quirks that pop up with the hardware and software involved.
A complex solution isn’t necessarily better if you find yourself not investing the time needed to maintain it. I think most people would be better off trusting one of the online backup services rather than spending thousands of dollars building and maintaining a pair of backup servers. If you’re worried about losing data, signing up for two separate backup services could still be cheaper than building and running two separate NAS servers.
Start with easy backup solutions first to get something going. Then consider the more complicated options later.
> most people would get burned out by the complexity unless you really like managing servers, automating things with scripts, and navigating the little quirks that pop up with the hardware and software involved.
I didn't see the author's stated OS they use for their NAS, but depending on what it is, yes. {Free, True}NAS are dead easy to use as long as you stay inside the confines of the GUI, and hugely reliable. Anything else is no man's land. Running ZoL and you did a routine kernel update? Enjoy your new read-only ZFS pool!
I have a setup very similar to the author's, with a bunch of Debian VMs on top of Proxmox. Can confirm that even with Packer and Ansible handling base images and configuration, weird edge cases pop up that take quite a while to track down.
I thought the same about Free/True NAS until I lost a ton of stuff. Be careful with these. Also, for big backups the performance has been terrible. I carefully followed their guide and the write speed is just dismal.
> {Free, True}NAS are dead easy to use as long as you stay inside the confines of the GUI, and hugely reliable.
I’ve used FreeNAS (now TrueNAS). Great at first but eventually I started hit snags and bugs. I still use TrueNAS but I wouldn’t say it’s 100% reliable, especially across upgrades.
> Anything else is no man's land. Running ZoL and you did a routine kernel update? Enjoy your new read-only ZFS pool!
FWIW, I have a separate ZoL server that has been bulletproof, even with frequent upgrades. I did, however, run into a kernel issue with TrueNAS after an upgrade that caused some hardware issues about once a week. My options were either to downgrade to an older TrueNAS until they fixed it (still not fixed) or buy new hardware.
I have something simpler than yours, with a small Proxmox server as my NAS (no ZFS) and backups stored to a second disk. Occassionally I will backup the contents of that machine to external drives.
In the future I plan on doing something similar to what you have, with a powerful ZFS server and my current server accessing the main server using a read-only interface as a second backup. It does take work to maintain, and I don't look forward to the day when I have a full server failure and need to scramble to replace the unit, restore from backups, and all that. Ideally I would have a pair of servers to handle that eventuality... and down the rabbit hole we go.
Ultimately, I guess this all is a tradeoff - I know many engineers who swear on their cloud services, and they simply don't have the time and experience to maintain dedicated servers. While Synology devices and FreeNAS make things simpler, that's for a best-case scenario as you need the technical knowledge to deal with the issues that will eventually pop up. Honestly the cloud is what I recommend for all but the most technical folk, with the addition of external disks off-site for the most critical files.
>Not only does encryption during data recovery make everything much more difficult, but should you pass away, your family members might not have the skills required to access the data
For your browser history, sure. But I for one make sure I rsync stuff to a Raspberry Pi4 with an external disk, at my parents place every now and then. In the event of my demise they can unplug it and have all our family pictures and videos. I'm really afraid that I make things to complicated for the people that I leave behind.
I dabble in "life storage" and your comment made me think that some sort of executable shipped alongside backup locations to read the data, if in some deduplicated backup form, seems valuable.
Eg Camlistore/Perkeep had the premise of using JSON to store data. However some random person isn't going to write code to parse all that data, pull files out, etc. A lifeboat .exe might be interesting.
Though doing it in the most simple, least configurable, least breakable way seems.. necessary. Yea some baked in UI would be cool, but more moving parts means less likely to work.
I have the same setup for the same reason as you. Only difference, I have my setup at home. As you say, in the event of my demise, I want my relatives to be able to grab the drive and have easy access to the pictures and videos.
My approach is 6 years old and I use Time Capsule. The best ever solution for me. You set it up and you don't case anymore. It's Apple specific unfortunately. Don't know if there is a similar solution for Windows.
I currently have a Synology RAID6 and I use a cloud backup service. But I’ve been thinking about adding another layer in the form of off-site tapes. Any recommendations welcome.
I have tapes as my last line of defence (ZFS snapshots, backups to NAS, online are my earlier lines of defence).
The hardware itself is relatively simple but (at least in my experience - I tried quite a few options but they were all missing at least one key feature) there's no good "set and forget" software solution out there in the same way as there is for online backups so you're going to spend a long time scripting it all and tweaking it properly. You can build a solution on top of whichever other tools fit your needs that way. I have a set of scripts which create my backups in single large files on disk ready to be spooled out, then I use mbuffer to actually put the data onto tape.
In terms of hardware, 1 or 2 generation old LTO drives are often available for a decent price in good condition on eBay. You'll also need either an SAS or Fibre Channel interface depending on what drive you buy. As far as I can tell they're pretty much all supported by Linux these days.
Given the complexity of scripting and potential for user errors, testing that you can restore properly is pretty well essential, of course.
I've been using btrfs and btrbk for a few months now and really enjoy it. I love that snapshots are easy to make, and btrbk is a fairly simple way to schedule taking snapshots and transferring them to another machine, as well as set up a retention strategy. I agree with the author that simpler is better -- if I lose data/delete a file I want back, it's as simple as mounting the snapshot and grabbing what I need.
I took the deep dive into tech trying to fix the 2x cd drive someone gave me for my x86 pc. Part of the passion I have for tech is learning and digging deeper, and reading about topics I haven't learned yet to the point where I get a bit of malaise when I'm not learning something new. I say this as I'm repurposing an old pc to a full time nas and learning about zfs pools. There's always a simpler, faster, more expensive solution out there that takes the complexity out of the equation, but part of the fun is the complexity for me.
For the past few years, I've created a photo book of the best family pictures from the past year. I store these photobooks in a fireproof safe. I think that's the easiest way to pass down pictures since it's paper doesn't require decryption or passwords.
This summer I forgot the PIN of my iPad, because I didn’t use it during all summer. And it’s only 6 digits. I had, however, made note of the PIN elsewhere so I was able to unlock it. I have similarly forgotten PINs and passwords in the past that I didn’t use for a while. I recommend unlocking your safe at a regular interval, so that you remember the combination, if it uses a code lock. And even if it’s a key lock it’s probably a good idea to do it too, so you know where the key is.
Thumb drives.
Financial stuff, copies of legal documents, large libraries of PDF books, reports, etc., etc.
And then back the main drive up to FOUR thumb drive backups.
Reasons:
(1) USB drives are huge now. (2) Convenient - can move from one PC to another easily. (3) Convenient - I do NOT have time to mess with other solutions. (4) Safety - valuable financial stuff almost always OFFLINE reducing hacker vulnerability. (5) All PCs eventually fail - I'm old and have a collection in my basement - plus I have two fairly new ones awaiting repair in my home office right now.
Only disadvantage - I do have multiple legal copies of Microsoft Office on multiple PC to make moving thumb drives around practical.
That is my understanding. I've heard numbers around 10yr +-5, but I've never seen a study on it. I wonder if storing them zip-locked in a freezer would help (but I guess you could just use a magnetic hardrive :D )
That the number I've personally experienced (I also replied to GP).
It's an interesting idea to deep-freeze your drives, but the thermal stress from modern frost-free freezers that automatically dethaw periodically may negate the benefit of chilly bits...
If you don't have that much to store, and are worried about data durability, I'd suggest looking into M-disc instead.
Yes they do. I used pen drives for backups after I found one that was like 10 years old and still had my old cv on it. I stored my backups on two identical pen drives I stopped doing this when I went to backup and discovered one of the pen drives was corrupt and lost all its data.
They are super cheap and fairly reliable but not quite reliable enough for backups.
I'll add - drives from different brands. I had a mirrored RAID 1 fail at one point (in the '00s, spinning magnetic hard drives), two identical drives, and they failed within days of each other. One drive died, I ordered a replacement, and before the replacement arrived the second drive died. The shock and disbelief I felt of my foolproof RAID solution going belly-up on me! And I still can't believe I didn't turn off the machine while I waited for the replacement to arrive.
Make sure you periodically refresh your backups that are stored on flash memory devices. Depending on the media, cell voltage drift can cause data corruption that overwhelms the built-in ECC in as little as 5-7 years.
"While I know of some really good cloud providers, such as rsync.net and Tarsnap, I recommend that you never trust cloud providers blindly."
This is very good advice.
That being said, humans need heuristics and shortcuts to aid in decision-making. I hope the fact that rsync.net has been doing this work since 2001 is helpful in that regard.
...
"There exist some really cool open source backup solutions such as Borg, Restic and duplicity, but you should never rely solely on these "complex" solutions. These tools work really great, until they don't! In the past I have lost data to duplicity and other tools."
I think this is very good advice as well - and that is coming from someone who has whole-heartedly endorsed 'borg' as a backup tool and regularly recommends it. It is the "holy grail of backups"[1] after all ...
It's also a magic black box for most users and would be difficult to work out failures.
The safest and (in my opinion) most useful workflow is to back up your data locally to some kind of NAS or fileserver using plain old rsync and then back up that fileserver to rsync.net (or whomever) using the fancy borg tool.
Now you have quick and simple local restores but still have a backup in the cloud that requires zero trust in the provider.
> That being said, humans need heuristics and shortcuts to aid in decision-making. I hope the fact that rsync.net has been doing this work since 2001 is helpful in that regard.
This is implied by the 'blindly' part. Searching "cloud storage provider", seeing rsync.net listed and picking it with a thrown dart would be blind. A quick search to see that it's been around for a while and doesn't have any crazy horror stories attached is part of becoming informed.
In case you didn't notice or realize or recognize, GP is the founder of rsync.net (I'm not saying this to assign any ulterior motive for that comment).
I feel like I have a fairly solid backup system for all of my important files. They are protected against elementary disasters and cyber attacks alike.
However the only thing I struggle with are my phone photos. I recently caved in and started using iCloud Photos due to the convenience of having all my phone photos alwas available, searchable and tagged, even if the library size exceeds my phone‘s capacity.
Does anyone know a reliable and automatable way to back up this iCloud photo library on a self hosted server?
I'm also interested in this. I have iCloud for Windows installed on a VM with lots of attached storage, but it doesn't seem to persistently/reliably download photos unattended.
I suppose another way to do it would be with a Mac and then periodically backing up the local Photo Library, but that still leaves the photos tied up in Apple's proprietary library format. Plus you need a spare Mac just laying around and always on.
I do this with Syncthing (not just photos, but all phone files), a very customizeable file sync tool, it can do trash bin and versioning similar to cloud storage providers. As I heard sadly it doesn't work so well in iOS due to its restrictions though.
I spent extensive time evaluating this problem, and tried a whole bunch of different things:
Nextcloud, Resilio, iCloud, etc etc etc.
Honestly? Just using Google Photos is probably the best bet for most folks, whether you’re on iOS or something else. I personally picked OneDrive for my use case since it is both more performant for large data sync and 6 TB + Office 365 for a family plan beat out the alternatives on pricing.
Eh... I'd definitely do a Google Takeout to verify everything's OK before you delete your originals.
You may think the downsampled image is sufficient for lossy backups, but older images stored in "high quality" in my personal GP library were almost wholly stripped of metadata, including when the photo was taken.
(Many of my users have suffered from this as well, which is why I built tag value inference into PhotoStructure to try to help spackle over these metadata holes).
That’s good advice for those that care, and I should have mentioned it, especially for HN crowd. I definitely noticed this problem when I did Google Takeout on my photos dating back to the Picasa days.
I still think that for most people (and on HN, where we care more about fighting data entropy, there might be significantly lower overlap with the majority), Google Photos is the best option.
> These tools work really great, until they don't!
Huh? Whats the problem? How is ZFS any different because its lower level?
> ZFS without ECC memory is no worse than any other file system without ECC memory.
You really do need ECC memory unless you are ok with your pool becoming corrupt every 6 months or so. Im just saying this from my experience of running a ZFS server without ECC, the data wasn't critical so I left it like that for 3 years before replacing the memory with ECC now its fixed.
I was so tempted to jump on this with my anecdata of successfully running ZFS without ECC for a time, but that was in the context of a dirt cheap home tinker box.
If you care about data integrity enough to use ZFS, just bite the bullet and use ECC. If you care about data integrity at all, use ECC. If you don’t, what are we even doing here?
The bit about ZFS was a really flawed argument, saying it's just as complex but in an unavoidable way so that's fine for some reason:
> One could argue that ZFS is complex as well, but that is on the filesystem level, a level on which you cannot avoid complexity no matter what you do.
I stored 2tb of actively used data on zfs without ecc ram from freebsd 8 to 12. I had no noticeable corruption. I only post this because there is a weird assumption that not using ecc ram is a death sentence for your zfs data.
I use ecc ram now and I think it's the proper way to do zfs, but let's not pretend you forfeit your data by not using ecc.
I am happy that worked for you but there is a huge difference between a 24 x 4tb pool and a single 2tb drive.
2tb is not a lot of data you would need to fill the drive 4 times a day before you would get close to hitting the single bit error rate, a pool of my size is guaranteed to hit that error rate just because of the volume of data.
If we are talking about running zfs on a desktop then yea you are right you don't need ECC, but if you are running a storage server for data you want to keep then you absolutely do.
ZFS doesn't need ECC but it does benefit from it. If your pool was becoming corrupt every six months, look to a shitty drive controller or cabling first.
I've run ZFS for quite a few years now on laptops and lower-end machines that can't use ECC memory and I've never had corruption, unrecoverable files/pools, etc.
143 comments
[ 2.5 ms ] story [ 175 ms ] thread1. Hot live data on my laptop.
2. Archive dir on NAS.
It’s either 1 or 2, no overlap.
Then:
1. Backup whole laptop to USB bootable drive “when I fancy”.
2. Backup home dir to NAS using duplicacy. This is accessible via SFTP, so it works no matter where I am.
3. Backup the Archive dir of the NAS to cheap cloud storage.
Ideally I should add an external drive for the NAS to backup #3 locally, too.
The method is simple but strong.
The most important part is that it is actually implementable, examples and reasoning are provided for all steps.
Otherwise it's still a SPOF for your backups (the borg files might corrupt, and you might be syncing that corruption to the cloud)
1. All my live, working data is in Dropbox. I've been using my (paid) Dropbox account for over a decade. This step isn't really intended to be a backup per se, but you get the backup behavior "for free" because this is how I keep my secondary machine in sync with my primary. My Dropbox folder is also replicated to a 3rd computer in my home as another level of redundancy.
2. Everything in my home dir, especially photography, is ALSO covered by Backblaze. Having some backups elsewhere is mandatory; not enough people really understand how important this is. Should my house burn down, I still have data.
3. My primary system is a Mac, so I use Time Machine. TM is the only backup on this list I've ever actually used as a backup. When our home was robbed a number of years ago in a quickie smash-window-and-grab affair, they got my laptop. I went out and bought a new computer, plugged it into the TM drive, and in an hour or so I was right back where I left off. Hard to beat that. Even my app windows were in the same place.
4. Periodically, I take a full clone of my main machine's drive using a drive imaging tool (Mac specific; I use SuperDuper) and **store that drive at a friend's house**. This probably only happens a few times a year a this point. I should do it more often.
That tertiary computer I mentioned in step 1 is also the home NAS server / home media server. It holds the photo archive in a large outboard disk. Backblaze covers that disk, and Time Machine on that computer keeps the outboard disk backed up as well. This data is mostly static, so the images I've taken of it and stored elsewhere don't need to be updated all that often (ie, just when I migrate prior year photo data to that drive).
However, I'm still somewhat confident in my strategy as I backup my all of my data to two entirely different repositories, one of them backed by Google Cloud, and another by the server sitting in my pantry. So one of these repositories could get irrecoverably corrupted and I still wouldn't lose any of my data. With cloud storage becoming so cheap I've also thought about adding a third repo.
Of course this would not protect me from a hypothetical bug in Restic that corrupts all my repositories before I notice, so maybe I should also add another auto-backup solution into the mix.
Doing manual things like moving data to external storage seems like a robust strategy, but I really don't trust myself to do something like that nearly often enough for it to be useful.
I had a client that asked me to setup their system. I setup the system, they got a tape drive and I had them rotate tapes daily. There was a cronjob to tar everything to tapes.
It was great until their hard drive failed and I found I had a typo where it was only backing up the current folder, not the whole drive. Needless to say, they found a new IT provider and I learned an important lesson. If you haven't tested your backups, you have no backups.
My important stuff is in the cloud.
I have a nightly "hot backup," that is a CC clone of my main and dev drives. It used to be every 4 hours, but I found that to be overkill.
I almost never need anything more than the "hot backup."
I wonder how OP does snapshot rotation.
https://johnnydecimal.com/
Down the rabbit hole I go...
Me and my partner have multiple devices (tablet, iPhone, Android phone, multiple Mac and Windows laptops) and we just sync folders to our desktop. We just store everything we don't want to lose in Drive. We share folders we both need. Photo's we take on our phones are automagically backed up in the cloud, music and movies are streamed.
My house can burn down overnight and we won't lose any valuable data.
It's rather unlikely, but so is the chance of your house burning down.
That's generally the reason people recommend the 3-2-1 rule. By having multiple separate backup solutions, you're hedging your bets. But as is commonly the case in computing, you have to play the convenience vs security/reliability game here and decide where the line between your time and convenience lies against the reliability of your backups and what risks your willing to take.
For example, if you use borg to backup remotely via ssh, you will need ssh keys as well as a passphrase for the encrypted backup. Where do you store those to make sure you have them if your computer is gone? What I did was create a self-extracting restoration script, which embedded everything needed. This is also encrypted, and synced to many places. The idea is, as long as I have the passphrase for that, it takes care of the rest.
It's usually listed included in the long list of benefits that almost nobody ever reads.
Each year I get a bill from a major American bank in the amount of $0 for a safe deposit box I don't use because I have a certain amount of money in one account.
Have't needed to use them yet, but testing it was great.
Never trust someone (including yourself) to truly look after your data unless they know this pain first hand.
This author’s setup looks good, but I think most people would get burned out by the complexity unless you really like managing servers, automating things with scripts, and navigating the little quirks that pop up with the hardware and software involved.
A complex solution isn’t necessarily better if you find yourself not investing the time needed to maintain it. I think most people would be better off trusting one of the online backup services rather than spending thousands of dollars building and maintaining a pair of backup servers. If you’re worried about losing data, signing up for two separate backup services could still be cheaper than building and running two separate NAS servers.
Start with easy backup solutions first to get something going. Then consider the more complicated options later.
I didn't see the author's stated OS they use for their NAS, but depending on what it is, yes. {Free, True}NAS are dead easy to use as long as you stay inside the confines of the GUI, and hugely reliable. Anything else is no man's land. Running ZoL and you did a routine kernel update? Enjoy your new read-only ZFS pool!
I have a setup very similar to the author's, with a bunch of Debian VMs on top of Proxmox. Can confirm that even with Packer and Ansible handling base images and configuration, weird edge cases pop up that take quite a while to track down.
I’ve used FreeNAS (now TrueNAS). Great at first but eventually I started hit snags and bugs. I still use TrueNAS but I wouldn’t say it’s 100% reliable, especially across upgrades.
> Anything else is no man's land. Running ZoL and you did a routine kernel update? Enjoy your new read-only ZFS pool!
FWIW, I have a separate ZoL server that has been bulletproof, even with frequent upgrades. I did, however, run into a kernel issue with TrueNAS after an upgrade that caused some hardware issues about once a week. My options were either to downgrade to an older TrueNAS until they fixed it (still not fixed) or buy new hardware.
In the future I plan on doing something similar to what you have, with a powerful ZFS server and my current server accessing the main server using a read-only interface as a second backup. It does take work to maintain, and I don't look forward to the day when I have a full server failure and need to scramble to replace the unit, restore from backups, and all that. Ideally I would have a pair of servers to handle that eventuality... and down the rabbit hole we go.
Ultimately, I guess this all is a tradeoff - I know many engineers who swear on their cloud services, and they simply don't have the time and experience to maintain dedicated servers. While Synology devices and FreeNAS make things simpler, that's for a best-case scenario as you need the technical knowledge to deal with the issues that will eventually pop up. Honestly the cloud is what I recommend for all but the most technical folk, with the addition of external disks off-site for the most critical files.
Uh, trust me, that's a good thing.
Eg Camlistore/Perkeep had the premise of using JSON to store data. However some random person isn't going to write code to parse all that data, pull files out, etc. A lifeboat .exe might be interesting.
Though doing it in the most simple, least configurable, least breakable way seems.. necessary. Yea some baked in UI would be cool, but more moving parts means less likely to work.
I currently have a Synology RAID6 and I use a cloud backup service. But I’ve been thinking about adding another layer in the form of off-site tapes. Any recommendations welcome.
The hardware itself is relatively simple but (at least in my experience - I tried quite a few options but they were all missing at least one key feature) there's no good "set and forget" software solution out there in the same way as there is for online backups so you're going to spend a long time scripting it all and tweaking it properly. You can build a solution on top of whichever other tools fit your needs that way. I have a set of scripts which create my backups in single large files on disk ready to be spooled out, then I use mbuffer to actually put the data onto tape.
In terms of hardware, 1 or 2 generation old LTO drives are often available for a decent price in good condition on eBay. You'll also need either an SAS or Fibre Channel interface depending on what drive you buy. As far as I can tell they're pretty much all supported by Linux these days.
Given the complexity of scripting and potential for user errors, testing that you can restore properly is pretty well essential, of course.
Seems like a lot of work to ultimately not do things not as well as army of engineers focused on this problem full time.
It's an interesting idea to deep-freeze your drives, but the thermal stress from modern frost-free freezers that automatically dethaw periodically may negate the benefit of chilly bits...
If you don't have that much to store, and are worried about data durability, I'd suggest looking into M-disc instead.
They are super cheap and fairly reliable but not quite reliable enough for backups.
This is very good advice.
That being said, humans need heuristics and shortcuts to aid in decision-making. I hope the fact that rsync.net has been doing this work since 2001 is helpful in that regard.
...
"There exist some really cool open source backup solutions such as Borg, Restic and duplicity, but you should never rely solely on these "complex" solutions. These tools work really great, until they don't! In the past I have lost data to duplicity and other tools."
I think this is very good advice as well - and that is coming from someone who has whole-heartedly endorsed 'borg' as a backup tool and regularly recommends it. It is the "holy grail of backups"[1] after all ...
It's also a magic black box for most users and would be difficult to work out failures.
The safest and (in my opinion) most useful workflow is to back up your data locally to some kind of NAS or fileserver using plain old rsync and then back up that fileserver to rsync.net (or whomever) using the fancy borg tool.
Now you have quick and simple local restores but still have a backup in the cloud that requires zero trust in the provider.
[1] https://www.stavros.io/posts/holy-grail-backups/
This is implied by the 'blindly' part. Searching "cloud storage provider", seeing rsync.net listed and picking it with a thrown dart would be blind. A quick search to see that it's been around for a while and doesn't have any crazy horror stories attached is part of becoming informed.
However the only thing I struggle with are my phone photos. I recently caved in and started using iCloud Photos due to the convenience of having all my phone photos alwas available, searchable and tagged, even if the library size exceeds my phone‘s capacity.
Does anyone know a reliable and automatable way to back up this iCloud photo library on a self hosted server?
I've used this to great success! As far as I could tell, it grabbed every photo and video at the highest quality. Very happy.
I suppose another way to do it would be with a Mac and then periodically backing up the local Photo Library, but that still leaves the photos tied up in Apple's proprietary library format. Plus you need a spare Mac just laying around and always on.
Nextcloud, Resilio, iCloud, etc etc etc.
Honestly? Just using Google Photos is probably the best bet for most folks, whether you’re on iOS or something else. I personally picked OneDrive for my use case since it is both more performant for large data sync and 6 TB + Office 365 for a family plan beat out the alternatives on pricing.
You may think the downsampled image is sufficient for lossy backups, but older images stored in "high quality" in my personal GP library were almost wholly stripped of metadata, including when the photo was taken.
(Many of my users have suffered from this as well, which is why I built tag value inference into PhotoStructure to try to help spackle over these metadata holes).
I still think that for most people (and on HN, where we care more about fighting data entropy, there might be significantly lower overlap with the majority), Google Photos is the best option.
Android doesn't know it, but their killer app is Syncthing.
Huh? Whats the problem? How is ZFS any different because its lower level?
> ZFS without ECC memory is no worse than any other file system without ECC memory.
You really do need ECC memory unless you are ok with your pool becoming corrupt every 6 months or so. Im just saying this from my experience of running a ZFS server without ECC, the data wasn't critical so I left it like that for 3 years before replacing the memory with ECC now its fixed.
If you care about data integrity enough to use ZFS, just bite the bullet and use ECC. If you care about data integrity at all, use ECC. If you don’t, what are we even doing here?
> One could argue that ZFS is complex as well, but that is on the filesystem level, a level on which you cannot avoid complexity no matter what you do.
I use ecc ram now and I think it's the proper way to do zfs, but let's not pretend you forfeit your data by not using ecc.
But from all filesystems ZFS works best of all when you do not have ECC RAM.
The 'requirement' that ZFS needs ECC RAM to work properly is one of the biggest myths of the Internet ...
2tb is not a lot of data you would need to fill the drive 4 times a day before you would get close to hitting the single bit error rate, a pool of my size is guaranteed to hit that error rate just because of the volume of data.
If we are talking about running zfs on a desktop then yea you are right you don't need ECC, but if you are running a storage server for data you want to keep then you absolutely do.
I've run ZFS for quite a few years now on laptops and lower-end machines that can't use ECC memory and I've never had corruption, unrecoverable files/pools, etc.
https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...