Ask HN: What's your backup setup?

127 points by iansowinski ↗ HN
I'm rethinking my backup workflow and I'm curious about other people setups - both hardware and software. How does your backup setup looks like guys and girls?

160 comments

[ 2.8 ms ] story [ 244 ms ] thread
Currently just use time machine to backup my iMac
same and have an encrypted usb drive on my keyring I copy everything to once a week (my offsite backup :-) )
I use a Synology NAS box
I used to have a subscription to crashplan but that wasn't flexible (or cheap) enough when you try to backup multiple machines/phones.

Now I have a raspberry pi with an encrypted USB drive attached where I sync all files from laptops/desktops/phones/truecrypt-drives (I have an instance of pydio-cloud running too).

Then, once a week (or once a day depending on the folder) I sync everything to rsync.net.

I rely mostly on Borg backup¹.

1. First, I run it locally on my desktop against a repository I keep on the same drive (/home/backup); then

2. I update, with rsync, a copy of this repository I keep on a dedicated server with full disk encryption; and, finally,

3. I spin up an EC2 instance, mount an S3 bucket with s3ql, and rsync the copy from the previous step up to the one I keep on this bucket.

This process is (embarrassingly) manual.

The backup repository is encrypted with Borg itself, and if I am in need of recovering something I do it from the local copy. I never mount the repository in remote locations.

¹https://github.com/borgbackup/borg

I'm also using borg, to a server I control and also to rsync.net.

Essential and small files like keys (which may be necessary to "bootstrap" the backup in case of complete failure of my workstation) are copied and verified manually to offline storage.

I also want to have a Borg setup[1] on my personal VPS and backup the "most essential" data (which will be dome docs and some photos - not all) to it along with everything being backed up by CrashPlan. Will explore it someday.

[1] Are there comparable tools which are open source and easier to use (preferably with a GUI or so)?

I basicly do the same, except I sync to google drive using rclone. I also sync to a USB drive about once a week. It basically is a backup of my home dir. the rest of my data already lives in google drive (pictures etc)
Why not "aws s3 sync" for #3?

edit: I see s3ql has many features that can be desirable like compression. How reliable is it?

My poison is flexbackup

    [I] app-backup/flexbackup
         Available versions:  1.2.1-r12 ~1.2.1-r13
         Installed versions:  1.2.1-r12(05:37:21 PM 03/03/2014)
         Homepage:            http://flexbackup.sourceforge.net/
         Description:         Flexible backup script using perl
Pretty old, but some software is like that, able to be finished.

I run a couple cronjobs on it, doing full backups every Sunday and differentials against the full backup every other day of the week. The backup target is a RAID1 backed disk on a NAS.

Flexbackup produces tarballs essentially, with indexes and the usual add/remove tracking. Compression can be naturally applied. It all relies on the common Unix tools. Just yesterday I updated my 4-year-old configuration to try out new partitions and incremental backups; A minimal example config for flexbackup:

    $set{'pictures'} = "/stor/amy/pictures/";
    $compress = 'gzip';
    $compr_level = '3'; #1-9
    $device = '/srv/harry/pictures-backup/';
Associated crontab:

    30 4  1,15  *  *    /root/backup-scripts/backup-pictures-full.sh
    30 4  2-14  *  *    /root/backup-scripts/backup-pictures-incr.sh
    30 4  16-31 *  *    /root/backup-scripts/backup-pictures-incr.sh
With the scripts essentially saying,

    /usr/bin/flexbackup -set pictures -full \
      -c /root/flex-confs/pictures.conf >> /root/backup.log 2>&1
and

   /usr/bin/flexbackup -set pictures -incremental \
     -c /root/flex-confs/pictures.conf >> /root/backup.log 2>&1 
respectively. (They also contain some find(1) invocations to remove older full backups and obsolete incrementals.)
> Pretty old

> 2014

When someone says "pretty old", I expect some Perl script from the 90s.

Sorry, the date's when I last installed or updated the package.

The version 1.2.1 is from 2003.

Ah, that's more like it. I too appreciate software that's well-hung. :)
Not sure if you're talking about data or actual dev workflow but I will share my setup with you:

In terms of Data, everything I own is backed up in Google Drive. (Photos and Documents mostly, I don't take tons of pictures and ALL the music I listen to is on soundcloud)

In terms of dev workflow, it's pretty interesting. My macbook air died on me last week, and because I can't afford to get another one (or even a decent pc for that matter) I've fallen back to my raspberryPi. The browser is a little bit slow sometimes, but I have to say that I'm quite impressed by how well it performs.

Because it's a bit limited in terms of hardware capabilities, I've bought 2 VPSs from scaleway which I've provisioned using an Ansible playbook I wrote.

I was up and running and ready to work within minutes.

Now it's a bit inconvenient because I'm used to being mobile and taking my laptop with me everywhere, but it's a perfect backup solution for now. Obviously I don't watch netflix on it of play video games, but for 35 quid you can't really expect much.

Edit: the playbook I mentioned can be found here if you want to take a look: https://github.com/Zabanaa/night-city

I love Arq and I've described my setup in a blog recently https://hello.rakh.im/backup/
Interesting how you broadcast on your blog that "it’s extremely unlikely that you’ll ever need a backup". Have you ever had to look a parent in the eye and say, "I'm sorry all your photos of your kids growing up are gone" ?

Backup now, backup always, backup often. You can't buy that stuff back.

my main stuff is backed up both on my google drive and my ibm server on a raid 5 array. cant imagine ill need anything more than that really
I have my "home" as a master Syncthing folder, that I sync with a RPI3: https://syncthing.net/ I have it set up to keep a few revisions of every file.

Syncthing is not really meant for backup, but I really like that it just happens over the network, in the background, without further intervention. I am clumsy, lazy and not disciplined enough for other backup setups that require action (e.g. connecting the computer to something sometimes, manually triggering backups, etc...)

For my personal devices, I also use Syncthing.. but I do not keep revisions. Instead, I have push only from Laptop (etc) and on a server running ZFS I take automatic hourly, daily, weekly and monthly snapshots which are auto-purged set on my retention strategy. It is much more efficient on disk space.

Occasionally, I will run rsync across the device by hand just to check consistency - but so far it has been reliable.

Backblaze for all 6 Macs in my family. SuperDuper! imaging nightly for my dev MBP to external drive.

Not sure I really need Backblaze when backing up nightly.

I use Backblaze also, for my virtual machine backups as it would be time-consuming to recreate them (I have several for various clients, server set-ups, etc). The seriously important stuff such as source code is in Bitbucket, and I also have a nightly AWS batch job that backs them up to S3.

I haven't seen anyone mention password managers yet. I use lastpass day-to-day but for my recovery codes I have a Keepass vault.

I have backups of that dotted around but I have also asked a couple of my friends to keep their own copies of this vault in the event I somehow lose all my computers and phone. I think that resolves the chicken-and-egg problem quite nicely.

I want total control, so a Synology NAS box setup with two disks in a mirror, 1 SSD as cache, and one hot failover. All laptops backup to it. It backs up to amazon s3 and to a second synology NAS.
I separate everything in years. Current year gets synced every week via rsync with a drobo (drobo duplicates the data amongst all the drives). I also have a disc in another location that gets synced once a year at christmas with the archive. It was a bit of an investment but then it's pretty cheap to run.

I know it's not perfect. If I delete something without noticing and sync it afterwards it will be lost forever, but I'm running this for the last 10 years and never really had a problem.

To each of you guys having those extensive backup solutions (like NAS + cloud sync, second nas, etc)...

.. do you actually TEST those backups?

This questions comes from my experience as a system engineeer who found a critical bug in our MySQL backup solution that prevented them from restoring (inconsistent filesystem). Also, a friend of mine learned the hard way that his Backblaze backup was unrestorable.

Very true. I overheard a similar conversation last week at work: "We have set up the backup procedure for our new production databases." - "Have you tested restore?" - "Well, uhm..." - sound of JIRA ticket being opened

By the way, I misread your username and, for a second, thought you were sytse.

it's not a backup till you test it - just complicated wishful thinking.
As a colleague of mine says: You don't want a backup, you want a restore.
I have restored files from BB multiple times. It is a great solution for non technical ppl or offices that have at least 24Mbps connections up/down.
I'm using CrashPlan and I have recovered multiple files over past couple of years, that I either mistakenly deleted or overwritten. I haven't tried any full-scale restore, yet, though.
CrashPlan lost some data of mine in 2013 from querying a corrupted Volume Shadow Copy Services database on Windows. (At least, that was their explanation. I'm surprised that their client did not independently verify the data after it was uploaded.)

I moved off CrashPlan in 2016 because their upload speed continues to be embarrassingly slow outside the US even with deduplication and compression turned off (they have a datacentre where I'm at, but it's for Enterprise customers only).

They also highly recommend having 1GB of RAM for every 1TB backed up, which sounded a bit unreasonable to me.

What did you switch to? Problem with BackBlaze and others is that they delete backups if not connected for 30 days, particularly external drives.
I moved to Acronis True Image when they offered unlimited cloud backups with their 2016 version. They probably couldn't sustain it, because I had to pay a lot more for backups when I wanted to renew in 2017.

Now, I'm using both Arq and Synology's Hyper Backup with Amazon Cloud Drive. One of the problems I foresee is that while Amazon doesn't care how much data one stores in Cloud Drive, they have suspended users for downloading past an arbitrary limit in a certain period of time — so full restores might not be possible.

".. do you actually TEST those backups?"

yes (of course), see my post above.

I have used time machine repeatedly to restore lost or damaged files. I also replaced harddrives several times and played back my carbon copy clone. It boots and I have never missed a file in years.

Excellent point, and that begs another question: how do you actually test your backups? Of course, each case is specific, but is there a "best practice" checklist, or some general points to check for basic restoration?
I've been thinking about this as well - it seems like it would fit in nicely with other CI jobs. With database backups, for example, you should be able to script the restore procedure and apply some assertions to check it worked. Bonus with this is that you now have a script when you actually need to restore.
Excellent question. I do test my backups and restores on a rather constant basis. Each environment within my infrastructure takes a bit of a different approach.

Application

This is by far the easiest for me to test. We have a CI/CD jon which literally makes a new environment, from scratch, and deploys our application to it in a production configuration. It runs a test suite which tests functionality across the application. Finally, it destroys the environment. It reports on each portion of the process. In this way we know exactly how long it would take to redeploy the entire application from scratch on a new infrastructure and get it up and running. This morning it took about about 6 minutes total before tests ran.

Database

We are running an RDBMS. We use a combination of daily full backup, incremental transaction log like backup, and point in time backup. Again, in our CI/CD when a full backup is taken it is pulled, loaded, and a test routine is run against it to check integrity. At this time, the recovery from the day before is destroyed. When a transaction log backup is made, CI/CD picks up this change and applies it to the full backup restore and runs a set of tests for integrity check. This leaves us with a warm standby ready to be switched over to in case of the main database server going down. We have never had to use the warm standby in an emergency but we have a test to make sure we can cut that over as well.

For point in time backup testing this goes back to our application test above. The application test will spin up with a point in time recovery of the database backup. It will test the integrity of that recovery and then test the application against it. Finally, it will swap from the point in time recovered database to the warm backup. It runs the test suite against that for integrity as well.

File Store

People often forget this but those buckets that get hold all of your file storage in the cloud can be destroyed so easily (sad, sad experience taught me this). We test those as well. I am sure you can guess at this point how we do that? CI/CD. It's a rather simple process with a ton of gain.

A few notes

People always ask me this, so I will answer it first. Yes this costs money. It's not as bad as running a second production environment. But it will cost you a bit. My follow up question is, how much does downtime cost you?

My CI/CD is always Gitlab CI at this point. I've used Jenkins. I've used Travis. I like Gitlab CI. You can do all of this with any of those.

We script literally everything. Computers are so good at repetitive tasks. Why would you EVER do anything manually? Really. If it has to do with your infrastructure, script it.

If anyone has any questions about these ideas, feel free to reach out.

How many (full time) devs and how long did that set up take?
We currently have 4 full time devs, a QA, a DBA consultant, and a Designer on the team.

Honestly, none of that took very long to set up at all. The application in this case is a Ruby on Rails backend, PostgreSQL database, Angular front end, with file storage and a few other smaller services.

Step one: We have a lot of tests and we believe in a good test suite. Are we perfect? Absolutely not. But it is important to be able to "know" the application works. Define what helps us to know it works, and automate tests to do that. Things like "Can you log in?", "Can you select a record of type X, Y, Z, A, B, and C and do those records have the data you would expect in the right places?" You can have a human do this, or you can automate it. Automate it.

Step two: Automate your deployment. The rails application is bundled into a docker container. We use ECS (Elastic Container Service) to maintain our environments. CI/CD first runs tests, second, builds the latest docker container, third, places the docker container into a repo, fourth , deploy out the container to the correct ECS environment, five, profit! This is all automated and works the same every time with checks and balances along the way. Our Angular application runs out of S3 buckets with cloudfront caching. This was a matter of using webpack to compile the angular application down to production deployable artifacts and than a simple bash script to move those artifacts to the S3 bucket. The database is an RDS instance so we get some fun things built in there. Note: All of the AWS setup is also automated with scripts. Create VPC, create autoscaling group, create targets, create rds instances, create s3 buckets, create cloudfront, and delete all of the above (and more, aws is complex), are all just scripts.

Step three: Because we have a test suite and deployment scripts the rest of the process is easy. Just use the scripts to create whatever environment we need, stick it on a schedule, record the results in CI/CD, alert the WHOLE FREAKING WORLD if something doesn't work.

Now you might say, easy to say in a Rails environment, with so few moving parts, with such a new project, etc etc etc (I have heard every excuse in the book). I have done this for many other companies. The last I did it at had about 50 engineers, ran a large Java mixed bag of applications on Tomcat servers, ran Oracle for their data, and had no tests and a ton of legacy code. We got to the same point as I have already explained by simply breaking it into chunks. First, automate the tests that were done manually. Second, automate the deployment steps that were done manually. Third, automate the environment things that were done manually. Finally, schedule everything and monitor.

I learned to do this at HP Labs where we used the same process with a very large API fronting a C based image processing system with Petabytes of storage, thousands of servers, and a huge number of moving parts. I promise, it can work anywhere.

I'm a big fan of setting up testing and dev environments from the production backups.

For personal backup of files, I just verify the results are in place. I've checked them once or twice, but honestly, I'm more concerned about my scripts stopping running than they running and not being correct.

That is fine, providing you don't operate in a confidential or regulated environment. :/
what happened to the backblaze backup?
Interesting question. I wonder if it was prior to when BB moved to the "direct wire" architecture in Storage Pod 4.0.
I had a company that I was doing some work for come to me to ask for a copy of the database. Their backs were corrupt, and it was not until they tried to restore it did they find out. But they have 5 years of bad backups
This is one thing i like about doing content addressed storage. I've been toying with my own implementation, quite similar to Camlistore.

The net result is it's super simple to verify an entire datastore as being valid or not.

I use rsnapshot to aggregate a bunch of machines on my NAS.

I've been intending to for a few months O:-) to then save this aggregated backup somewhere on the internet. Not sure if e.g. tarsnap, or a minimal vserver with rsnapshot or rsync yet again.

For the past few years, I have been using a mix of rsync against an in-house and an external server + encrypted USB drives[0]. The key to encrypt the external drives is using a simple algorithm based on the serial number of the drive and a very long string stored in a Yubikey.

I never reuse the drives, just accumulate them.

[0]: https://www.ceondo.com/ecte/2016/09/simple-secure-backup/

Last year I tossed my Western Digital external hard drive in the trash. Who needs it when I have multiple clouds (iCloud/Amazon/Google)?
I need a local disk backup as to download my stuff even with my 70mbsp line would take a long, long time.
Backblaze on my MacBook. Also have a time capsule at home. Pretty simple setup.
Fun story. I ran "rm -rf ~" by mistake just the other day. A misconfigured program had created a dir named ~ inside of my home folder and I was a bit quick to type the command. No harm done because I had setup a cron to rsync everything daily as late as last weekend. Upgraded my backup solution to rsnapshot, still looking out for even better solutions. Phew!
I always love seeing answers to this question!

I'm all about Duplicity:

http://duplicity.nongnu.org/

Sometimes I feel like it's a bit complicated, but I've yet to find anything that will do any better.

I used to use duplicity but recently switched over to borg. Not having to do full backups is nice, I can mount any backup at any time and delete intermediate backups in any order.
While a large part of my backup system currently consists of manually mirroring pictures to various solid state media (that occasionally are moved to separate fire zones) - and a good helping of prayer/good luck - the part which is set up uses duplicity driven by backupninja (from Debian repos - upstream is): https://0xacab.org/riseuplabs/backupninja

It was complicated to set up separate signing and encryption keys such that the server sending backups could not decrypt (assuming the symmetric session key wasn't somehow kept around). But once setup the only worry was making sure the backup server didn't run out of space.

I have a large (24TB) RAID6 at home and backup all my files there. It's large so I have room for all my DVDs, BluRays, and developer VMs. I have a smaller (6TB) RAID1 in another state at my parents house for off-site backup of important files. Both are running mdadm and set up to email me with any events. I have a cron job that runs rsync once a week and emails me the result. Both systems are on an UPS. I have tested to make sure they are working as expected. All my systems are running Linux, so I can access with sshfs or sftp using ssh keys.
I have an HP proliant microserver with 16GB of RAM in the office. It has 4x2TB disks in mirrored Vdev ZFS (RAID1) running gentoo.

All my backups get there first. Some of them are stored in the cloud using tarsnap.

I use crown scripts and riff-backup to fetch daily snapshots of my servers (EC2, RPis, etc).

Mac: Carbon Copy Cloner and Time Machine on separate usb disks. I use the system scheduler to wake the machine at night, mount the disks, start both backups, unmount and sleep the Macbook again. Rock solid, runs every night since years. Even swapping the harddrive is a matter of 30 minutes to play back the latest ccc clone.

I have to find a similar backup solution now also for my Linux based Thinkpad. I am looking into Mondo rescue, because it promises to create a bootable image on an external drive (just like Carbon Copy cloner). For me, it still fails, but this is Linux. Needs more time and research.

This is a personal backup of one computer only. I have bad experiences with centralised backup solutions. In every case you need to reinstall the operating system at least before you can access the backup. I also forgot my password once, because access to the backup is not frequently needed and well meaning admins constructed crazy pw rules. So even though I had a backup, it was not accessible any more.

I would suggest always having one of the three disks in a seperate location. Never have the three disks physically close together, not even when swapping them out.

Also, have an offline backup that is not connected to power. Power surge at night and all your data is gone.

Well, yes and no. People who take their backup drives away from the computer tend to have - a completely outdated backup. There is no 100% safety. If power fails while the backup is done, well there is still the original disk. How high is the possibility that it crashes in the same moment?
It talking about a power spike. If lightning strikes you could toast your computer and the connected backup drives. That is why offline storage is important.
ZFS or btrfs in most places.

All devices with snapshotting capabilities, keep hourly, daily, weekly, monthly snapshots.

Once per day, all devices rsync their /home to NAS. (I would use ZFS send/receive, but I want more selective backups.)

NAS also keeps snapshots. A daily snapshot of the most critical data is encrypted and sent off to an offsite server.

Hadn't lost a single byte in years (since shortly before implementing this scheme :P).