I'm currently on Heroku so my code is all in git and Heroku provides free automated backups for Postgres. I'm eventually moving everything over to Digital Ocean though. In which case I'll probably have a bash script that runs on cron job to just rotate backups. Something like https://wiki.postgresql.org/wiki/Automated_Backup_on_Linux
I don't have massive amounts of important data to manage though so a simple solution like this is all I need.
on servers i run a couple of homebrew shell scripts to do backups. on some i use the backup-gem (https://github.com/backup/backup)
on my workstation/notebook (both apple computers) i use time machine with different external harddrives i rotate weekly.
all setups run automated, but sometimes i trigger them manually (usually right before and/or after some major change in configuration, or when lots of new data are on the drives (eg: i copy all the holliday photos onto my laptop)
i run regular checks on server backups, e.g. check if they "are there" and if i could restore them properly if needed.
If your backups are automated, you need an automated backup checker in order to ensure your backups are reliable and will work the very day you'll need them https://github.com/backupchecker/backupchecker
Personally, I use Macrium Reflect incremental backups on my Windows machines, Time Machine on my MacBook and tar scripts on linux. These are copied to a NAS which is further synced with Dropbox.
Automated Portion: Every morning at 7:30am, a cron job runs to rsync my main computer's /home directory to my NAS that is sitting in a closet. The NAS has two drives that are mirrored to guard against hard drive failure. Every Saturday morning, a homebrewed script will separately check to make sure all my backups are current and will email me to advise me that everything is current and OK. If there is a problem, it will tell me which days are missing.
Manual Portion: Once every couple months, I backup the portion of my /home directory that is not heavy media like video to a large USB stick, which I keep at my office.
I don't love this setup, particularly because (1) an offsite backup does not happen automatically, (2) my video is not backed up offsite, and (3) it is not resilient to natural disasters in my area, since both my home and office are within a half-mile of each other in downtown New York City.
Moreover, because I keep my NAS mounted even when no backup is in progress, my backups are vulnerable to malicious code executed even at the non-superuser level. For example, a while back there was a bug in some gaming software (Steam?) where a script executed that would rm -rf /* because of a badly written shell script. If that had happened to me, it would wipe my /home directory and my NAS backup.
Eventually, I'd like to set up a Raspberry Pi running in another area of the country and have rsync push daily backups over ssh to that offsite computer.
Windows:
Genie (which sort of sucks performance-wise but all of the backup solutions on Windows seem to be terrible. I miss the built-in one from Windows 7.) to NAS, plus Backblaze.
Thanks! Looks really slick, I'm giving it a try. It's too bad there's no snapshots or restore/recovery but that's probably fine, with Windows those end up being pretty flakey anyways.
I use BackupPC (http://backuppc.sourceforge.net/) on an always-on 8-watt Fit-PC2 (http://www.fit-pc.com/web/products/fit-pc2/). BackupPC grabs twice-daily backups of my servers and once-daily backups of my personal machine, and emails me if there's any trouble. The backups aren't dependent on any third-party service, I don't have to worry about who else has access to the files and data on my personal system, and I habitually check its web interface for extra peace of mind.
I think it's hands-down the best cross-platform backup system currently available, but for some reason it's not a popular approach.
Seconded for BackupPC. I've been using it since I first saw a release announcement for it on Slashdot ~15 years ago. I combined it with automated weekly archival uploads to an off-site server (that I was also using for other purposes), as well as a manual monthly transfer I would make to an external hard drive (that would otherwise remain powered off, in case someone/something managed to get access to all of my live machines).
https://www.backblaze.com/ when I was mostly using Windows. $5/mo or $50/year per computer with no limit on filesize! Seriously, I've had multiple terabytes on there and it never complained. And the upload speeds keep getting faster.
One limitation of Backblaze is a limited file history. If a file gets modified and you don't notice, the old version will eventually disappear. So I have my "important" stuff in a Dropbox account with "packrat" enabled for infinite history.
My Linux-based data is on a RAID-1 which is not backup. But I can't find a cheap enough service to hold another 7.5TB of data.
I used to use BackupPC for onsite backups (which is a great product!), but wasn't comfortable with my lack of an offsite. I've since moved to Crashplan (http://www.code42.com/crashplan/), mostly because they have a Linux client as well as Windows and also allow peer to peer backups.
Mozy for personal (non-prod) computers. it's like $60/year. Set and forget, my computer was stolen last year, was up and running a day later, most of which was spent purchasing the new computer (Macbook)
Prod is more complicated, my company works with several multi-TB data sets (MongoDB) consisting of billions of swiftly changing kilobyte-sized documents. This can't be "backed up" in the traditional sense because it's impossible, barring custom hardware/great expense, to take a consistent snapshot of a multi-TB dataset distributed across dozens of machines. So we do the usual distributed replication across datacenters, put RAID underneath, etc. I worry more about corruption due to application errors than losing a disk for this stuff, though.
Since Attic is fast and duplicating I do hourly backups via cron at work. At the end of the day I prune them all and do a daily backup via systemd on shutdown. For offsite I manually sync via usb stick and unison, merge at home where daily backups are done as well. Rarely I update a backup hd stored at my parents.
Time Machine backup to my Synology NAS, then a time-honored offsite backup -- I clone my drive and physically leave the latest copy at my parents' house.
every time I visit I bring a newer drive and swap.
Windows desktop: has script to sync a handful of directories to Linux server. Run manually. Desktop has another drive that has images of the other drives and backups of game service downloads (Steam, GOG, etc.) that aren't anywhere else, to save space.
Linux server: has script to sync data drive to encrypted external drive. Run manually because drive needs to be manually connected and password entered (probably a way to work around this, but I'm lazy). I have 3 external drives that are swapped about 1-2 weeks to work, and 1-4 months to another state.
It might seem too manual, but I have a workaround. I host a podcast[0] where I say "don't forget that today is international backup awareness day[1], so backup your stuff" to remind me.
Attic ( https://attic-backup.org/ ) backups to my home NAS, then the NAS backs up to CrashPlan. I keep a certain number of daily/weekly/monthly snapshots and the backup script prunes once a day and verifies once every other day.
I also like to encrypt my backups before uploading them to CrashPlan. Each machine has a randomly generated key and the list of keys are then encrypted with a master key. I back up the list of keys to CrashPlan as well, but the master key is not. Instead, I split the key using Shamir's secret sharing and left copies of the component keys in a variety of places. If something ever totally destroys my home and I lose my local copy of the master key I can recover it by recombining a certain number of the component keys.
I have been looking into both of these technologies recently.
Attic has the most attractive feature set of the backup management tools I have looked at (deduplication and compression, support for remote repos, built in encryption, ability to mount repo as a FUSE file system), with the main downsides being that it is fairly new (and so just doesn't have as much testing as some of the older options) and its development model is not ideal. It is solely maintained by the original developer as a personal project and so when he is busy development stops. Recently, a handful of active community contributors started a fork (https://borgbackup.github.io/) because of this. The original developer still seems committed to working on Attic when time permits, but I'd prefer to see a team of capable developers maintaining something I was relying for the integrity of my data.
CrashPlan was the most attractive offsite backup solution because its price was competitive (similar to several other options, ~$6/month to backup one computer with no data size limit) and it provided cross-platform support for Windows, OS X, and Linux. I also like that it has a few different encryption options, including the option to generate your own encryption key that you do not share with CrashPlan.
I am curious about your decision to encrypt your data before uploading with CrashPlan. Are you trying to avoid storing any unencrypted data on your NAS because it is more exposed to the internet? Or do you not trust CrashPlan with your data? I would think that using CrashPlan with a custom key would be fairly secure. If you don't trust CrashPlan in that case, you probably shouldn't install it on your machine at all. The CEO of CrashPlan has commented on double encrypting data (http://superuser.com/a/589686).
What is the restore process like when combining Attic and CrashPlan? I don't think CrashPlan has any file system mount option, so you would need to restore an entire Attic repo to recover any files from it, correct? I guess this is okay since you have the NAS to restore from for individual file backup and you only need to restore from CrashPlan in the case of catastrophic failure (the NAS dies entirely) when you would need to do a full restore.
My current backup solution is an rsync script backing up to a remote machine I own. I previously used Bitcasa for offsite backup but was unhappy with the service and with the way the company kept changing its business model (shifted away from consumer cloud storage to a business / app market). I'd like to replace my rsync script with something more sophisticated like Attic and use another offsite option like CrashPlan.
I encrypt the data before shipping it off to CrashPlan mostly because it's the most flexible. I have a small amount of "sensitive" data related to my research work. It's not really anything that is actually worth protecting IMO, but I'm obligated to meet certain requirements when storing it. Encrypting it before uploading lets me have more fine-grained control over how it's encrypted. I also like to use different keys for different machines. I do actually trust CrashPlan, but I like doing it this way. The NAS isn't exposed to the internet, so that's not the issue.
Restoring from CrashPlan is basically as you describe. Assuming my home NAS isn't destroyed, I've just restored from the appropriate Attic repo for the machine off the NAS. If the NAS was destroyed too, well I'd have to restore from CrashPlan. My backups are targeted though, so a full restore is only at most like a TB of data (basically, my documents, configuration files and music collection).
65 comments
[ 2.3 ms ] story [ 143 ms ] threadI don't have massive amounts of important data to manage though so a simple solution like this is all I need.
Mac:
- Arq (Highly recommended) to Google Drive (Unlimited) and Amazon Glacier
- TimeMachine to two disks and a QNAP NAS
Servers:
- Tarsnap
- duplicity (https://blog.notmyhostna.me/automated-and-encrypted-backups-...)
on my workstation/notebook (both apple computers) i use time machine with different external harddrives i rotate weekly.
all setups run automated, but sometimes i trigger them manually (usually right before and/or after some major change in configuration, or when lots of new data are on the drives (eg: i copy all the holliday photos onto my laptop)
i run regular checks on server backups, e.g. check if they "are there" and if i could restore them properly if needed.
Well, that's not right... lol
#!/bin/sh
find /media/wd1T/backup -mtime -1 | mail -s "Backup Data 1 day old alt mdorf/valun " xx@xxxx.com
Manual Portion: Once every couple months, I backup the portion of my /home directory that is not heavy media like video to a large USB stick, which I keep at my office.
I don't love this setup, particularly because (1) an offsite backup does not happen automatically, (2) my video is not backed up offsite, and (3) it is not resilient to natural disasters in my area, since both my home and office are within a half-mile of each other in downtown New York City.
Moreover, because I keep my NAS mounted even when no backup is in progress, my backups are vulnerable to malicious code executed even at the non-superuser level. For example, a while back there was a bug in some gaming software (Steam?) where a script executed that would rm -rf /* because of a badly written shell script. If that had happened to me, it would wipe my /home directory and my NAS backup.
Eventually, I'd like to set up a Raspberry Pi running in another area of the country and have rsync push daily backups over ssh to that offsite computer.
Macs: Time Machine to a NAS, plus Backblaze.
Windows: Genie (which sort of sucks performance-wise but all of the backup solutions on Windows seem to be terrible. I miss the built-in one from Windows 7.) to NAS, plus Backblaze.
Linux: rdiff-backup to NAS.
Servers: Linode backup.
I think it's hands-down the best cross-platform backup system currently available, but for some reason it's not a popular approach.
- rsync-based backup of a sub-home directory where I keep sources, documents, and other things I want to be saved in case of disaster
- backup of all Trello cards via a custom python script
- machine shutdown
Arq: backs up photo library to S3 Glacier automatically
Carbon Copy Cloner: clones main hard drive to external once every two weeks and photo library to external (stored at parents' house) once every month
One limitation of Backblaze is a limited file history. If a file gets modified and you don't notice, the old version will eventually disappear. So I have my "important" stuff in a Dropbox account with "packrat" enabled for infinite history.
My Linux-based data is on a RAID-1 which is not backup. But I can't find a cheap enough service to hold another 7.5TB of data.
(Also I'm going to plug Archive Team's backup of the Internet Archive. http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/g... If you have a Mac or Linux box, put your spare hard disk space to good use.)
Automated: I have SpiderOak set up the auto backup select folders. I have about 1 TB of offline storage for $12/month.
Prod is more complicated, my company works with several multi-TB data sets (MongoDB) consisting of billions of swiftly changing kilobyte-sized documents. This can't be "backed up" in the traditional sense because it's impossible, barring custom hardware/great expense, to take a consistent snapshot of a multi-TB dataset distributed across dozens of machines. So we do the usual distributed replication across datacenters, put RAID underneath, etc. I worry more about corruption due to application errors than losing a disk for this stuff, though.
every time I visit I bring a newer drive and swap.
Simple, cost effective, foolproof.
Bup to snapshot the synchronized files every 5 minutes.
Backblaze for persistent cloud backup.
Semi-Manual (I need to dock my latptop and plug in the drives)
Weekly copy to external HDs (at home) using SyncBack
Manual
I backup by hand my FileZilla profiles, Google Docs, project management web app (XML file).
Linux server: has script to sync data drive to encrypted external drive. Run manually because drive needs to be manually connected and password entered (probably a way to work around this, but I'm lazy). I have 3 external drives that are swapped about 1-2 weeks to work, and 1-4 months to another state.
It might seem too manual, but I have a workaround. I host a podcast[0] where I say "don't forget that today is international backup awareness day[1], so backup your stuff" to remind me.
0: http://thenexus.tv/category/cs/
1: http://blog.codinghorror.com/international-backup-awareness-...
Near line storage. (https://cloud.google.com/storage/docs/nearline)
I also like to encrypt my backups before uploading them to CrashPlan. Each machine has a randomly generated key and the list of keys are then encrypted with a master key. I back up the list of keys to CrashPlan as well, but the master key is not. Instead, I split the key using Shamir's secret sharing and left copies of the component keys in a variety of places. If something ever totally destroys my home and I lose my local copy of the master key I can recover it by recombining a certain number of the component keys.
Attic has the most attractive feature set of the backup management tools I have looked at (deduplication and compression, support for remote repos, built in encryption, ability to mount repo as a FUSE file system), with the main downsides being that it is fairly new (and so just doesn't have as much testing as some of the older options) and its development model is not ideal. It is solely maintained by the original developer as a personal project and so when he is busy development stops. Recently, a handful of active community contributors started a fork (https://borgbackup.github.io/) because of this. The original developer still seems committed to working on Attic when time permits, but I'd prefer to see a team of capable developers maintaining something I was relying for the integrity of my data.
CrashPlan was the most attractive offsite backup solution because its price was competitive (similar to several other options, ~$6/month to backup one computer with no data size limit) and it provided cross-platform support for Windows, OS X, and Linux. I also like that it has a few different encryption options, including the option to generate your own encryption key that you do not share with CrashPlan.
I am curious about your decision to encrypt your data before uploading with CrashPlan. Are you trying to avoid storing any unencrypted data on your NAS because it is more exposed to the internet? Or do you not trust CrashPlan with your data? I would think that using CrashPlan with a custom key would be fairly secure. If you don't trust CrashPlan in that case, you probably shouldn't install it on your machine at all. The CEO of CrashPlan has commented on double encrypting data (http://superuser.com/a/589686).
What is the restore process like when combining Attic and CrashPlan? I don't think CrashPlan has any file system mount option, so you would need to restore an entire Attic repo to recover any files from it, correct? I guess this is okay since you have the NAS to restore from for individual file backup and you only need to restore from CrashPlan in the case of catastrophic failure (the NAS dies entirely) when you would need to do a full restore.
My current backup solution is an rsync script backing up to a remote machine I own. I previously used Bitcasa for offsite backup but was unhappy with the service and with the way the company kept changing its business model (shifted away from consumer cloud storage to a business / app market). I'd like to replace my rsync script with something more sophisticated like Attic and use another offsite option like CrashPlan.
Restoring from CrashPlan is basically as you describe. Assuming my home NAS isn't destroyed, I've just restored from the appropriate Attic repo for the machine off the NAS. If the NAS was destroyed too, well I'd have to restore from CrashPlan. My backups are targeted though, so a full restore is only at most like a TB of data (basically, my documents, configuration files and music collection).