De-dupe, will only transfer blocks changed since last snapshot.
Performance, it's super damn fast. Because the ZFS snapshot itself is just a stream you can also compress it on the wire.
Flexibility. You don't need to send the snapshot to another ZFS server. You could just take the ZFS snapshot streams and store them on say S3 compressed and re-assemble them. This would take custom tooling but it's definitely possible.
You can restore super quickly to an older version if you haven't deleted the local snapshots.
Similar to above but if you don't want to restore (as in make said snapshot the current active dataset) you can create a writable clone of one of the snapshots to play with it before committing to a restore or just to "go back in time" and play with something.
As a bonus you also now have your data on ZFS which has tons of great benefits apart from the point in time snapshots.
I'm using zfs-auto-snapshot[1] and zfs send/recv to backup my work desktop machine to a remote server. The naive approach of just doing 'zfs send -R -I snap1 snap2 | ssh remote zfs recv -dF pool'[2] has a number of drawbacks:
* Assumes remote is available 100% of the time. Recovery from downtime at the remote probably requires some manual intervention to get things back in sync.
* If the remote is in a different physical location with limited bandwidth between the hosts, compressing the stream on the fly isn't going to be particularly efficient with bandwidth.
I've built some scripts to help[3], dumping the necessary incremental streams into a local directory and then compressing them with lrzip[4]. This decouples the host and remote systems and the zfs streams compress really well: a 33Mb incremental stream I have here compresses to 4.4Mb with lrzip. Once you have a directory of compressed streams you can push them wherever you want (a remote server where you convert them back into zfs snapshots, giving you a live filesystem; S3 buckets etc.) You also are able to restore using standard operating system tools.
I'd assume btrfs is comparable, but haven't tried it myself.
I've been using backup2l with a custom driver that just calls "tar | gpg". I wouldn't flag it as "lacking". Backup2l can be used with any archiver.
If duplicity had an option to chose a backup ancestor, you could use backup2l with duplicity.
backup2l is strikingly simple and very effective at managing restores from it's incremental backup hierarchy.
In one of my tests (more than 2 years ago), the backup efficiency of daily, 3-level hierarchical tar (such as backup2l) vs Duplicity for a 1 year period was only 15% greater yet giving much more redundancy in case of failure.
If you consider how simple archives are and can be restored in case of problems, it's almost a no brainer.
I would also add that Tarsnap is probably the best option if you only need to backup a small size of data (a few KBs, a few MBs or even a few hundred MBs), since you have top-notch encryption with the picodollar pricing and stellar dedup you will pay much less than with any other provider as you pay per usage. For instance, rsync.net is a good destination (can be used with some of the tools in the list, like duplicity or attic) but you need to buy at least 50GB so for small backups it's not worth it.
We love tarsnap and think it is wonderful. We want to live in a world where people like Colin are selling things for picodollars.
Our strong suit is the ability to point any SSH tool you like at our storage (rsync, mostly, but some folks point duplicity or unison).[1]
Also, as we run on ZFS and have daily/weekly snapshots enabled by default, you can just forget about incrementals or versions or datasets ... just do a dumb mirror to us every day and we'll maintain, live, browseable, in your account, what is essentially an offsite "Time Machine".
We have an HN readers discount. Email and ask about it.
Yes, and I think your service is great. Just saying that it only starts to make sense when you need to backup 50Gb or more... If you just want to backup like 10Mb, or 200Mb or even 5Gb... On the other end, the more you need to backup the more sense it does, since your prices go down per Gb.
Thanks for the suggestions. I'm more inclined to use Open Source solutions, though, which Tivoli and Legato are not.
I agree with ZFS but I think it shines as a last resort backup (like rsync.net offers) and not as a main backup system since it doesn't do incremental and deduplication and so on...
I am not sure I understand your comment, but just to be clear: we (rsync.net) have ZFS snapshots enabled in your account, by default.
You can request any schedule of days/weeks/months you want, and their (very small) space usage counts against your paid quota. The first 7 dailies are always free (don't count).
What this means is you can just do a dumb rsync to us. No incrementals, no expiring datasets, no logic at all - just rsync to us every night and our snapshot rotation schedule does the rest for you. Just browse right in (using the SSH/SFTP based tool of your choice[1]) and grab a file from 6 days ago.
Email us and ask about the HN readers discount for new customers.
[1] Like filezilla, or SSHFS, or our windows drive mapper
ZFS is a also a good way to get consistent fast backups of databases by snapshotting. By ZFS copy-on-write its incremental by nature as it you only have to send over deltas incremental streams. Thus for site-to-site backups its usually very fast! Another advantage compared to rsync is that you do not need to directory traverse the whole file system in order to find the differences, I believe its similar to BTRFS in that aspect.
Then there is the feature that ZFS has checksums, so that when you write to disk you know what you get otherwise you can get corruption. RAID5-60 for me is a gamble that you can get hidden write disk errors unless there is checksums in software on a higher layer.
It's admittedly been 8-10 years since I was responsible for any serious backups so my experience is quite out of date, but Legato Networker is at the end of the day the best backup solution I've ever used. Yes it costs quite a bit of money and it's not trivial to set up, but once you've got it working it works so much better and easier than anything else I've ever used.
I have a few years experience with TSM and always found it to be pretty fast and very reliable. Like most IBM software, it looks overly complicated and obtuse at first glance, but once you overcome a few "standard" obstacles(1), you realize there's some pretty serious engineering under there and it's actually quite likable. It's not cheap though...
(1) IBM software seems so be designed by smart people for really hard scenarios, but it all gets buried under tons of enterprisey crap and UIs designed by monkeys...
1. Drop the excruciatingly slow and clunky web console and learn to use the CLI (really, it's horrible);
2. Resist the urge to throw your arms up in disgust because the syntax and concepts seem foreign (it probably won't look so foreign to mainframe people, and it isn't actually bad, just different);
3. ??
4. Realize the CLI and storage concepts are actually very powerful.
You have obnam: `Really slow for large backups (from a benchmark between obnam and attic)` , but then, no section for attic itself.
It's probably also wise to roughly group the backup systems by algorithm-class (e.g. separate rsnapshot from rdiff-backup from duplicity from attic/obnam/zbackup) since they result in different bandwidth and storage properties. Duplicity will need a full re-upload to avoid unbounded growth of both storage and restore-time, but such a re-upload is prohibitive for DSL users.
The only novel thing is its use of symmetric encryption keys which are
used to encrypt the data and are included in the backup repository, encrypted by your regular private gpg key. This allows giving additional gpg keys access to the backup after it has been made. http://liw.fi/obnam/encryption/
(Which is useful for eg, backing up a server. You can make a dedicated gpg key for that server, but give your personal gpg key access to the backups to restore later.)
Anyway, the generation of the symmetric encryption key is what needs an entropy source. AFAIK this is done once per repository.
What I would like to use is some kind of small programs from which backup script might be crafted. Actually most of the tools are available. Incremental backups with rsync. Encryption with openssl. Archiving/compression with tar/gzip/7-zip. But scripts will be quite verbose and error-prone. Better solution should be available.
Yes, that is another option, but that would be in the category "roll your own". You can start rolling your own with some shell scripts, or with some python, etc
Actually I think this is how most solutions started...
I'm quite happy with duplicity for backing up to S3. It's incremental backups that fits in the with the S3 charging model, so you only pay for restores. I did find that it's important to set it to do a full backup from time to time, e.g. once per month. Otherwise, a restore will take very long and also more expensive. I discovered this the hard way, by doing my second test restore 7 months after the last full backup. Fortunately it was a test restore. I use the duply wrapper script to make setting it up even simpler.
Back in 2005 I've used Bacula to setup a distributed backup system for a big company. I don't work there since years, but AFAIK it still runs well and manages around ~50 (both servers and desktops running different operating systems).
Yes it's hard to approach (we spent ~2 weeks to learn and test it before deploying in production) but it had the features we needed:
* scheduling
* retention policies (store for 1 year then rotate, multiple tapes, etc...)
* backup on DST tapes
Perhaps, if you just need to backup a single server, Bacula isn't the right solution or, at least, is overkill :-)
As far as I remember, you can use file storage and set the size of the file itself as well. so, for example, you might have 4GB file images to burn on dvd. Of course, it really shines with tape backups :-)
One nice feature of multi-volume backups in Bacula is the fact that it remembers (through its Catalog) where to find a backup, given the file name and/or a date. So it might ask you: "Insert volume X-Y" to restore the data you need.
While you might disqualify it due to lack of built in encryption, you should note that given that one of your comments implies you control the server where the backups enter cold storage that you can encrypt the disk/array where the backups are stored independent of the backup tool used to copy the data to the backup server.
When I tried duplicity it worked great for all of my servers but one. I am not sure what caused the issues on the one server, possibly clock skewing, but backups kept consistently getting corrupted. I'm now a happy bup user, and as far as I'm aware they're working on pruning old backups.
Attic seems promising. Does anyone have any experience of that versus bup?
Attic has still problems for large backups (when the block index gets >2gb). There are also several minor issues due to the tool being relatively new (file selection is lacking). Check it's issues page: https://github.com/jborg/attic/issues/
To work around some of attic's more UI-level shortcomings, I made a wrapper script called atticmatic that adds a declarative config file, excludes file, etc:
Interesting overview, some have to look into. But nobody/nothing mentions the option of the application handling the data you want to backup, to perform the backups.
Imho it's ideal to have the application handling the data also create the backups. And then transport them to remote locations via any means viable or any of the in the article mentioned backup systems again. Though having the application back things up vs backup software, i tend to think there is less tuning required, less chance on external locks etc etc.
That is assuming there is a single source of data in need of backup and that the data is generated/handled by an application. Data could be from many different sources and formats: databases, images, video, audio, pdf, text...
But the same applies to that. These 'other data formats' from other sources too, could also be replicated from the software handling them, assuming again! :o)
edit
Speaking from a perspective where i have seen many parties that went wrong with their '3rd party backup application', i tend to try to replicate most from the application themselves to other (safe) locations, then to rely on a 3rd party app which got setup in 2005, but in the meantime received little attention. Hence i prefer to look at the application generating/receiving/handling the data.
Using bacula with a twin cabinet Quantum tape library with two LTO4 tape drives (all fibre channel), looking to replace with an equivalent cabinet with LTO6. Bacula really has a difficult learning curve but the features are equivalent to other enterprise-grade pieces of software like networker, etc.
Pros: it's stable, supports several different transfer methods, is completely simple (and still flexible) to admin, and it's totally reliable. It does file deduplication and compression and manages its own pool very efficiently. I've had it responsible for networks of 50+ systems before and it worked without any trouble. It has a sensible and reliable email notification system. It gets the basics really right, which for some reason seems to be a problem with a lot of other backup software. The documentation is good. The developer is friendly, has been working on it for over a decade now, and is easy to reach by email. There's a quiet mailing list with some folks that have used BackupPC for Truly Large Networks and know it about as well as the developer. Version 4.0 is pretty awesome. It has saved my butt a few times and a client's butt at least twice. Also, it's free, assuming you have a server somewhere to run it on; it's not a SaaS or PaaS or YaWaaS, so there's no monthly cost.
Cons: it doesn't do encrypted backups. It's written in Perl, so that's probably a deal breaker for some people who don't know any better. Initial setup can be a bit of a pain, especially if you're new to it. The web interface doesn't use jQuery or LESS or CSS3 transitions or a lot of stock photography, so some people might find it scary-looking. It doesn't hold your hand, you'll have to be comfortable with the CLI every once in a while if you need to do something fancy (like, say, restore a batch of files to their original locations using a text file as input -- which I've done with it, btw). It won't make you coffee in the morning.
Good point, and I'm no cryptologist either to find any flaws on it or to verify its robustness... Maybe someone in the community has the required knowledge about encryption to say something about this?
Using gpg is not so awesome. Obnam uses gpg. Since Obnam invokes gpg in batch mode, if you want to have a passphrase, you have to use gpg-agent, which at least for me took more effort than I found reasonable to set up on a GUIless server.
Furthermore, all the crypto config depends on gpg defaults or your gpg.conf. Whether this is good or bad depends on whether you are OK with gpg's defaults that are chosen for a non-Obnam use case and whether you like tweaking gpg config.
While figuring this out, I started wishing that Obnam used libsodium instead of gpg to avoid configuration and especially gpg-agent. (libsodium didn't exist when Obnam was created.)
Having to be aware of tools like this is the problem when you face the requirement of having to set up gpg-agent and you don't already know how to do so in an environment where a desktop environment from your distro hasn't done it for you.
I use and have been using Amanda Network Backup for ~5 years now, and am extremely happy with the results. Some of the highlights of my decision to use this tool include:
- Uses native OS tools (tar, gzip, gpg in my case) for the actual backups and restores, and includes the actual command used to create the archive in the header of the archive! You can use this to restore it without Amanda in the event of an emergency.
- Supports on-disk backups in a holding area for quick restores
- Supports S3 as a virtual tape library
- Supports vaulting (i.e. moving an archive from one tape library to another)
- Your choice of client, server, or no compression & encryption
- Highly scriptable
- Works over SSH, among other methods
- Catalog data is easily backed up itself via simple OS commands, and stored in S3.
The systems I backup are all in AWS, so this is ideal for me. I've frequently thought it would be ideal to adapt Amanda's script agents to creating EBS snapshots, but I simply haven't had the time. It's on my someday-maybe list. Remember to vault your backups to another region!
I honestly can't understand why distro makers like Cannonical don't have some out-of-the-box solution that works like Timemachine. Every step one has to configure himself can lead to a mistake and thus to a potential data loss. IMO an OS without a very-simple-to-set-up backup system that supports at the very least remote storage, incremental backup and restore during reinstallation, is incomplete.
It shouldn't be desktop-only. It could be a well tested script with a good CLI that also has a simple GUI. Something like a backup shouldn't have many options anyway. Apple has the right ideas there IMO: target volume selection and defining backup exceptions are pretty much all you need. I'd prefer if the target were simpler than a sparsebundle though, which should totally be doable with an rdiff or rsync backed script solution.
After having a look at duplicity [1], it actually reads as pretty decent. I still wonder why research is needed for such a thing, instead of Cannonical taking it, installing a good default script, adding some GUI in their installation image (choosing your sftp backup target, username, password (managed by keyring)), so everyone would already be on the right path.
I've been using duplicity for a few years now and have been quite satisfied. It did seem slow when I first started using it. I learned to live with it; you run backups overnight anyways.
From what I can remember, a lot of the slowness came from GPG, and specifically files being compressed before encryption. Disabling compression speeds things up, but trades off disk space (and security).
I wonder what it'd take to reengineer the thing to take advantage of multicore -- being CPU-bound on a single core is I think what makes it slow.
Something like zfsnap? (https://github.com/zfsnap/zfsnap) No special GUI but you can just browse to /.zfs/snapshot/<timestamp> and manipulate the files like normal files.
I'm a bit confused by this. So is ZFS a requirement or not? Do I have to reformat an existing Linux to ZFS in order to use it? Or is that only for the target volume? Even OSX, which it claims to support? I've never seen anyone using ZFS in production there to be honest.
Yes ZFS is a requirement, and it is designed to run on any OS that runs ZFS. You don't need the full machine to be running ZFS, just whatever volume you want to snapshot.
It is a bit unclear if you haven't been looking at other snapshot management utilities. Many other tools have various dependencies which may or may not work on your chosen operating system.
So generally the requirements are:
- ZFS running on the volume you want to snapshot
- Bourne shell
- Gregorian calendar
As far as OS X goes, I'm not sure how many people run ZFS in production but I know there were some people working on it. I've never used it but it still seems to be updated: https://github.com/openzfsonosx/zfs
130 comments
[ 25.5 ms ] story [ 1848 ms ] threadPerformance, it's super damn fast. Because the ZFS snapshot itself is just a stream you can also compress it on the wire.
Flexibility. You don't need to send the snapshot to another ZFS server. You could just take the ZFS snapshot streams and store them on say S3 compressed and re-assemble them. This would take custom tooling but it's definitely possible.
You can restore super quickly to an older version if you haven't deleted the local snapshots.
Similar to above but if you don't want to restore (as in make said snapshot the current active dataset) you can create a writable clone of one of the snapshots to play with it before committing to a restore or just to "go back in time" and play with something.
As a bonus you also now have your data on ZFS which has tons of great benefits apart from the point in time snapshots.
* Assumes remote is available 100% of the time. Recovery from downtime at the remote probably requires some manual intervention to get things back in sync.
* If the remote is in a different physical location with limited bandwidth between the hosts, compressing the stream on the fly isn't going to be particularly efficient with bandwidth.
I've built some scripts to help[3], dumping the necessary incremental streams into a local directory and then compressing them with lrzip[4]. This decouples the host and remote systems and the zfs streams compress really well: a 33Mb incremental stream I have here compresses to 4.4Mb with lrzip. Once you have a directory of compressed streams you can push them wherever you want (a remote server where you convert them back into zfs snapshots, giving you a live filesystem; S3 buckets etc.) You also are able to restore using standard operating system tools.
I'd assume btrfs is comparable, but haven't tried it myself.
[1]: https://github.com/zfsonlinux/zfs-auto-snapshot
[2]: see https://github.com/adaugherity/zfs-backup for example
[3]: currently in my local fork of zfstools at https://github.com/mhw/zfstools/blob/master/bin/zfs-save-sna...
[4]: http://ck.kolivas.org/apps/lrzip/README
http://backup2l.sourceforge.net
So while yes, there's no "native support" it's definitely not hard to add.
While this is just a random google result[1] as I don't have access to the aforementioned snippet, you'll get the idea I hope.
[1]: http://www.iniy.org/?p=151
If duplicity had an option to chose a backup ancestor, you could use backup2l with duplicity.
backup2l is strikingly simple and very effective at managing restores from it's incremental backup hierarchy.
In one of my tests (more than 2 years ago), the backup efficiency of daily, 3-level hierarchical tar (such as backup2l) vs Duplicity for a 1 year period was only 15% greater yet giving much more redundancy in case of failure.
If you consider how simple archives are and can be restored in case of problems, it's almost a no brainer.
Our strong suit is the ability to point any SSH tool you like at our storage (rsync, mostly, but some folks point duplicity or unison).[1]
Also, as we run on ZFS and have daily/weekly snapshots enabled by default, you can just forget about incrementals or versions or datasets ... just do a dumb mirror to us every day and we'll maintain, live, browseable, in your account, what is essentially an offsite "Time Machine".
We have an HN readers discount. Email and ask about it.
[1] http://www.rsync.net/resources/howto/remote_commands.html
Other than that my vote is on ZFS snapshots reliable.
I agree with ZFS but I think it shines as a last resort backup (like rsync.net offers) and not as a main backup system since it doesn't do incremental and deduplication and so on...
You can request any schedule of days/weeks/months you want, and their (very small) space usage counts against your paid quota. The first 7 dailies are always free (don't count).
What this means is you can just do a dumb rsync to us. No incrementals, no expiring datasets, no logic at all - just rsync to us every night and our snapshot rotation schedule does the rest for you. Just browse right in (using the SSH/SFTP based tool of your choice[1]) and grab a file from 6 days ago.
Email us and ask about the HN readers discount for new customers.
[1] Like filezilla, or SSHFS, or our windows drive mapper
Then there is the feature that ZFS has checksums, so that when you write to disk you know what you get otherwise you can get corruption. RAID5-60 for me is a gamble that you can get hidden write disk errors unless there is checksums in software on a higher layer.
Always scrub your ZFS source and backup pool.
(1) IBM software seems so be designed by smart people for really hard scenarios, but it all gets buried under tons of enterprisey crap and UIs designed by monkeys...
1. Drop the excruciatingly slow and clunky web console and learn to use the CLI (really, it's horrible);
2. Resist the urge to throw your arms up in disgust because the syntax and concepts seem foreign (it probably won't look so foreign to mainframe people, and it isn't actually bad, just different);
3. ??
4. Realize the CLI and storage concepts are actually very powerful.
And it has a neat curses based GUI called NinjaHelper
That's correct, Backupninja can use Duplicity as a backend: https://labs.riseup.net/code/projects/backupninja/wiki/Dup
It's probably also wise to roughly group the backup systems by algorithm-class (e.g. separate rsnapshot from rdiff-backup from duplicity from attic/obnam/zbackup) since they result in different bandwidth and storage properties. Duplicity will need a full re-upload to avoid unbounded growth of both storage and restore-time, but such a re-upload is prohibitive for DSL users.
lru-size=1024
upload-queue-size=512
See http://listmaster.pepperfish.net/pipermail/obnam-support-obn...
The only novel thing is its use of symmetric encryption keys which are used to encrypt the data and are included in the backup repository, encrypted by your regular private gpg key. This allows giving additional gpg keys access to the backup after it has been made. http://liw.fi/obnam/encryption/
(Which is useful for eg, backing up a server. You can make a dedicated gpg key for that server, but give your personal gpg key access to the backups to restore later.)
Anyway, the generation of the symmetric encryption key is what needs an entropy source. AFAIK this is done once per repository.
Actually I think this is how most solutions started...
Yes it's hard to approach (we spent ~2 weeks to learn and test it before deploying in production) but it had the features we needed:
* scheduling
* retention policies (store for 1 year then rotate, multiple tapes, etc...)
* backup on DST tapes
Perhaps, if you just need to backup a single server, Bacula isn't the right solution or, at least, is overkill :-)
While you might disqualify it due to lack of built in encryption, you should note that given that one of your comments implies you control the server where the backups enter cold storage that you can encrypt the disk/array where the backups are stored independent of the backup tool used to copy the data to the backup server.
The offsite backups are on USB drives that are encrypted.
We just created a shell script to automate that, and run the database dumps prior to running rsnapshot.
https://spideroak.com/faq/questions/67/how_can_i_use_spidero...
Attic seems promising. Does anyone have any experience of that versus bup?
http://torsion.org/atticmatic/
Imho it's ideal to have the application handling the data also create the backups. And then transport them to remote locations via any means viable or any of the in the article mentioned backup systems again. Though having the application back things up vs backup software, i tend to think there is less tuning required, less chance on external locks etc etc.
But the same applies to that. These 'other data formats' from other sources too, could also be replicated from the software handling them, assuming again! :o)
edit
Speaking from a perspective where i have seen many parties that went wrong with their '3rd party backup application', i tend to try to replicate most from the application themselves to other (safe) locations, then to rely on a 3rd party app which got setup in 2005, but in the meantime received little attention. Hence i prefer to look at the application generating/receiving/handling the data.
And then make hard-links of all files to another folder, named BACKUP_yymmdd
This way, the backup is incremental, and you have snapshots of older versions of the backup, where there is structural sharing between snapshots.
Pros: it's stable, supports several different transfer methods, is completely simple (and still flexible) to admin, and it's totally reliable. It does file deduplication and compression and manages its own pool very efficiently. I've had it responsible for networks of 50+ systems before and it worked without any trouble. It has a sensible and reliable email notification system. It gets the basics really right, which for some reason seems to be a problem with a lot of other backup software. The documentation is good. The developer is friendly, has been working on it for over a decade now, and is easy to reach by email. There's a quiet mailing list with some folks that have used BackupPC for Truly Large Networks and know it about as well as the developer. Version 4.0 is pretty awesome. It has saved my butt a few times and a client's butt at least twice. Also, it's free, assuming you have a server somewhere to run it on; it's not a SaaS or PaaS or YaWaaS, so there's no monthly cost.
Cons: it doesn't do encrypted backups. It's written in Perl, so that's probably a deal breaker for some people who don't know any better. Initial setup can be a bit of a pain, especially if you're new to it. The web interface doesn't use jQuery or LESS or CSS3 transitions or a lot of stock photography, so some people might find it scary-looking. It doesn't hold your hand, you'll have to be comfortable with the CLI every once in a while if you need to do something fancy (like, say, restore a batch of files to their original locations using a text file as input -- which I've done with it, btw). It won't make you coffee in the morning.
Backup data rests on a seperate, encrypted partition.
Furthermore, all the crypto config depends on gpg defaults or your gpg.conf. Whether this is good or bad depends on whether you are OK with gpg's defaults that are chosen for a non-Obnam use case and whether you like tweaking gpg config.
While figuring this out, I started wishing that Obnam used libsodium instead of gpg to avoid configuration and especially gpg-agent. (libsodium didn't exist when Obnam was created.)
Did you try Keychain¹? I've used it in the past to auto-sign deb packages, and it was simple to set up.
¹ http://www.funtoo.org/Keychain
Having to be aware of tools like this is the problem when you face the requirement of having to set up gpg-agent and you don't already know how to do so in an environment where a desktop environment from your distro hasn't done it for you.
- Uses native OS tools (tar, gzip, gpg in my case) for the actual backups and restores, and includes the actual command used to create the archive in the header of the archive! You can use this to restore it without Amanda in the event of an emergency.
- Supports on-disk backups in a holding area for quick restores
- Supports S3 as a virtual tape library
- Supports vaulting (i.e. moving an archive from one tape library to another)
- Your choice of client, server, or no compression & encryption
- Highly scriptable
- Works over SSH, among other methods
- Catalog data is easily backed up itself via simple OS commands, and stored in S3.
The systems I backup are all in AWS, so this is ideal for me. I've frequently thought it would be ideal to adapt Amanda's script agents to creating EBS snapshots, but I simply haven't had the time. It's on my someday-maybe list. Remember to vault your backups to another region!
(Edits: formatting)
The problem is though, that the last time I've checked, it was excessively simple- it didn't support individual files selection.
[1] https://help.ubuntu.com/community/DuplicityBackupHowto
From what I can remember, a lot of the slowness came from GPG, and specifically files being compressed before encryption. Disabling compression speeds things up, but trades off disk space (and security).
I wonder what it'd take to reengineer the thing to take advantage of multicore -- being CPU-bound on a single core is I think what makes it slow.
It is a bit unclear if you haven't been looking at other snapshot management utilities. Many other tools have various dependencies which may or may not work on your chosen operating system.
So generally the requirements are:
As far as OS X goes, I'm not sure how many people run ZFS in production but I know there were some people working on it. I've never used it but it still seems to be updated: https://github.com/openzfsonosx/zfs