I use a great tool called tarsnapper to cull old backups. It lets you configure schedules like "keep daily backups for the last 30 days, then keep a backup every 3 months for the last year, then keep yearly backups".
I have two custom python scripts that manage my tarsnap process, one that performs my daily backups (dumps various dbs the performs backups) the other scrubs backups older than 45 days based on the name of the backup skipping over. All automated backups are named with an ISO timestamp. Has worked rather well so far, and when needed I can disable the cron that deletes temporarily if needed.
Can someone shed some light on why one might use tarsnap over another service? The tarsnap pricing model seems wild ($0.25/gb/month) considering you can do the exact same thing for 10% the price with something like Borg for deduping/encryption [0] + Rclone for syncing to the cloud [1] + S3 (which is what tarsnap uses on the backend). And this is only one of many different toolsets that can do the same dedupe/encryption/sync. And you can do it for even cheaper using something like Backblaze B2 instead of S3. Despite this, I see tarsnap recommended a lot amongst the tech crowd, but don't understand why. What am I missing?
tarsnap is simple. The rest of the stuff you mention requires me to set things up and write scripts to wrap it all and sign up for AWS and monitor it and ....
Also your method might be much cheaper, but tarsnap is still inexpensive for most use cases. So how much actual money is saved vs. the effort level put in?
I previously used Tarsnap, but I moved to using Borg on rsync.net. Borg can do all the stuff Tarsnap can do and comes with 'borg prune' included. In fact for Tarsnap I would have needed to write a custom script to perform regular pruning.
By relying on rsync.net's automated ZFS snapshots I get the same or similar protection compared to Tarsnaps's write-only keys.
For my personal use case cost wasn't an issue, though. It's less than 10 GB of data stored.
After many years, I moved from tarsnap to Borg as well (hosted with borgbase.com).
It wasn't about price for me either. On the few occasions I actually had to restore some files, Tarsnap was glacially slow. That made me nervous if I ever hit a situation where I had to do a full restore.
Like you say, there's the quality-of-life features that I now don't have to script. Pruning, missed backup alerts, monthly reports are all things I've realized were sorely missing from my Tarsnap experience.
I believe it inherits the same pruning behavior of Borg itself.
I don't know the architecture of Borg storage too closely, but I assume that since it is all end-to-end encrypted, Borgbase cannot do the pruning for you.
If you use append-only keys, they can't (destructively) prune, you have to use regular keys to do that.
Setting up borg + rclone + B2 took me about an hour to set up and zero time since then. Something like Duplicati or Duplicacy makes it even simpler.
I'll agree that tarsnap is still 'simpler' since I guess maybe it takes 15 mins to set up versus an hour, but I'm still surprised that someone would consider the 45 minutes saved to be worth possibly hundreds/thousands of extra dollars in backup costs.
>Also your method might be much cheaper, but tarsnap is still very inexpensive for most use cases. So how much actual money is saved vs. the effort level put in?
What are the use cases you use tarsnap for? Is tarsnap only used for small amounts of data or something? I guess I just cannot get over the hurdle that tarsnap seems obscenely priced given the cost of storage these days. You can get 2TB of storage for ~$120/yr on something like Dropbox/B2/Google Drive/etc. That same 2TB would cost six thousand dollars a year on tarsnap!!
Setting up restic/duplicacy/kopia/etc is easy and you get to choose your storage from wide array of choices that's pretty much 1/10 the price. Does tarsnap even have retention rules?
Just because someone behind may be famous, charging 10 times for something that's not even better is absurd and should be seen as an obsolete service when there are a dozen others with competitive pricing.
I see you're happy to pay $100 for something I pay $10 because making a cron job is tough for you.
Maybe not a good enough reason, but the author of tarsnap is very active on HN (cperciva). He's also very smart... ("Dr. Colin Percival studied mathematics at Simon Fraser University, entering at age 13, before going on to a doctorate in Computing at Oxford University, where his thesis covered problems in string matching, data compression, and file synchronization.")
Tarsnap's easy for both me and accounting, works, and has very good support. I don't feel like setting up a solution that is not understandable to the person who will look at this while I take vacation.
I've learned that the ultimate test of documentation, training, scripts, and good services is "can I go on an undisturbed vacation" when the less experienced people have to run the ship.
I think it's basically mostly because the Tarsnap tooling has a very principled design, is reasonably easy to use for power users with basic use cases, and perhaps most of all because the operator of the service (Colin Percival) is a tremendously well respected figure in the cryptography world.
I describe the tarsnap tool as being "principled" because it draws a pretty tight boundary of what functionality is in-scope. As a result, I think most real "backup" scenarios will require some kind of tooling to be used as a wrapper (e.g. GFS or even simpler retention scheduling).
I was a Tarsnap user for some time, but in the end I've found that using Duplicati with B2 is actually easier than Tarsnapper and significantly less expensive, as my usage totals a bit over 7TB - not affordable for me with Tarsnap. As a result I haven't been recommending Tarsnap for some time.
I do think it's important to emphasize that "B2" or "S3" does not necessarily mean doing scripting or whatever yourself - probably the biggest advantage of choosing one of these commodity storage services is that they are supported by a wide variety of open-source and commercial tools, generally with much a much more sophisticated featureset than the tarnsap tool. Duplicati as mentioned is a more GUI-centric (web interface) tool with block dedup and retention management for .NET/Mono. "rclone" is a command-line tool for Linux that's fairly similar in usage to the tarsnap tool but also has additional features. Both can be set to encrypt backups before uploading. Both support either S3 or B2 (and I assume also Google Cloud Storage, Azure Storage, etc).
tarsnap is dead simple to use if you have classic Unix sysadmin knowledge, it takes care of the encryption/deduplication work behind the scenes, I trust it to work properly and be secure because
the creator is someone who has integrity (you can email him any time and get no-nonsense and refreshingly honest replies) and the price only becomes an issue if you’re storing massive amounts of data, at which point you have probably outgrown it.
31 comments
[ 10.9 ms ] story [ 84.1 ms ] threadtarsnap --list-archives | prunef | xargs tarsnap -d
https://git.sr.ht/~apreiml/prunef/
GNU xargs: Normally runs the command at least once, but you can turn that off with "-r" or "--no-run-if-empty".
OpenBSD xargs: Like GNU, but only has the short "-r" version of the flag.
FreeBSD xargs: Never runs the command if the input is empty, has "-r" (but not "--no-run-if-empty") as a no-op compatibility flag for GNU.
macOS xargs: Never runs the command if the input is empty (has no "-r" or "--no-run-if-empty" flags).
0: https://borgbackup.readthedocs.io/en/stable/
1: https://rclone.org/
Also your method might be much cheaper, but tarsnap is still inexpensive for most use cases. So how much actual money is saved vs. the effort level put in?
By relying on rsync.net's automated ZFS snapshots I get the same or similar protection compared to Tarsnaps's write-only keys.
For my personal use case cost wasn't an issue, though. It's less than 10 GB of data stored.
It wasn't about price for me either. On the few occasions I actually had to restore some files, Tarsnap was glacially slow. That made me nervous if I ever hit a situation where I had to do a full restore.
Like you say, there's the quality-of-life features that I now don't have to script. Pruning, missed backup alerts, monthly reports are all things I've realized were sorely missing from my Tarsnap experience.
I don't know the architecture of Borg storage too closely, but I assume that since it is all end-to-end encrypted, Borgbase cannot do the pruning for you.
If you use append-only keys, they can't (destructively) prune, you have to use regular keys to do that.
I'll agree that tarsnap is still 'simpler' since I guess maybe it takes 15 mins to set up versus an hour, but I'm still surprised that someone would consider the 45 minutes saved to be worth possibly hundreds/thousands of extra dollars in backup costs.
>Also your method might be much cheaper, but tarsnap is still very inexpensive for most use cases. So how much actual money is saved vs. the effort level put in?
What are the use cases you use tarsnap for? Is tarsnap only used for small amounts of data or something? I guess I just cannot get over the hurdle that tarsnap seems obscenely priced given the cost of storage these days. You can get 2TB of storage for ~$120/yr on something like Dropbox/B2/Google Drive/etc. That same 2TB would cost six thousand dollars a year on tarsnap!!
Setting up restic/duplicacy/kopia/etc is easy and you get to choose your storage from wide array of choices that's pretty much 1/10 the price. Does tarsnap even have retention rules?
Just because someone behind may be famous, charging 10 times for something that's not even better is absurd and should be seen as an obsolete service when there are a dozen others with competitive pricing.
I see you're happy to pay $100 for something I pay $10 because making a cron job is tough for you.
This is against HN rules FYI. No reason for this kind of trolling/goading. My use case doesn't even come close to matching that cost anyway.
https://news.ycombinator.com/newsguidelines.html
I've learned that the ultimate test of documentation, training, scripts, and good services is "can I go on an undisturbed vacation" when the less experienced people have to run the ship.
I describe the tarsnap tool as being "principled" because it draws a pretty tight boundary of what functionality is in-scope. As a result, I think most real "backup" scenarios will require some kind of tooling to be used as a wrapper (e.g. GFS or even simpler retention scheduling).
I was a Tarsnap user for some time, but in the end I've found that using Duplicati with B2 is actually easier than Tarsnapper and significantly less expensive, as my usage totals a bit over 7TB - not affordable for me with Tarsnap. As a result I haven't been recommending Tarsnap for some time.
I do think it's important to emphasize that "B2" or "S3" does not necessarily mean doing scripting or whatever yourself - probably the biggest advantage of choosing one of these commodity storage services is that they are supported by a wide variety of open-source and commercial tools, generally with much a much more sophisticated featureset than the tarnsap tool. Duplicati as mentioned is a more GUI-centric (web interface) tool with block dedup and retention management for .NET/Mono. "rclone" is a command-line tool for Linux that's fairly similar in usage to the tarsnap tool but also has additional features. Both can be set to encrypt backups before uploading. Both support either S3 or B2 (and I assume also Google Cloud Storage, Azure Storage, etc).
0: https://news.ycombinator.com/item?id=9224
Funny I just found out through yesterday's thread about top HN comments: https://news.ycombinator.com/item?id=24531972
https://support.powertools-tech.com/safonso/tsar