HN Explorer

Show HN: Low-cost backup to S3 Glacier Deep Archive (github.com)

120 points by mrich ↗ HN
Hi,

most people (hopefully) have local backups. However, when that backup fails, it is good to have a backup stored somewhere off-site. In the old days you would ship physical drives/tapes, which is cumbersome, costly, and slow. With fast upload speeds, it is now possible to upload your data to the cloud. I have found S3 Glacier Deep Archive to be a great solution for this:

- It is very cheap ($1/TB/month for US region) - Very reliable (99.999999999% data durability, data spread over 3 Availability Zones)

However, usability out of the box is not that great, I'm not aware of any automated backup solution for Deep Archive. This free project provides that.

Currently, ZFS is required, but that might change. Please try it out and provide feedback!

143 comments

[ 86.8 ms ] story [ 4769 ms ] thread
ogig
This recalled a horror story about huge charges when retrieving the data. I searched the link and seems like pricing was changed since the blog entry [1].

Still good idea to check the extra charges when reading.

[1] https://medium.com/@karppinen/how-i-ended-up-paying-150-for-...

quickthrower2
This is mentioned in the link, about $100 to get your 1TB out.

I can see home backup use cases where you might be happy to get it out at 100Gb/month over time. For example, a home photo/video library where you can view arbitrary photos, and slowly pull out all of them.

Ekaros
I think Amazon Glacier should be considered the third or fourth backup. First being local, second cloud or distributed and then fourth as last resort off-site. Used when everything else has catastrophically failed.

At which point 100$ per TB isn't that bad.

formerly_proven
.
hampereddustbin
Just a single drive can be more than 16TB nowadays, having just a couple of those makes $90/TB seem quite high.
Sebb767
That's 90$ for redundant tape storage, stashed away securely in some place far away from your home. A 16TB drive is cheap, but you need a few of them to be safe against file corruption and bit rot and you need to stash them somewhere where a house fire, robbery or natural disaster can't reach them. Also, the storage itself is quite cheap, just the retrieval is expensive. But, as other people said, if you need to get this data, that few dollars won't be your problem.
hampereddustbin
Disasters often bring financial difficulty, in which cost of repair is very much a problem. It's hard to justify for any individual when much cheaper and easier to use and understand services are available. It's great if your enterprise is looking for a compliance checkbox to tick, Glacier can save a lot of time in that case.
jjice
Yeah, it's putting low cost up front, but if you need the retrieval NOW, you're going to pay it. On any cloud provider, you should absolutely be reading their pricing carefully so you can account for the cost of these situations.
hampereddustbin
outbound traffic costs don't change at all regardless of when you need your data, it will always be atleast $90/TB
patrickthebold
As mentioned, AWS has simplified the pricing model, so getting it all at once isn't hugely expensive anymore.
Etheryte
Since I'm not that familiar with the pricing model behind this, did I understand correctly that it costs roughly $1/TB/month to store and roughly $95/TB to restore? The price seems steep at first, but comparing it to regular backup services where the cost usually adds up to roughly $100 per year it starts to make sense. I have backups but I don't think I would ever need them more than once a year, even that would be alarmingly often.
iLoveOncall
The price is not steep at all. This is a last resort solution, when all your other backups have failed and you need to retrieve your data. The idea is that you never need to restore it.
hampereddustbin
The backup and store service is cheap, general outgoing bandwidth is very non-trivial as it also applies to every other service in AWS; It's designed to keep your data in and heavily discourage multi-cloud
[deleted]
(comment deleted)
speed_spread
Glacier's primary business use is not as a backup solution but as a regulatory compliance tool. The idea is that you can guarantee regulatory bodies that whatever historical data that needed by law to be kept for x years has been preserved based on AWS' own claims of persistence.
iLoveOncall
I can totally believe that, but I meant in the context of this post and what the OP is presenting.
[deleted]
(comment deleted)
contravariant
What does 99.999999999% durability mean exactly? Does it mean a probability of 0.000000001% (1 in 100_000_000_000) that your bits will randomly disappear? Is that yearly?

One interpretation is that about 1 bit per 100 GB will randomly flip each year. That or S3 Glacier is expecting to hit a catastrophic event every 100 billion years (which doesn't seem nearly frequent enough).

mrich
This explains it pretty well:

https://blog.synology.com/data-durability

> Data durability: the ability to keep the stored data consistent, intact without the influence of bit rot, drive failures, or any form of corruption. 99.999999999% (11 nines) durability means that if you store 10 million objects, then you expect to lose an object of your data every 10,000 years.

wongarsu
Doesn't AWS only spread your data over three availability zones, which are three data centers in the same city, often located in close proximity of each other? I'd rate the possibility of Amsterdam suffering a catastrophic flood or nuclear strike that wipes out three data centers at once at more frequently than once every 10,000 years.
contravariant
I mean we're already pretty certain that their current design isn't enough to last when the sun explodes so that puts a hard limit of all data disappearing once every 10 billion years or so.
gw99
Think we'll have an AWS region in Alpha Centauri by then...

Edit: and it'll still break when us-east-1 is destroyed by the sun exploding.

Aachen
"an object"

Shouldn't that be multiplied by the number of bytes of this object? I still don't quite understand the math apparently because then we'd all be losing 10KB+ files yearly.

mbreese
I don’t think so. The question is: for any given file, what is the likelihood that if I restore it, there will be some corruption? (Total loss, bitrot, anything)

For the purposes of this calculation, you would probably use the average size of a stored file, determine the number of chunks that was split into, and then the likelihood of losing/corrupting a chunk.

By looking at the durability of objects, you (a) reinforce the object-store nature of S3, and (b) avoid reporting “durability per kb”, which isn’t linear, and more difficult to interpret.

Honestly, the durability number is so high, that adding the bytes as a unit would just make it difficult to reason about. And S3 doesn’t store “bytes” — it stores objects (made of bytes). So, that’s the metric they chose to focus on.

xani_
If we assume for the sake of argument simple RAID 6 setup, to lose say a block of data you'd need to

* have a bad block on one drive * have a bad block in *EXACTLY SAME LOCATION* on the second drive * have a bad block in *EXACTLY SAME LOCATION* on the third drive * all of that happening within one scrub period (which is usually once a month).

And that's just simple RAID6 setup without external redundancy.

As you can see it would be far easier to lose data due to just correlated drive failures, than from random block corruption.

wongarsu
Also note how Amazon doesn't claim that the service has 11 nines of data durability, but that it is designed for 11 nines of durability. That's not quite the same.

Backblaze apparently gets the same number by looking at the failure rate of their drives, and how long it takes to recover from a drive failure (reaching the same redundancy as before the failure). By being able to recover from three drive failures before the first drive is rebuilt, they get 11 nines of durability [1]. Or in other words: 0.000000001% of drive failures are non-recoverable and lose data. I assume AWS does basically the same calculation, adjusted for whatever technology they use.

1: https://www.theregister.com/2018/07/19/data_durability_state...

sliken
Not sure exactly. But Backblaze has been pretty transparent on their approach.

Backblaze takes user data, splits it into 17 pieces, add 3 pieces of redundancy, and split it across 20 racks. Backblaze has a tiered response system, my memory is something along the lines of 1 of 20 disks dying = schedule for replacement the next business day. The 2nd disk triggers a quicker/higher priority response. 3rd disk triggers pages and all hands on deck.

Given that disks have a annual failure rate on the order of 1%, and losing the first disk has a chance of 1% * 1/365, if a second disk is 4 hours that 1% * 1/(3656), and a 3rd disk is 1%/(36524) you end up with lots of zeros:

0.01(1/365) 0.01 * 1/(3656) 0.01 * 1/(365*24) .00000000000000014281

That's just to remove the redundancy, you still need to lose another disk before data is lost.

Sure there's other failure methods, including those that could take out an entire data center. I did find a statement saying that data stored in the US-West region would be stored in both Sacramento and Phoenix, but not sure if that means each B2 object uploaded would be in both.

knorker
I've moved all my stuff off of glacier because of their ridiculous pricing model, and user-hostile metadata handling.

I.e. you have to maintain your own index of files, where they could have just done this for you.

The pricing model for downloads is too easy to shoot yourself in the foot. I'd rather pay a tiny bit more to not have bankruptsy traps built into the product. So that's what I do now.

yuz
Can you share alternatives you found, if any?
thrtythreeforty
Glacier is a different product than S3 Glacier. Plain Glacier is indeed hard to use, but S3's Glacier storage tier makes it all pretty easy to use.
imhoguy
What I would really like to have with AWS is some kind of prepaid scheme, first to avoid surprise costs at early stage, second to fill up enough to keep resources running in case I get into coma or my digital identity gets disrupted by some glitch. I accept that if I won't topup on time then resources are gone.
laundermaf
> AWS is some kind of prepaid scheme

I heard from a friend who chose AWS to handle backups specifically because he prepaid for them, that presumably is already possible.

sreitshamer
I think maybe you're referring to the original Glacier vault model? Nowadays you can put objects in an S3 bucket (with your own object name and metadata) and set the storage class to Glacier Flexible Retrieval or Glacier Deep Archive. Then you can use the S3 APIs to get object lists etc. Also, they got rid of the insane "peak hourly request fee" stuff when downloading. https://www.arqbackup.com/blog/amazon-glacier-pricing-change...
capableweb
> 99.999999999% data durability, data spread over 3 Availability Zones

I'd love to see what source(s) this claim has in practice. How did you arrive at 9-9s? Shouldn't it be spread over 3 Regions rather than Availability Zones, as otherwise it could just end up in the same geographical region while giving the impression it's spread across many.

nik736
Also there is afaik no SLA for this! Just marketing.
harshaw
this is all over AWS documentation; you get the durability via erasure encoding / replication over multiple AZs in one region. Regional blast radius protects again one AZ going down. You don't run a service like S3 across AZs in different regions because of speed of light. if you are paranoid about a region going down you use replication.
loloquwowndueo
I use tarsnap for off-site backups :

https://www.tarsnap.com/

It’s probably not as cheap as glacier but it’s cheap enough for my needs, secure and encrypted and was very easy to set up.

IgorPartola
It’s definitely not as cheap: $1 buys you 4GB/month.
perihelions
I use rsync.net's borg offering, which is similar (encrypted + deduplicated) but 17x cheaper (1.5 cents vs. 25 cents per GB-month, and also zero bandwidth fees). I think it's even less with the HN discount; I'm too lazy to check.

https://www.rsync.net/products/borg.html

To be completely fair: since Tarsnap doesn't have a minimum order size, it's still a better value in the 0-6 GiB range.

quickthrower2
Edit: was wrong.

Original:

0-40 GiB, due to the minimum and assuming value means "$ to store all my X GiB" as opposed to "$/GiB"

bloopernova
I pay $150/year for BorgBase backup and store 1.66TB, mostly because I really like Borg backup and am comfortable with it. That's my last-tier backup. I highly recommend BorgBase, they're good folks.

I also pay $120/year for Google drive which is my "online" backup, then store the files locally of course.

Seems to work OK for me, and I'm insulated against the problem of: "google's algorithm decided I was a Bad Person and terminated my accounts".

londons_explore
> 3 or more AWS Availability Zones, providing 99.999999999% data durability

I think you need to multiply that by the durability of amazon as a company...

I suspect that in any given year there is perhaps a 1% chance that they shut down AWS with no chance to retrieve data. (either due to world war, civil war in the usa, change in laws, banning you as a customer, change in company policy, bankrupcy, etc.)

blahgeek
> in any given year there is perhaps a 1% chance that they shut down AWS with no chance to retrieve data

That would translate to 64% chance in 100 years. Seems plausible :)

londons_explore
I think it has already happened to customers based in Iran last time the USA changed their stance? I don't think those customers got any notice to download data.
weird-eye-issue
The distribution of probabilities over a 100 year period won't be uniform
pydry
Usually when that happens you are given a warning and a window (often of many months) when you can still access your data.

And, most likely you still have your original data. P(Data loss) = P(Bankruptcy happens dec 2022) x P(drive failure happens dec 2022).

The likelihood of the company going down one day, without notice, and, coincidentally, your regular data store getting trashed the next day is extraordinarily low. Absent some kind of end of the world scenario the two events are, for all intents and purposes, independent and low probability.

If you're planning for that scenario you've probably slipped over the line from sensible backupper to prepper.

quickthrower2
In that scenario, your local copy is the backup!
electroly
"Banning you as a customer" is like a billion times more likely than any of those other things. That's the only real thing to be concerned about in that list. AWS is too big to fail; if it came to it, I bet the US Government would bail them out. But they can ban you at any time; that's a serious concern of mine and the main reason why I mirror all of my AWS data to Backblaze B2.
dewey
This service already exists for a very long time. I have been using it for many years with https://www.arqbackup.com as a fallback for my TimeMachine and Backblaze backup. Google also offers a very similar service as Glacier called Coldline: https://cloud.google.com/storage/docs/storage-classes#coldli...
mrich
Arq does not seem to be in the same ballpark as Deep Archive with regards to pricing. Also I needed a Linux solution, they support Win/Mac only unfortunately (same as Backblaze).
sreitshamer
I'm confused by this comment. Arq can store your backups in your own AWS account, using Glacier Deep Archive storage class. It's your own AWS account, so you pay exactly the Deep Archive price.
mrich
I understand, didn't see this mentioned on the pricing page directly. How do they store the data in Deep Glacier (single files or archives? the former would be very expensive?) Do they have restore fully integrated? (since this is a two-step process that takes 12+ hours I could imagine it does not fit the normal flow)
dewey
Arq is a GUI client that stores your backups in the location you set it up for. See screenshot in the "Back up to your own cloud account." section on https://www.arqbackup.com.
thrtythreeforty
With a little setup (no more than described here), rclone just... does this out of the box.

Specifically I have an S3 remote configured to use the Deep Archive tier. On top of that I have an encryption remote (pointing to the S3 remote). Then I just rclone my pool to this remote, and all my crap is shipped off to Ireland.

Like in the link, I expect never to need it; restore is so expensive that it's a "house burns down" insurance only.

mrich
Thanks, sounds interesting, will take a look. Do they store files 1:1 or do they create tar archives? Single files are quite ineffective on Deep Glacier.
weird-eye-issue
I use rsync.net as a secondary backup and it's especially handy because you can easily configure it to keep N snapshots for every X days/weeks/months/quarters/years
mrich
I found two problems with rclone, would be interested to hear if I misunderstood anything or if you have a workaround:

- It will not create archives by default, so potentially upload a lot of files (just like rsync). There are a few workarounds in the documentation to mitigate the cost.

- It cannot do restore of Deep Archive data

mrich
Edit: Can do restore, but it's a manual step.
blahgeek
I’m curious, has anyone really experienced any data loss on public storage service like s3? I’m not sure if the count of 9s actually matter …
tatersolid
We experienced a few dozen unrecoverable files on Rackspace Cloud Files (which at the time used Openstack under the hood). This is out of about 10M objects that totaled about 2 TB, but they were less than 3 years old. The API call just failed with a 404 even though the files were listed in the list API call.

The corruption was uncovered when we started migrating to another cloud platform, we had to restore from local copies.

hampereddustbin
This often disregards the cost of retrieving said data, which is at $90/TB for outboud network traffic on top of the costs of making the backups available.
tokamak-teapot
This doesn’t disregard that. It lays it out with some nice numbers so that it’s clear. It’s much appreciated.

I’m very happy to pay USD 200 if I ever need to retrieve a last chance backup. If I just want the data in no rush I can trickle it back over 20 months for free.

hampereddustbin
Outbound traffic costs always cost the same regardless of your retrieval speed to API. If you only have a couple TBs worth of high-value data, this may be acceptable to you. It's far from being a universal truth though.
mrich
But you get 100 GiB/month free download :)
hampereddustbin
Downloading backups for a decade is untenable
Aachen
100 GiB per month for a decade... how many terabytes of irreplaceable pictures have you got?

But, yeah, making use of a free service loophole, it taking ten months for a terabyte, and presuming you're using zero other Amazon services that count towards your free trial amount, I agree with you it's a stretch. But if you're really cheap or hate Amazon with a passion (but not enough not to host with them) it's not that far fetched.

traceroute66
Aside from the general concept issues that others are addressing (e.g. "what does 99.999999999% durability mean exactly").

There's also code smell.

I had a quick glance through and couldn't help noticing the stench of assumptions and poor (or non-existent) exception handling.

mrich
Definitely not enterprise-ready code but I'd say it's solid. Please point out some specifics.
Markoff
Why not mention this straight in your post:

"Restore and download is quite costly:

Restore from S3 tape to S3 blob: $0.0025/GiB ($2.56/TiB) for Bulk within 48 hours $0.02/GiB ($20.48/TiB) for Standard within 12 hours Download: The first 100 GiB/month are free, then 10 TiB/Month for $0.09 per GiB ($92.16/TiB) and discounts for more."

TLDR if I read it correctly 2.56+92.16 USD to get your 1TB back home

Not that bad, but I feel like buying every half year 1TB drive for ~50USD and just storing it wherever outside your home would be cheaper option. But it depends how often you need to perform backup.

mnw21cam
Or if you're willing to dribble the data back home at a rate of 100GiB/month then it's $2.56/TiB.
laundermaf
Do you really want to spend 10 months retrieving that data? I don’t think there's a single real-life scenario where you do need data, just not this year. So $2.56/TiB is unattainable.
quickthrower2
Also, the issue with waiting 10 months will be you now only have one copy - the one on S3 and you may lose that data. Not because of the 99.9 recurring % guarantee of course, but another fubar issue - like someone hacking into your account - or some fubar with the backup script - and now you have no backup of the backup.

If you need the last resort backup, then you would probably want to revert to normal operations ASAP where you have the original and the first resort backup back in place!

Ekaros
It is not that bad price, when home and place of your offline backup burned down and google or whoever locked your account.
Crontab
Glacier is nice, but due to the cost of data retrival, people should use it as the restore of last resort.
lathiat
I use rclone and Backblaze B2 for this. B2 used to be cheaper though not sure if that’s true with the new deep archive but it is much less fiddly and no crazy fees at restore time.

Rclone is also multi threaded so goes much faster compared to rsync

jwr
Here's my "me too" — I've been happily using rclone for things like photo archives (together with my small consistency checker to check file hashes for corruption https://github.com/jwr/ccheck). I also use Arq Backup with B2 as the destination. This gives me very reasonable storage costs and backups I can access and test regularly.
369548684892826
"me too" - rclone with Azure Archive tier for last-resort backups. I like that rclone doesn't tie me to one storage provider, and I'm not sure why anyone would use the linked project over rclone.
mrich
As mentioned in another reply below, I found these issues with rclone, perhaps you can tell if I'm missing something:

- It will not create archives by default, so potentially upload a lot of files (just like rsync). There are a few workarounds in the documentation to mitigate the cost.

- It cannot do restore of Deep Archive data

Edit: Can do restore, but it's a manual step.

369548684892826
Sorry, I don't know about that, it just does what I want out of the box.
lathiat
With backblaze B2 both of those are not an issue. So not sure.
willis936
I've been doing that via truenas scale for the past year. The upload job hangs then fails. There must be a bad permission or filename somewhere, but I don't know what the right tool is for debugging this so I've just put it off. If anything's missing it can't be large (judging from the space taken on B2).
kbumsik
S3 Deep Archive is two time cheaper than B2. BUT you are not supposed to just rclone to S3 Deep Archive because it charges you based on the number of files you upload.

The upload fee (PUT request fee) is $0.065/1000files so it will charge you a lot when you have millions of files.

oneplane
Yep. Best to tar backup chunks as a whole before sending to DA.
WirelessGigabit
B2 is $5 per TB per month, versus S3 Deep Archive ~$1 per TB per month.

So if hot access isn't a concern then S3 can be 1/5th of the cost.

croes
The question is what are the costs when you need to retrieve the data.

They prices might be higher next year.

agurk
I've created similar functionality just using a simple bash script that will send the latest version of ZFS datasets to S3/Glacier, including dealing with incremental changes. I have mentioned this previously on HN and got a few useful changes submitted for it, especially making it more platform agnostic.

I have some open tickets asking about (script based) restoring. I haven't tried this yet as this has been a backup of last resort for me, but hopefully posting this again will nudge me into looking at that.

https://github.com/agurk/zfs-to-aws/

bzxcvbn
> I'm not aware of any automated backup solution for Deep Archive.

What about the well-known rclone? https://rclone.org/s3/

dosshell
I would also like to mention restic where rclone can act as a backend but you get a very nice frontend for backup
NBJack
Note that, in the event of retrieval, it's not just per GB, it's also per request.

Synology has offered Amazon Glacier and S3 as a destination option with Hyper Backup for years as part of their NAS offerings. Given the available automatic archive feature to move an existing store to Glacier Deep Archive, budget permitting I'd recommend a NAS over this for three reasons:

- Initial setup costs aside, the power draw of a two bay unit like the DS218 (15W at load) would be ~$16/year at peak usage assuming a cost of $0.12/W

- Uploading/syncing your local files to your NAS should be considerably faster, technically 'free', and can be done more frequently as you desire; should you need them, it would also be 'free' to retrieve them locally barring a catastrophic event

- The remote push of the NAS contents to S3/Glacier storage can be done asynchronously of your PC's state (and, to save money, less frequent if you wish), which as you point out could take days; additionally, you can save money given you can reduce the number of requests via automatic archiving/compression

Given how unlikely it is for you to retrieve data from Glacier Deep Archive with such a setup, I highly recommend it. You can still rest knowing your data is offsite.

hampereddustbin
Backups should always be tested atleast once to ensure that they do in fact work. Nothing worse than losing data, and discovering that you've been paying a service for naught all those years. This could get prohibitively expensive if you can't replicate your setup in the cloud to test backups within AWS network.

Or you could just blindly trust that it'll work. Kind of like schrödingers backup

willis936
You could make a backup of one text file. If you can restore that then you can likely restore something 1E9 larger.
hampereddustbin
Could be, but it could also be a fluke. Perhaps everything after the first file-index file is corrupted, and you only restored things from the first one, or perhaps maybe the entire diff-index can be so large that it's not actually recoverable once you grow it to the size you want.

There are a myriad of easy-to-miss failure scenarios that don't appear in small-scale tests. Only way to ensure you can restore your backups, is to restore your backups

mrich
> Note that, in the event of retrieval, it's not just per GB, it's also per request.

This should fall in the free tier in most cases, since the script creates archives of the specified size. The free tier has a few thousand requests for free.

onychomys
> Synology has offered Amazon Glacier and S3 as a destination option

For the record, QNAP does too, I've got an automated job that backs up any new files it sees every Sunday morning.

mattbillenstein
Hmm, I just ship the files directly to Glacier using the aws cli -- aws s3 sync /foo/bar s3://<bucket>/bar/
mrich
Wasn't aware of this :) The problem is that this will sync files as-is without creating archives. Not sure how it's implemented internally, if they're using a single request per file this would get expensive quickly. In any case it will be much more expensive since each file costs a small extra amount for storage. Also curious whether they can handle auto restore during sync cloud -> local. Will give it a try.
mrich
Checked it, it uploads single files. This is very expensive. Also, they only have server-side encryption. Also cannot handle restore :(
mattbillenstein
Yeah, I wasn't doing an entire disk backup or anything - some hundreds or thousands of largish files.
kosolam
Cheap archive. But very expensive data out from aws. omg
dspillett
This is part of why my main backups aren't on something like that plan. I like to checksum them fully occasionally, no matter what assurances the provider gives about them protecting from bit-rot, and that could get quite expensive.
FounderBurr
Put all the data you want in, pay ungodly amounts to get it out tho.
teeray
Yeah, it’s definitely a backup of last resort for that reason. A bit like the Roach Motel: data checks in, but it doesn’t check out.
andag
https://github.com/andaag/zfs-to-glacier

I built something similar a while back that I've been using for years now.

Something worth noting. There is a minimum cost to files. If you have tons of tiny kb sized files (incremental snapshots..) it's drastically cheaper to fallback to s3 for them.

resoluteteeth
Can't you also just use existing tools to backup to s3 and then move it to deep archive?
hampereddustbin
You can, though this will incur some costs related to temporarily storing your data in S3, as it either won't be immediately transferred when using lifecycle policies, or will incur extra charges if you force a type change based on filesizes and filecount.

Note that there is also a minimum filesize of 128KB per object in Glacier as well as 32KB extra metadata, and everything smaller will also be counted as the minimum size. Some mitigate this by bundling files in larger chunks, at the cost of retrievability and now having to keep an map of file-bundle associations.