My guess is they don't realise. AWS let's you setup lifecycle rules to archive S3 objects to Glacier automatically. It doesn't explicitly say it is cheaper (it says "this may reduce your costs"), but they probably just saw the headline prices for Glacier and moved everything.
If you write a AWS-native service that uses S3 as a back end (a very reasonable thing to do), it may create many small files in S3. It would be unreasonable to incur the costs to retrieve them and tar them up, and it also does not make cents to "click the box" to have them archived to glacier.
No additional metadata, but you get charged at least for 128 KB . Luckily lifecycle transition doesn't move files smaller than 128 KB. However, even using lifecycle transition you pay $0.01 per 1000 transitions. It doesn't seem much but for smaller items it can decrease savings a lot. E.g. if you average file is 6 MB, than you will loose 4 days of savings on S3 IA comparing to standard class.
I primarily wanted to note that I used 2^X notation in the math, in case it was confusing for folks. I used KB/MB/GB to be consistent with the language AWS uses on their S3 pricing page (https://aws.amazon.com/s3/pricing/).
Personally, I'm looking forward to Backblaze's cloud storage, which is being advertised at $0.005/GB/month. Cheaper than glacier, without all of the transaction fees.
This is one of the things I love about GCS Nearline, it really is a penny per byte and then the whole retrieval charges (which for my lazy rsync backup is never).
Disclaimer: I work at Google on Compute Engine, but not on GCS.
It is useful for keeping archives of massive data you're unlikely to ever need, but legally obliged to keep around, or just want to have available for a very improbable later examination. Think some huge transaction logs of two years back.
For a case like this, you don't need fast retrieval, and mostly you don't need retrieval at all. You plan ahead to only ever retrieve a small percent of these data. The rest will be silently discarded when retention period has expired.
If your use case is not like that, Glacier probably makes little sense for you.
This is totally not a backup which you likely keep in order to restore the entire state from as soon as possible.
16 comments
[ 1.8 ms ] story [ 73.0 ms ] threadTar everything and send it up. And then do an incremental backup
Amazon charges per operation, reduce the number of operations
No additional metadata, but you get charged at least for 128 KB . Luckily lifecycle transition doesn't move files smaller than 128 KB. However, even using lifecycle transition you pay $0.01 per 1000 transitions. It doesn't seem much but for smaller items it can decrease savings a lot. E.g. if you average file is 6 MB, than you will loose 4 days of savings on S3 IA comparing to standard class.
> KB: 1,024 bytes, expressed as 2^10
Why not just use KiB? Unlike KB, it's unambiguously binary.
https://www.backblaze.com/b2/cloud-storage.html
Disclaimer: I work at Google on Compute Engine, but not on GCS.
It is useful for keeping archives of massive data you're unlikely to ever need, but legally obliged to keep around, or just want to have available for a very improbable later examination. Think some huge transaction logs of two years back.
For a case like this, you don't need fast retrieval, and mostly you don't need retrieval at all. You plan ahead to only ever retrieve a small percent of these data. The rest will be silently discarded when retention period has expired.
If your use case is not like that, Glacier probably makes little sense for you.
This is totally not a backup which you likely keep in order to restore the entire state from as soon as possible.