Kinda: "In the coming months, Amazon S3 will introduce an option that will allow customers to seamlessly move data between Amazon S3 and Amazon Glacier based on data lifecycle policies."
This is fantastic. I've long searched for a solution like that. This is really suitable for a remote backup that only needs to be accessed if something really bad happens (i.e. a fire breaking out, etc). I'm a lone entrepreneur, so I do have backup hard disks here, but being able to additionally save this data in the cloud is great.
I'm often creating pretty big media assets, so Dropbox doesn't necessarily offer enough space or is - for me - too expensive in the 500gb version (i.e. $50 a month).
Glacier would be $10 a month for 1 terabyte. Fantastic.
Agreed - I'm really happy about this. I have a home NAS solution that is a few TB and it's too expensive to store on S3. This is perfect to prevent the "house burned down" scenario on very large storage devices!
I'm trying to work out the best way to back-up my NAS to the cloud just in case my house burns down. It's really annoying how lots of cloud storage solutions dont cater for home users with a NAS. I agree that this could be a good, cheap way of backing stuff up.
I recently heard about a startup (spacemonkey) that will be offering 1TB with redundancy and no access delay for $10/month. The way they do it is really clever as well. Cloud storage has always seemed way too expensive to me, but these lower prices have me re-evaluating that.
The only issue I see is that verifying archive integrity (you don't want to find out the archive was bad after you lost the local backup...) would be somewhat complicated, given their retrieval policies. Also, the billing for data-transfer out plus peak retrievals sounds so convoluted, I can't begin to work out what a regular test-restore procedure would cost me. Nevertheless, it's some exciting progress in remote storage!
They could provide salted hash verification: send some salt, get a list of files with SHA1(salt | filedata) via email some hours later (so they can do the verification as a low priority job).
The salt is used to prevent amazon from just keeping the hashes around to report that all is well.
To avoid abuse, restrict the number of free verification requests per month.
Or just build the verification into the storage system, and send a SNS message if data loss has occurred (just like what happens when a Reduced Redundancy object has been lost in S3).
> Glacier would be $10 a month for 1 terabyte. Fantastic.
+ the $120 or so per TB to transfer it outside of AWS if you need the whole thing back as fast as possible. Still likely to be very cheap as long as you treat it as a disaster recovery backup, though. Will definitively consider it.
(an alternative for you is a service like Crashplan, which also allows you very easy access to past file revisions via a java app and can be very cheap and also allow "peer to peer" backups with your friends/family; the downside with Crashplan is that it can be slow to complete a full initial backup to their servers or to get fully backed up again if you move large chunks of data around)
I had a quick skim through the marketing stuff and the FAQs and didn't see anywhere that actually details what the backend of this is. I'd be curious if they're actually using tape, older machines, Backblaze pods, etc. I guess if it's the latter, the time to recover could be an artificial barrier to prevent people from getting cute.
Someone further up mentioned a very plausible (in my experience) answer. Magnetic tape, using hard drive arrays as RAM. The wait time in this situation would be the time needed to complete all the current tasks waiting to be written/read in the queue before your data is written from tape to hard drive so you can access it.
It appears to use S3 as its basic backend. My guess is that S3 has been modified to have "zones" of data storage that can be allocated for Glacier. Once these zones have been filled with data (and of course that data is replicated to another region) the hard drives are spun down and essentially turned off.
This is why the cost of retrieval is so high: every time they need to pull data the drives need to be spun back up (including drives holding data for people other than you), accessed, pulled from, then spun back down and put to sleep. Doing this frequently will put more wear and tear on the components and cost Amazon money in power utilization.
As is Glacier should be extremely cheap for AWS to operate, regardless of the total amount of data stored with it. Beyond the initial cost of purchasing hard drives, installing, and configuring them the usual ongoing maintenance and power requirements go away.
That'd make a ton of sense for forensic and compliance needs: lots of storage, limited access in special situations where a little delay is reasonable.
Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months.
So I guess that means it would still work well for a scheme like time machine uses, where incremental changes are added but deletions are simply made note of. At least I think that's how it works.
It will be interesting to know if they will upgrade AWS Storage Gateway to use this kind of backend instead of S3
http://aws.amazon.com/storagegateway/
Which of course means that (if they're telling the truth) the probability of losing your data mostly comes from really big events: collapse of civilization, global thermonuclear war, Amazon being bought by some entity that just wants to melt its servers down for scrap, etc. (Whose probability is clearly a lot more than 10^-11 per year; the big bang was only on the order of 10^10 years ago.)
There's some clever wordplay/marketing here... "designed to provide 99.99..99%" means that the theoretical model of the system tells you that you lose 1 in X files per year when everything is working as modeled (e.g. "disks fail at expected rate as independent random variables"). If something not in the model goes wrong (e.g. power goes out, a bug in S3 code), data can be lost above and beyond this "designed" percentage. The actual probability of data loss is therefore much, much higher than this theoretical percentage.
A more comical way to look at it: The percentage is actually AWS saying "to keep costs low, we plan to lose this many files per year"; when we screw up and things don't go quite to plan, we lose a _lot_ more.
per object. So although the chance of losing any particular object is tiny, the chance of you losing something is proportional† to the number of objects. Still extremely small.
Yes. Though I bet the real lossage probabilities are dominated by failure events that take out a substantial fraction of all the objects there are, and that happen a lot more often than once per 10^11 years.
Agreed. More likely a catastrophic and significant loss for a small number of customers rather than a fraction of a percentage of loss for a large number.
Similar deal for hard drive bit error rates, where the quoted average BER may not necessarily accurately represent what can happen in the real world. For example, an unrecoverable read error loses 4096 bits (512 byte sectors) or 32768 bits (4k sectors) all at once, rather than individual bits randomly flipped over a long period.
Amazon S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years....
Storage experts: I'd love to know more about what might be backing this service.
What kind of system has Amazon most likely built that takes 3-4 hours to perform retrieval? What are some examples of similar systems, and where are they installed?
Could be for example some tape robot where you can have huge amounts of tapes in the storage but only have few devices for reading/writing them. With tape you can't really stream the data to the web. Instead you would probably first copy it somewhere. If there is lots of data, say few terabytes, even this process takes some time.
Or in case they are using regular hard drives, you might want to have this kind of time limits in order to pool requests going to a specific set of drives. This would enable them to power down the drives for longer periods of time.
The 3-4 hour estimate may also be artificial. Even if you can in most cases retrieve the data faster, it would be good to give an estimate you can always meet. They might also want to differentiate this more clearly from standard S3.
And we should not forget that it does take time to transfer for example one terabyte of data over network.
There'll be a near-line HDD array. This is for the recent content and content they profile as being common-access.
Then there'll be a robotic tape library. Any restore request will go in a queue annd when an arm-tapedrive becomes free they'll seek to the data and read it into the HDD array.
Waiting for a slot with the robot arm - tape drive is what will take 4 hours.
Wouldn't there also need to be a lot of logic to prevent fragmentation? You'd probably want data from one user near other data from that user, i.e. on the same tape.
The multiple-hour window could give you a lot of wiggle room here though. It's unlikely to take 3 hours to restore from a single tape, so even if you have to visit 2-3 tapes then you have plenty of time.
I'm sure that there is a general tiered storage platform (as mentioned above) which keeps some of the data online as well. That would let you run a "defrag" algorithm later if you find you need it.
I'd guess that they ignore that problem and have baked the time it takes to get data from several tapes into the 3-4 hour estimate.
If you think about it, writes are more common than reads on average, so it's more efficient to just write to whatever tape is online and deal with the fragmentation problem on the read end, as opposed to queueing writes until the 'correct' tape can be brought online just save some time reading. Also in backup situations like this, it's more important to get the backup done in a timely manner.
Close. Tiered, yes. But remember who we're talking about.
First, no tape. The areal storage density of tape is lower than hard disks. Too many moving parts involved. Too hard to perform integrity checks on in a scalable, automated fashion without impacting incoming work.
Second, in order to claim the durability that they do (99.999999999%), that means every spot along the pipe needs to meet those requirements. That means the "near-line HDD array" for warm, incoming data needs to meet those requirements. Additionally, if the customer has specified that the data be encrypted, it needs to be encrypted during this staging period as well. It also needs to be able to scale to tens if not hundreds of thousands of concurrent requests per second (though, for something like Glacier, this might be overkill).
They've already built something that does all that. It's called S3. The upload operations likely proxy to S3 internally (with a bit of magic), and use that as staging space.
Take a look at Linear Tape File System (LTFS), which allows for ad-hoc file retrieval. CrossRoads Systems out of Austin has the leading implementation, and they did bid on this Amazon contract. I have no idea if they won it, though (next corp conference call is Aug 29th). They just closed a joint investment with Iron Mountain, so my money is on an LTFS solution with CrossRoads as a systems vendor.
> Paying $12 to store a gigabyte of data for 100 years seems like a pretty intriguing deal as we emerge from an era of bit rot.
As long as that data is decode-able and more importantly, find-able (out of all the GBs frozen for 100 years, why would you want to look at any particular one of them?).
To be fair, if improvements in hardware and software continue at the rate they have been, or some moderate percentage thereof, in 100 years it will be no problem to trawl a few exabytes of data for anything interesting.
There's a blog that's analyzing Geocities, that's about 1 terabyte of 1 KB files. http://contemporary-home-computing.org/1tb/ The analysis tracks changes in template design, follows modifications to logos and gifs, and unearths collections of shrines to dead children etc.
But that's from when it was harder to make and upload data, so people only put meaningful (to them) stuff online. These days we'd have a hundred thousand copies of a few popular MP3's and everyone's crappy digital photos. The percentage of meaningful stuff would be a lot lower.
As long as that data is decode-able and more importantly, find-able (out of all the GBs frozen for 100 years, why would you want to look at any particular one of them?)
I'd store my pictures there. Finding old pictures of grandparents when they were little, or even older stuff, is amazing. Wouldn't it be cool if my descendants could still look at pictures of my family in 100 years?
Provided that downloading from this 100 year store is something I could do X times per year, and so long as I could append more data to it over time, it's an interesting business model.
"Paying $12 to store a gigabyte of data for 100 years"
I'm not sure what kind of organisation I'd actually trust to store data for that length of time - a commercial organisation is probably going to be more effective at providing service but what commercial organisation would you trust to provide consistent service for 100 years? A Swiss bank perhaps? Governments of stable countries are obviously capable of this (clearly they store data for much longer times) but aren't set up to provide customer service.
Yeah, I'm kind of wondering the same thing. It's certainly the kind of timeframe that changes your perspective. Maybe a tiny bit of Danny Hillis rubbed off on me from working at Applied Minds (man, I sure hope so!)
Because as we answer issues of cost and availability, a logical thing to wonder is "how long can I really depend on it though?" As quickly as cloud services (where "lifetimes" are measured at six years) have entered our economy, that's a question begging to be answered.
Amazon at least seems to be an "eventually durable" datastore, though. Meaning that if you are told in the future that it will go offline, you have an excellent chance to make other suitable arrangements. Say there's a 0.01% chance of this product being discontinued next year, up to a 10% chance 5 years from now. I have to think there will almost certainly be other services you can move your data to, on similar terms, for a long time.
That's assuming you're around at all, and nothing reeeeeally bad happens even so. Making data last after your death (or even after you stop paying!) is a lot harder in this environment, and achieving true 100-year durability is a tough nut indeed.
I like your bank idea, since the preservation of a bank account is just a specialized simple case of data preservation. Data preservation seems rather more reliable when the data is directly attached to money. Then again, maybe banks themselves are on their way out for this purpose — Dropbox could become the new safety-deposit box.
But there are 100-year domain registrations, after all. Maybe we're ready for organizations to at least offer 100-year storage, too.
And, since you mentioned Danny Hills, it's also worth mentioning that (ironically?) Jeff Bezos is one of the principle supporters of http://longnow.org/clock/.
And that makes me think of Anathem and the potential issues around long-term data storage that is capable of surviving through falls of civilisations and/or sacks of storage areas.
Probably what would be required is an array of arrays of separate storage providers and services providing "RAID" on top of these storage providers - and you won't want to trust any of these you'll want a few of them... (hence the array of arrays).
leastauthority.com (the Tahoe LAFS folks) are trying to promote the concept of "RAIC" (C=Cloud). I'm not sure what the status of the project is, though...
FWIW, we are happy to support this kind of use, and see our customers doing it in an ad-hoc way every day. We have s3cmd in our environment, and support it. As soon as their is a glacier complement to s3cmd we will put that into place as well, although with the strange traffic and retrieval pricing, I'm not sure how useful folks will find it ...
> Governments of stable countries are obviously capable of this
I don't consider that obvious. I live in Berlin, the capital of what most would consider a stable country, but my apartment (which is even older) has been a part of 5 different countries in the last 100 years (German Empire, Weinmar Republic, Nazi Germany, East Germany and finally, the Federal Republic of Germany).
Sorry, what I meant by "stable" there is countries that have been relatively stable for a few hundred years and seem reasonably likely to continue that integrity for at least a century or so.
Of course, predicting future stability is complete guesswork!
We are straying a bit off topic here. I don't think any country has been stable for a "few" hundred years. A few is 3. This is before the US was founded.
I would say a stable country is one which has had a legit democracy for 70 or so years and doesn't share a border with a non-democratic / non-legit-democratic state. These two points suggests its unlikely to have a revolution or be invaded any time soon.
Of course, if you look at the UK governments track record with IT.. you wouldn't trust it. I would say the same with the US, especially in concern with data security.
I think I could argue that England has been pretty stable since at least the end of the Civil War - which is 360 years ago. A lot of the institutions that form part of the current UK go back an awful lot further than that.
I'll reject your democratic requirement out of hand, as democracies haven't proven particularly stable. There was that trial-run in Athens which lasted 501 years (508 - 7 BCE, with interruptions). Other than a few small/outlying instances (most notably the Althing in Iceland, it didn't re-emerge until the short-lived Corsican Republic (1755), and of course, the United States (1776).
Japan (660 BCE) and China (221 BCE) have both had feudal / bureaucratic governments exhibiting very high levels of stability. While dynasties and eras are marked, the overall states persisted largely intact.
Even the US had a close call with a fairly nasty civil war in that time frame, and in three days the 198th anniversary of the Burning of Washington happens: http://en.wikipedia.org/wiki/Burning_of_Washington "On August 24, 1814, after defeating the Americans at the Battle of Bladensburg, a British force led by Major General Robert Ross occupied Washington, D.C. and set fire to many public buildings. The facilities of the U.S. government, including the White House and U.S. Capitol, were largely destroyed."
The Thai king is the longest-reigning current head of state, ascending the throne on 9 June 1946. Elizabeth II of England is 2nd, 6 February 1952.
The oldest country (not government) is likely Vietnam (2897 BCE). Other contenders: Japan (660 BCE), China (221 BCE), Ethiopia (~800 BCE), or Iran (678 BCE).
Few of today's modern states pre-date the 19th Century, many antedate World War II or the great de-colonialisation of the 1960s including much of Africa and Oceana (some of the longest inhabited regions of Earth).
Among the more long-lived institutions are the Catholic Church (traditionally founded by Jesus ~30 AD, emerging as an institutional power in 2nd Century Rome). The oldest company I can find is Kongo Gumi, founded in 578, a Japanese construction firm. The record however is likely held by the Shishi Middle School founded in China between 143 and 141 BCE.
My own suggestion would be the Krell, though some might disqualify this based on a requirement for human organization.
That was my thought as well. However it spent a great deal of time under foreign rule: under the Greeks and Romans, the Turkish / Ottoman empire, and later under British occupation. And, I just discovered what boxer Muhammad Ali's referent was.
No offense, but Germany is far from what I'd call a stable nation. It has, in fact, only been a nation for 141 years. and during that time has, as you've noted changed governments many times.
There are some pretty old commercial organisations out there. One list [1] has several that have existed for over 1000 years. Whether they would be capable of storing data (or would even be interested in doing so) is hard to say, but there have at least demonstrated that long-term organisational continuity is possible, and that presumably requires some organisational 'memory'.
Having a long history obviously isn't a predictor of future stability. According to the Long Now Foundation site [2], a Japanese company that existed since 578CE went bust in 2007.
But, it has not been as stable as you might think. During the 1300s, the popes resided in France.
As recently as 1870, during the Italian unification, the church was stripped of its power to govern Rome after an armed confrontation between armies at the gates of Rome ("XX Septembre"). During this period, the Pope (the ninth Pius) seriously considered fleeing Italy.
The Royal Mint has existed for 1,100 years. That's pretty much the most stable government-owned business-like entity I can find.
The Stora Kopparberg mining company has existed since it was granted a charter from King Magnus IV in 1347.
A few banks tend to last for a long time [1]. Banca Monte dei Paschi di Siena has existed for about 540 years.
Beretta, the italian firearms company, has existed for 486 years (and has been family owned the entire time).
East and West Jersey were owned by a land proprietorship for around 340 years starting from King Charles II bestowing the land to his brother James in 1664. [2]
At first I thought multinational corporations would be more stable because they could move from land to land to avoid wars and such. But apparently they haven't lasted nearly as long as their single-nation counterparts.
The Knights Templar were granted a multi-national tax exemption by Pope Innocent II in 1139, and lasted almost 200 years until most of their leadership was killed off in 1307.
The Dutch East India Trading Company was one of the first [modern] multinational corporations, spanning almost 200 years from 1602-1798.
However, the longest-lasting companies have been family owned and operated. [3] [4]
It appears most all companies that have lasted a long time are due to two factors: dealing in basic goods and services that all humans need, and looking ahead to the future to change with the times.
If you really want the organization to survive for a long term, you need to make it a religion.
If you can figure out a way to convince a few dozen people every decade that the best way to glorify God is to isolate themselves off somewhere maintaining your archival data, you'll be set for centuries.
This seems like a very interesting business idea. It'd require some level of initial operating capital and a relatively competent server farm team, but I don't think it'd have to be fancy.
"Long Data, LLC... We secure your data for the long-term".
If you really wanted to preserve something, I'd consider printing it on gold leaf. One gram of gold turns into about 4 sq ft, and assuming we can print at 1200dpi, we'd have 4 x 144 x 1200 x 1200 bits, or about 100 megabytes. So you'd need about ten grams, at a price of $530 or so, plus storage. Though you could just bury it in your backyard.
I hope they can get the access time down from 3-5 hours to about 1 hour - that's the difference for me between it being a viable alternative for storing backups of my client's web sites or not.
I might create a script that uploads everything to Glacier and just keeps a couple of the latest backups on S3 though.
Per Werner's blog post [1] "in the coming months, Amazon S3 will introduce an option that will allow customers to seamlessly move data between Amazon S3 and Amazon Glacier based on data lifecycle policies."
that's certainly interesting. as there will be migration from s3 to glacier, it would be nice if tarsnap had an option to store only the (say) last week in s3 (with .3$/gb/month) and the rest in glacier (with, say, .03$/gb/month).
that would certainly be very nice. cperciva, what do you think?
I can't see any way for Tarsnap to use this right now. When you create a new archive, you're only uploading new blocks of data; the server has no way of knowing which old blocks of data are being re-used. As a result, storing any significant portion of a user's data in Amazon Glacier would mean that all archive extracts would need to go out to Glacier for data...
Also, with Tarsnap's average block size (~ 64 kB uncompressed, typically ~ 32 kB compressed) the 50 microdollar cost per Glacier RETRIEVAL request means that I'd need to bump the pricing for tarsnap downloads up to about $1.75 / GB just to cover the AWS costs.
I may find a use for Glacier at some point, but it's not something Tarsnap is going to be using in the near future.
While I have no idea how you would fit it in your current infrastructure, I certainly see a (BIG) use-case for: I have this 100 GB, store it somewhere safe (in glacier), I won't need it for the next year (unless my house burns down).
I agree that is a bit different from ongoing daily backups with changes, but its also not THAT different from a customer perspective.
That it doesn't fit with how you store blocks on the backend won't matter to a lot of customers.
i understand. i hoped tarsnap knew what blocks (do not) get reused.
it's unfortunate, because some backups happen to just lie around for very long. it would be nice to take advantage of (the low cost of) glacier for that.
that said, if it's not possible with tarsnap now, it's not possible now. :D. if you find a satisfying possibility to incorporate it in the new backend(s) design (if that's fixable in the backend(s) alone), i'd surely be pleased.
I'm curious about data security. It says it is encrypted with AES. But is it encrypted locally and the encrypted files are transferred? I.e. does Amazon ever see the encryption keys?
Or is the only way to encrypt it yourself, and then transfer it?
AFAIK the keys are managed by Amazon, just like S3. It's more for compliance reasons rather than real security. Encryption still has to be done yourself to protect the data.
I'm a long time user of backblaze, and I'm a big fan of the product - it does a great job of always making sure my working documents are backed up, particularly when I'm traveling overseas, and my laptop is more vulnerable to theft or damage.
With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,vud,vmdk,vmsn,vmsd,hdd,vdi,vmwarevm,nvram,vmx,vmem,iso,dmg,sparseimage,sys,cab,exe,msi,dll,dl_,wim,ost,o,log,m4v" files. They also don't want to backup your /applications, /library, /etc, and so on locations. They also make it clear that backing up a NAS is not the target case for their service.
I can live with that - because, honestly, it's $4/month, and my goal is to keep my working files backed up. System Image backups, I've been using Super Duper to a $50 external hard drive.
Glacier + a product like http://www.haystacksoftware.com/arq/ means I get the benefit of both worlds - Amazon will be fine with me dropping my entire 256 Gigabyte Drive onto Glacier (total cost - $2.56/month) and I get the benefit of off site backup.
The world is about to get a whole lot simpler (and inexpensive) for backups.
What's even more important, you will be able to encrypt your backups without having to disclose the encryption key in case you ever need to restore (client-side encryption and decryption). This is not the case with Backblaze, which is why I switched to CrashPlan — but I'm still looking for other solutions.
Then no, it does not worry me. Obscuring password entry in an application that I almost never run is not a problem. Plus, if I do actually run the app and want to enter the password, I don't do it with a bunch of people peering over my shoulder.
CrashPlan defaults to using the account password as the encryption password.
However, you can also secure the encryption with a password not associated with the account. Or even provide your own 448-bit key. If you do either of these options, CrashPlan support will not be able to help you
This setup allows CrashPlan to easily help non-technical home users, while allowing technically savvy users to securely hang themselves with their own encryption.
My question is not about the setup, that's OK. I am wondering why CrashPlan shows the encryption key in the clear and does not store it in the user or system keychain.
I'm assuming this because their desktop client is a java app and it it's been made to run on linux, mac and Windows. Is there such a thing as a keychain on Windows? I've only used it on linux and mac.
As others have noted, it is partially because Crashplan is a Java based app. Partially, it is also because Crashplan runs as System, not as the user. That way, I can have Crashplan backup my wife's user account and my user account.
Furthermore, you can use the encryption key + a custom password, or your own encryption key with a passphrase. In this case, it is encyrpted locally and the key is not sent to Crashplan[1].
You'd prefer that they lie to you and pretend it's not stored in the clear? If you don't have to type it in every time a backup runs, it's in the clear - everything else is just window dressing.
Right, which is a largely meaningless distinction: if the process runs as you, any successful attacker can simply read it out of the CrashPlan process in memory.
Again, if you're not typing the password in every time, a local compromise is almost certainly game-over. Apple's keychain helps reduce the damage if the data's not actively used but for something like CrashPlan which is always running the attacker is probably going to be lucky.
I agree that showing the encryption key in the clear is not a serious security flaw but it's IMHO against best practice.
Passwords and key are usually shown in an obscured form, usually with asterisks, and stored in the user or system keychain. You are absolutely right that the security value of these standard practices should not be overvalued but still … what else within the security framework of CrashPlan is not done in accordance with best practice?
(I assume, BTW, that CrashPlan does not use the system or user keychain on the Mac because it is not a real Mac citizen but a Java-based app. Firefox and Wuala – the latter Java-based too – don't use the user or system keychain either.)
Those arguments are typically made from the wrong perspective. It's just so common that if you don't do it, your product will be perceived as insecure. And as long as actual security is not horrible, the perception of security is what drives sales, not the actual security.
The other feature that's great about CrashPlan is that it will allow you to backup to other systems running their software. I backup my laptop to both their service and one of my dedicated servers. That way if anything ever happens to their online service I have a second remote copy that I control.
Backblaze does support the optional use of a separate encryption key which they claim to never store, but there's no information about whether it is ever disclosed to them or not. There's little information on their website, but it can be set in Settings > Security in the Backblaze client.
Home-use is probably the only situation Glacier is good for backup though.
A home user is fine with a 3.5-4 hour window before their backup becomes available for download (as it will probably take them days to download it anyway).
In a corporate environment, I don't want to wait around for 3.5-4 hours before my data even becomes available for restore in a disaster recovery situation.
Seems good for archive-only in a corporate environment (as the name implies).
Here, we are producing logs of simulation runs that may be needed in the next couple of years. Most of the logs will be destroyed unused. This is a perfect use case for glacier. We do not care about 5 hours recovery time.
In a corporate environment, I wouldn't want to depend on the cloud as my primary backup solution in the first place. I'd be much more comfortable using it as the offsite mirror of an onsite backup. If you're at a point in disaster recovery where you have to restore from your offsite, you (likely) have bigger problems than a 4-hour wait time.
I personally believe that data should never be deleted (or overwritten), but only appended to. Kinda like what redis/datomic does. So, keep live data onsite, along with an onsite (small) backup, and all the old data in Glacier.
You can believe that, but legal realities dictate otherwise. There are certain classes of information that you are not permitted to keep beyond a defined horizon, either temporal or event-based. Legal compliance with records management processes means having the ability to delete or destroy information such that it cannot be recovered. Note that if the information is encrypted, you can just delete the decryption key and it is effectively deleted.
That's just the beginning - if you've ever been in a ediscovery process, having large amounts of historical data is actually a liability - if instead of 100GB you have 10TB, you'll need to hand that over, and before that, to cull it so you don't inadvertently hand the opposition a huge lever to be used against you. Processing and reviewing 10-100x the data can take inordinate amounts of time more than you expect.
Interesting to note - those of us who used Iron Mountain/Data Safe for years - 4 Hours was considered a "Premier Rapid Recovery" service that we paid a lot of money for (as recently as 2003, actually).
In a true disaster recovery (Building burned down or is otherwise unavailable) - it usually takes most businesses a week or so just to find new office facilities.
But - agreed, there will be some customers for whom Glacier wouldn't work well for all use cases.
Now - a blended S3/Glacier offering might be very attractive.
Yes, that's what I am thinking: a week of backups to S3 and a script that does moves the oldest one from S3 to Glacier. You would rarely need a backup older than a week if you've already got a week of backups. I'm still figuring out if there is a practical way to do this with incremental backups without introducing to much risk that the backup process gets messed up.
"In the coming months, Amazon Simple Storage Service (Amazon S3) plans to introduce an option that will allow you to seamlessly move data between Amazon S3 and Amazon Glacier using data lifecycle policies."
I don't think DR is the correct use-case. I think it more likely to be (as in the case of banks which I know best) a regulator asking for electronic documents or emails relating to a transaction from 6 years ago (seven years is the retention requirement).
In that case, 3-4 hours would be more than acceptable.
You can keep your recent backups in a fast access location for disaster recovery. It would be a good place to keep older backups though that you don't need to access in an emergency.
It's made for archiving, not 'backups'. You're expected to keep most recent backups on-site, so if a RAID array dies or you happen to do something wrong, it just takes minutes to restore. Off-site, or tape archive, it's usually another thing. Most of the data is never, ever accessed (but required by company policy to be kept).
"In a corporate environment, I don't want to wait around for 3.5-4 hours before my data even becomes available for restore in a disaster recovery situation."
I am guessing you work in a company with a good IT department then, I am guessing this is not the average. Many companies I have worked for 4 hours would be a miracle with a 1-3 day operation minimum.
And don't forget that the X00MB type size limits many IT departments puts everywhere it not because getting a TB hard drive is cheap, but because all of the extra backups add to the cost of each new MB. Having another extremely cheap way they could backup large amounts of data (encrypted?) would help to reduce the cost of each extra GB.
In a corporate environment where you have low RTO goals for DR scenarios, relying on backups instead of replicated SAN etc is not a sound practice, especially with large data sets.
With large volumes the real issue is not the storage but the upload speed. I've done some experiments and with the Comcast link (20Mbps down/whatever up), I got 1 Gb/hour upload rate. So, it'll take 11 full days to upload 256 Gb. Or, more realistically, if you do it overnight (8 hours) - the entire month.
"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. —Tanenbaum, Andrew S." - http://en.wikipedia.org/wiki/Sneakernet
Makes me wonder what Amazon is doing to commoditize its complements, i.e. what it's doing to improve high speed internet access. There are a lot of people in my vicinity who have no option faster than a wireless 3Mbps connection capped at a few GB per day. And this is within easy distance of Amazon's East Coast data center.
In an earlier version of Arq, there was no possibility to see which data was actually selected for backup (and which not). Has Arq become more user-friendly?
I'm a current user of ARQ, and it has, in my opinion (compared to Backblaze) a phenomenally user friendly description of exactly what is being backed up, and when, and when something was added, or modified.
Just FYI, I too would view Glacier support as something worth a normal upgrade fee. I am using Arq more as a long-term back-it-up-and-forget-about-it storage solution anyway, so it seems like a natural direction to go for a user like me (same strategy, but even lower cost).
That said, I am reminded that I should not forget about the Arq backups and do a few test restores sometime. :)
I think the key here is not to just provide a toggle for using Glacier instead of S3, but to have the historical snapshots migrated to Glacier from S3 and deleted every 90+ days.
Unless your files change a lot, keeping the latest backup version in S3 and previous versions in Glacier would mean that most of your backup data are still in S3 I think. Right?
Wow, yes, please do! I've never used Arq, and at the S3 prices it doesn't make sense on top of my Dropbox. But with Glacier, I'd love to be able to back up my entire HDD off-site for $5/month, and Arq looks like a good way to do that.
I'm a happy Arq customer today and I'd pay for an upgrade to Glacier! Longer term I'm sure you will have competitors on Glacier that you will be competing with so best to move sooner than later.
I use s3cmd sync to keep my files backed up.
But for my pictures, tunes, I use a Linux server and rsync. Now glacier is perfect for my tunes and family pictures.
What I find fascinating with Amazon's infrastructure push is the the successful 'homonymization' of their brand name Amazon.
Amazon simultaneously stands for ecommerce and web infrastructure depending on the context. e.g "Hey I want to host my server".. "Why don't you try Amazon". "Do you know where I can get a fair priced laptop?" "Check Amazon".
Is there any other brand that has done this successfully?
This actually confused me when I saw the headline. I saw 'Amazon' and, despite having been working with AWS all night, thought ecommerce first and couldn't figure out what they could name 'Glacier'. I was kind-of-not-really hoping it was going to be a new shipping option guaranteed to take a long time. That said, the actual service looks solid.
Virgin (200+ businesses operating or having operated, under the Virgin brand, ranging from infrastructures - trains, airlines - to records, banking, bridal saloons under the name Virgin Bride...).
Mitsubishi and Samsung springs to mind as two of the best known ones internationally where their brands are known in multiple markets internationally, though many of their businesses are less known outside Asia (e.g. Mitsubishi's bank is Japans largest). Any number of other Asian conglomerates.
ITT used to fall in that category back in the day: Fridges, PC's, hotels, insurance, schools,telecoms and lots more. I remember we at one point had both an ITT PC and fridge. The name was well known in many of its markets.
The large, sprawling, unfocused conglomerate have fallen a bit out of favor in Europe and the US. ITT was often criticized for their lack of focus even back in the 80's, and have since broken itself into more and more pieces and renamed and/or sold off many of them (e.g. the hotel group is now owned by Starwood).
Musical instruments (especially wind instruments) and motorcycles share a lot of similar design principles. Ever compared a saxophone to a WWII-vintage motorcycle engine?
In India, the Tata and Birla groups come to mind. The Tatas[1] particularly so. Both are family owned conglomerates and have been around before India gained Independence from the British.
The Japanese Keiretsu[2], from what I have read, is similar. The companies are somewhat loosely connected, but are connected nevertheless.
Looks like the perfect solution to backup all my photos. Considering 100 GB of photos, that's still just 100 * 0.01 * 12 = $12 a year! I'm sold on this!
This is a really good offering for media that you typically will keep locally for instant access, yet you want to have an off-site backup in a way that lives for a very long time.
Dropbox should work here, but it's simply too expensive. My photo library is 175GB. That isn't excessive when considering I store the digital negatives and this represents over a decade.
I don't mind not being able to access it for a few hours, I'm thinking disaster recovery of highly sentimental digital memories here.
If my flat burns down destroying my local copy, and my personal off-site backup (HDD at an old friends' house) is also destroyed... then nothing would be lost if Amazon have a copy.
In fact I very much doubt anyone I know that isn't a tech actually keeps all of their data backed-up even in that manner.
I find myself already wondering: My 12TB NAS, how much is used (4TB)... could I backup all of it remotely? It's crazy that this approaches being feasible. It's under 30GBP per month for Ireland storage for my all of the data on my NAS.
To be able to say, "All of my photos are safe for years and it's easy to append to the backup.". That would be something.
A service offering a simple consumer interface for this could really do well.
They store the last 30 days worth of versions of every file you modify. Dropbox could keep versions from the last 10 days in S3 but move the rest to Glacier. Restores for older versions wouldn't be instant, but if S3 storage is a nontrivial expense for them moving the bulk of previous versions to Glacier would cut down on costs.
I currently just backup my digital photos (~20Gb) to S3 via replication from my QNAP NAS... works out about $3 per month... I'll probably use the option they mention here to auto move content to Glacier from S3 coming soon...
"In the coming months, Amazon Simple Storage Service (Amazon S3) plans to introduce an option that will allow you to seamlessly move data between Amazon S3 and Amazon Glacier using data lifecycle policies."
sigh Dropbox should not be used as a backup system. A system that synchronizes live should never get that role, except if they can guarantee that old data is never overwritten, new is always appended. This is not the case with dropbox - I've experienced multiple scary occurrences of old versions going to the nirvana with certain user actions. In certain cases the old data just appears to be gone, in other cases the web interface shows them but a restore results in an error message. Especially data move appears to be buggy. Dropbox appears to be simple, but the backend processes are really not and there is too much going on for it to be a reliable backup system, especially if you also share some of your data to team mates.
No sigh was necessary, I understand. I even mentioned that I have a NAS and off-site backup... so either you didn't read it all or you stopped the moment you encountered the word "Dropbox" and started typing.
That said, people DO use DropBox as backup.
If you take a walk around the British Library and asked every PhD student working there how they "Backup" their research and work in progress, I bet every single person who believes that they have a backup will say "Dropbox", and the only exceptions will be a few who don't really have a backup.
I know that because I ensured my girlfriend does have a real backup solution in place that is tested. Not one of her peers seems to.
DropBox is used for backup because they've made file sync so damn easy that most people can be convinced that if a file exists in many places, it is backed-up.
My whole point is that now storage for long term backup is priced in a way that is affordable to most, that consumer services may emerge that offer true backup to consumers and can successfully migrate people from lesser solutions (DropBox, stacks of CD-ROMs, etc).
One of the things about backup is that it needs to be easy. Currently the size and cost of backups make it expensive, and the only way to reduce the cost makes it difficult (HDD local copies stored at a friends' house for example).
By reducing the cost, perhaps we can finally increase the ease... and then a day may come in which most people have a real backup solution.
"I even mentioned that I have a NAS and off-site backup... so either you didn't read it all or you stopped the moment you encountered the word "Dropbox" and started typing."
No offense, but this post doesn't quite match what you wrote originally. My sigh was in response to the phrase "Dropbox should work here, ...". You didn't state security as your concern as to why not use Dropbox, rather it was cost. This might lead someone who only needs <2GB backed up to believe that Dropbox is perfectly fine for that task.
"DropBox is used for backup because they've made file sync so damn easy that most people can be convinced that if a file exists in many places, it is backed-up."
And that's exactly what I'm scared about and why I'm saying it again and again that you shouldn't do it - if only one person listens and avoids potential data loss because of it I've already reached my goal.
"One of the things about backup is that it needs to be easy. Currently the size and cost of backups make it expensive, and the only way to reduce the cost makes it difficult (HDD local copies stored at a friends' house for example). By reducing the cost, perhaps we can finally increase the ease... and then a day may come in which most people have a real backup solution."
Agreed. IMO no consumer backup system is quite there yet. TimeMachine is very close, if only it would do better logging and have some more intelligence about warning messages.
"No offense, but this post doesn't quite match what you wrote originally. My sigh was in response to the phrase "Dropbox should work here, ...". You didn't state security as your concern as to why not use Dropbox, rather it was cost. This might lead someone who only needs <2GB backed up to believe that Dropbox is perfectly fine for that task."
A fair point.
In my case, Dropbox use is in addition to local RAID (scratch) NAS (network scratch, access of larger files) + off-site (backup).
I only use Dropbox for syncing and sharing.
The sync vs backup is an interesting one, simply because most consumers couldn't tell you the difference.
For example: Q: "Are your contacts backed up?". A: "Yes, they're sync'd to Google".
I did conflate my scenario with thinking about my girlfriend's peers in my post. And then reacted from my perspective again... my bad.
"Agreed. IMO no consumer backup system is quite there yet. TimeMachine is very close, if only it would do better logging and have some more intelligence about warning messages"
Vigorous agreement here too, except for the TimeMachine bit as that is Mac only and doesn't work for <insert any other system or device that isn't Apple Mac OSX).
"Vigorous agreement here too, except for the TimeMachine bit as that is Mac only and doesn't work for <insert any other system or device that isn't Apple Mac OSX)."
Well yes, obviously it only works in a mac-only household/office. I still think it's a good solution for non technically experienced users who only have macs since it's so simple that you could literally explain your grandma how to setup. I don't think mobile devices are that important on the other hand. The important data on them should usually be synched to your computer and as long as that is backuped, you should be fine. On windows I think that the built-in backup since Windows 7 is finally decent, albeit not yet grandma-proof ;).
To be fair though, Dropbox is way better than any other backup solution consumers usually use. Personally, I've never seen data loss occur on Dropbox, but I'm sure it can happen - it's just way less likely than the average user messing up their own backup.
Though you make a good point, your comment could have avoided sounding pretentious if you shortened it by removing the first few words.
Also, please don't use uppercase for emphasis, as mentioned in the guidelines [1]. If you want to emphasize a word or phrase, put asterisks around it and it will get italicized.
Alright, I've edited it, thanks for the heads up. Dropbox and backup in one phrase tends to awaken my temper as I've had some horrible almost data losses (as in I've had to restore data from the dropbox cache folder that could basically be purged at any point).
Yes, I'm aware of that. This helps, up to the point where you don't notice that something's gone for 'a few days'. Let's say you have your student project's folder shared with two other colleagues. You take a few days off, use your PC for casual browsing, meanwhile your colleagues are working on the project. At the end, one of them (who doesn't quite understand how dropbox works) deletes the files while the other is still working on it and dropbox gets confused. Your PC gets synched at a moment where you don't notice. Come back from your holidays and you will have a nice surprise waiting.
It doesn't matter whether he notices it or not, you can't count on social factors when it comes to backup systems. He might notice it but then he thinks, oh no problem, it's all in Dropbox anyway, I'll talk to Dylan16807 when he gets back.
I've never had any complete data losses until now, but I've had this situation where the Dropbox cache was the only place to retrieve files twice and it showed me that it can't be trusted this way. I'm certainly not gonna wait until the real deal happens just to have a personal story on how Dropbox can go horribly wrong. I've been shown the possibility and that's enough for me.
As for specifics, the case I mentioned is the one I saw where it can go wrong. IMO the only really save way to use Dropbox without additional backups is if you never share folders and only ever use one system with write-access at the same time - which is not the usual use-case of that product.
I just want to understand better why you had to resort to the cache. The only time I've ever done that was when I accidentally deleted a bunch of data and didn't want to restore each file by hand or put in a support ticket.
I can't quite put my foot down on what the scenario is exactly. One time might have been a case where a folder with lots of small files got moved away and Dropbox only allows single file restore, which would have taken hours, possibly longer than the lost work. Once I've done something stupid where I wanted to exchange a whole Dropbox folder with a different version. I switched off the client on one PC (by mistake, obviously) and exchanged the folder on the other, then switched it on again. Then I became aware the the old folder had important stuff in it. The versioning was corrupted at that point, many old files would either not show up online or would produce an error message.
So usually there is some user behaviour involved, yes. But the whole point of a backup system is that you can rely on it, even when the user behaves stupidly up to a certain degree. Sync is for day to day collaboration and data management, it's not for backup.
If dropbox works well for you for source control, that is great. But frankly, if you get to the point where you have to start writing hacks to keep it working, it is probably time to move onto something designed for the task.
The ideal solution is the Quadfecta (is that a word?) - Dropbox for (in my experience) excellent versioning/synchronizing (Never failed me) + Backblaze (or its ilk) for continuous Offline backups + Super Duper (weekly/whenever) - for Image Backups - + Something (Arq?) on top of Glacier for long time off-site-archival.
For $50 in software (Arq+SuperDuper), $100 for an external HD, and less than $25/month ($4-backblaze, $10 Dropbox, $10 Glacier) you have a backup system that is next to air tight for a Terabyte of Data and a working set (on dropbox) of 100 Gigabytes.
Dropbox stores their stuff in S3 a little different. It's not a 1:1 correspondence between user files and objects under Dropbox's S3 account. The fact that they use S3 as their backing store means very little. It certainly sounds good to have S3 in back when you talk about scalability and durability, but 1) they could just as easily use something else, and 2) depending on their sharding strategy, a single lost object could impact multiple files at the user level.
Right - the point I was trying to make, is he was putting all his eggs in one basked. If anything catastrophic happened to S3, he might lose both his S3 as well as his Dropbox backups.
If you are going to the effort of having dual-backup systems, may as well try and find something that can't be impacted by a single disaster.
And yes, I might have daily-monday / daily/tuesday.... and so on, and it would work better, but this way works for me. There is a lot of room for improvement, and it is not hard to implement with cron-jobs and more buckets.
You could try adding objects with the date in their names to a 'backup' bucket rather than create separate buckets for each backup. S3 also supports object expiration, so you can set your daily backups to expire after 7 days and weekly backups to expire after 4 weeks, for example.
Let's say your data loss occurs on the Sunday, 31th of a month at 23:55 and then gets synched across all your S3 backups (or it could occur at some point before that but you don't notice it). And poof goes your data.
THIS. I'd say at least half of the time I went to retrieve a good revision of a file from backup, it was already too late and the backup was trash as well.
>A system that synchronizes live should never get that role, except if they can guarantee that old data is never overwritten, new is always appended. //
Why don't Dropbox enable this use-case. It would surely be very easy to implement. I guess that pack-rat does do this in a way but seems like overkill.
I'd like something along the lines of duplicating all files but requiring confirmation of deletions and overwrites.
It's a bit strange how S3 has a 'US Standard' region option, while Glacier has the usual set of regions (US East, US West, etc). I wonder if this means that unlike S3, Glacier isn't replicated across regions?
Keep in mind that even if your data is in one AWS region, it'll still be stored in multiple different datacenters some distance apart. Just not on the other side of the US.
I just started a project where I'm keeping a raspberry pi in my backpack and am archiving a constant stream of jpgs to the cloud. I've been looking at all the available cloud archival options over the last few days and have been horrified at the pricing models. This is a blessing!
391 comments
[ 3.0 ms ] story [ 234 ms ] threadhttp://aws.amazon.com/glacier/?utm_source=AWS&utm_medium...
I'm often creating pretty big media assets, so Dropbox doesn't necessarily offer enough space or is - for me - too expensive in the 500gb version (i.e. $50 a month).
Glacier would be $10 a month for 1 terabyte. Fantastic.
The salt is used to prevent amazon from just keeping the hashes around to report that all is well.
To avoid abuse, restrict the number of free verification requests per month.
+ the $120 or so per TB to transfer it outside of AWS if you need the whole thing back as fast as possible. Still likely to be very cheap as long as you treat it as a disaster recovery backup, though. Will definitively consider it.
(an alternative for you is a service like Crashplan, which also allows you very easy access to past file revisions via a java app and can be very cheap and also allow "peer to peer" backups with your friends/family; the downside with Crashplan is that it can be slow to complete a full initial backup to their servers or to get fully backed up again if you move large chunks of data around)
This is why the cost of retrieval is so high: every time they need to pull data the drives need to be spun back up (including drives holding data for people other than you), accessed, pulled from, then spun back down and put to sleep. Doing this frequently will put more wear and tear on the components and cost Amazon money in power utilization.
As is Glacier should be extremely cheap for AWS to operate, regardless of the total amount of data stored with it. Beyond the initial cost of purchasing hard drives, installing, and configuring them the usual ongoing maintenance and power requirements go away.
Drawback is "jobs typically complete in 3.5 to 4.5 hours."
Seeing as how people tend to be pack rats, I can see this being huge.
Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months.
They make so many backups so quickly that there is only a 0.00000000001% (I didn't count the zeros) chance of this occuring.
A more comical way to look at it: The percentage is actually AWS saying "to keep costs low, we plan to lose this many files per year"; when we screw up and things don't go quite to plan, we lose a _lot_ more.
†roughly proportional if you have << 1e11 objects
Similar deal for hard drive bit error rates, where the quoted average BER may not necessarily accurately represent what can happen in the real world. For example, an unrecoverable read error loses 4096 bits (512 byte sectors) or 32768 bits (4k sectors) all at once, rather than individual bits randomly flipped over a long period.
Amazon Glacier synchronously stores your data across multiple facilities before returning SUCCESS on uploading archives.
Amazon S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years....
Sure hope transferring 10 or 20 gigabytes of data from S3 to Glacier is easy.
What kind of system has Amazon most likely built that takes 3-4 hours to perform retrieval? What are some examples of similar systems, and where are they installed?
Or in case they are using regular hard drives, you might want to have this kind of time limits in order to pool requests going to a specific set of drives. This would enable them to power down the drives for longer periods of time.
The 3-4 hour estimate may also be artificial. Even if you can in most cases retrieve the data faster, it would be good to give an estimate you can always meet. They might also want to differentiate this more clearly from standard S3.
And we should not forget that it does take time to transfer for example one terabyte of data over network.
There'll be a near-line HDD array. This is for the recent content and content they profile as being common-access.
Then there'll be a robotic tape library. Any restore request will go in a queue annd when an arm-tapedrive becomes free they'll seek to the data and read it into the HDD array.
Waiting for a slot with the robot arm - tape drive is what will take 4 hours.
EMC(kinda), Fujitsu etc make these.
http://en.wikipedia.org/wiki/Tape_library
http://www.theregister.co.uk/2012/06/26/emc_tape_sucks_no_mo...
I'm sure that there is a general tiered storage platform (as mentioned above) which keeps some of the data online as well. That would let you run a "defrag" algorithm later if you find you need it.
If you think about it, writes are more common than reads on average, so it's more efficient to just write to whatever tape is online and deal with the fragmentation problem on the read end, as opposed to queueing writes until the 'correct' tape can be brought online just save some time reading. Also in backup situations like this, it's more important to get the backup done in a timely manner.
First, no tape. The areal storage density of tape is lower than hard disks. Too many moving parts involved. Too hard to perform integrity checks on in a scalable, automated fashion without impacting incoming work.
Second, in order to claim the durability that they do (99.999999999%), that means every spot along the pipe needs to meet those requirements. That means the "near-line HDD array" for warm, incoming data needs to meet those requirements. Additionally, if the customer has specified that the data be encrypted, it needs to be encrypted during this staging period as well. It also needs to be able to scale to tens if not hundreds of thousands of concurrent requests per second (though, for something like Glacier, this might be overkill).
They've already built something that does all that. It's called S3. The upload operations likely proxy to S3 internally (with a bit of magic), and use that as staging space.
After that, the bottleneck is likely I/O to Glacier's underlying storage - but again, not tapes. See this post for deets: http://news.ycombinator.com/item?id=4416065
What would be absolutely fascinating is a pay-before-you-go storage service — data cryonics.
Paying $12 to store a gigabyte of data for 100 years seems like a pretty intriguing deal as we emerge from an era of bit rot.
As long as that data is decode-able and more importantly, find-able (out of all the GBs frozen for 100 years, why would you want to look at any particular one of them?).
But that's from when it was harder to make and upload data, so people only put meaningful (to them) stuff online. These days we'd have a hundred thousand copies of a few popular MP3's and everyone's crappy digital photos. The percentage of meaningful stuff would be a lot lower.
I'd store my pictures there. Finding old pictures of grandparents when they were little, or even older stuff, is amazing. Wouldn't it be cool if my descendants could still look at pictures of my family in 100 years?
Provided that downloading from this 100 year store is something I could do X times per year, and so long as I could append more data to it over time, it's an interesting business model.
I'm not sure what kind of organisation I'd actually trust to store data for that length of time - a commercial organisation is probably going to be more effective at providing service but what commercial organisation would you trust to provide consistent service for 100 years? A Swiss bank perhaps? Governments of stable countries are obviously capable of this (clearly they store data for much longer times) but aren't set up to provide customer service.
Because as we answer issues of cost and availability, a logical thing to wonder is "how long can I really depend on it though?" As quickly as cloud services (where "lifetimes" are measured at six years) have entered our economy, that's a question begging to be answered.
Amazon at least seems to be an "eventually durable" datastore, though. Meaning that if you are told in the future that it will go offline, you have an excellent chance to make other suitable arrangements. Say there's a 0.01% chance of this product being discontinued next year, up to a 10% chance 5 years from now. I have to think there will almost certainly be other services you can move your data to, on similar terms, for a long time.
That's assuming you're around at all, and nothing reeeeeally bad happens even so. Making data last after your death (or even after you stop paying!) is a lot harder in this environment, and achieving true 100-year durability is a tough nut indeed.
I like your bank idea, since the preservation of a bank account is just a specialized simple case of data preservation. Data preservation seems rather more reliable when the data is directly attached to money. Then again, maybe banks themselves are on their way out for this purpose — Dropbox could become the new safety-deposit box.
But there are 100-year domain registrations, after all. Maybe we're ready for organizations to at least offer 100-year storage, too.
FWIW, we are happy to support this kind of use, and see our customers doing it in an ad-hoc way every day. We have s3cmd in our environment, and support it. As soon as their is a glacier complement to s3cmd we will put that into place as well, although with the strange traffic and retrieval pricing, I'm not sure how useful folks will find it ...
I don't consider that obvious. I live in Berlin, the capital of what most would consider a stable country, but my apartment (which is even older) has been a part of 5 different countries in the last 100 years (German Empire, Weinmar Republic, Nazi Germany, East Germany and finally, the Federal Republic of Germany).
Of course, predicting future stability is complete guesswork!
I would say a stable country is one which has had a legit democracy for 70 or so years and doesn't share a border with a non-democratic / non-legit-democratic state. These two points suggests its unlikely to have a revolution or be invaded any time soon.
Of course, if you look at the UK governments track record with IT.. you wouldn't trust it. I would say the same with the US, especially in concern with data security.
[NB I'm not English]
Japan (660 BCE) and China (221 BCE) have both had feudal / bureaucratic governments exhibiting very high levels of stability. While dynasties and eras are marked, the overall states persisted largely intact.
The oldest country (not government) is likely Vietnam (2897 BCE). Other contenders: Japan (660 BCE), China (221 BCE), Ethiopia (~800 BCE), or Iran (678 BCE).
Few of today's modern states pre-date the 19th Century, many antedate World War II or the great de-colonialisation of the 1960s including much of Africa and Oceana (some of the longest inhabited regions of Earth).
Among the more long-lived institutions are the Catholic Church (traditionally founded by Jesus ~30 AD, emerging as an institutional power in 2nd Century Rome). The oldest company I can find is Kongo Gumi, founded in 578, a Japanese construction firm. The record however is likely held by the Shishi Middle School founded in China between 143 and 141 BCE.
My own suggestion would be the Krell, though some might disqualify this based on a requirement for human organization.
http://en.wikipedia.org/wiki/Egypt_history
Having a long history obviously isn't a predictor of future stability. According to the Long Now Foundation site [2], a Japanese company that existed since 578CE went bust in 2007.
[1] http://www.bizaims.com/content/the-100-oldest-companies-worl... [2] http://blog.longnow.org/02008/06/13/the-100-oldest-companies...
As recently as 1870, during the Italian unification, the church was stripped of its power to govern Rome after an armed confrontation between armies at the gates of Rome ("XX Septembre"). During this period, the Pope (the ninth Pius) seriously considered fleeing Italy.
You could have a foundation established to choose and pay an array of commercial organizations that do the archiving.
The Stora Kopparberg mining company has existed since it was granted a charter from King Magnus IV in 1347.
A few banks tend to last for a long time [1]. Banca Monte dei Paschi di Siena has existed for about 540 years.
Beretta, the italian firearms company, has existed for 486 years (and has been family owned the entire time).
East and West Jersey were owned by a land proprietorship for around 340 years starting from King Charles II bestowing the land to his brother James in 1664. [2]
At first I thought multinational corporations would be more stable because they could move from land to land to avoid wars and such. But apparently they haven't lasted nearly as long as their single-nation counterparts.
The Knights Templar were granted a multi-national tax exemption by Pope Innocent II in 1139, and lasted almost 200 years until most of their leadership was killed off in 1307.
The Dutch East India Trading Company was one of the first [modern] multinational corporations, spanning almost 200 years from 1602-1798.
However, the longest-lasting companies have been family owned and operated. [3] [4]
It appears most all companies that have lasted a long time are due to two factors: dealing in basic goods and services that all humans need, and looking ahead to the future to change with the times.
[1] https://en.wikipedia.org/wiki/List_of_oldest_banks [2] https://docs.google.com/viewer?a=v&q=cache:nYQU4NpfD74J:... [3] http://www.businessweek.com/stories/2008-05-14/centuries-old... [4] http://www.bizaims.com/content/the-100-oldest-companies-worl...
If you can figure out a way to convince a few dozen people every decade that the best way to glorify God is to isolate themselves off somewhere maintaining your archival data, you'll be set for centuries.
"Long Data, LLC... We secure your data for the long-term".
I might create a script that uploads everything to Glacier and just keeps a couple of the latest backups on S3 though.
1. http://www.allthingsdistributed.com/2012/08/amazon-glacier.h...
that would certainly be very nice. cperciva, what do you think?
Also, with Tarsnap's average block size (~ 64 kB uncompressed, typically ~ 32 kB compressed) the 50 microdollar cost per Glacier RETRIEVAL request means that I'd need to bump the pricing for tarsnap downloads up to about $1.75 / GB just to cover the AWS costs.
I may find a use for Glacier at some point, but it's not something Tarsnap is going to be using in the near future.
it's unfortunate, because some backups happen to just lie around for very long. it would be nice to take advantage of (the low cost of) glacier for that.
that said, if it's not possible with tarsnap now, it's not possible now. :D. if you find a satisfying possibility to incorporate it in the new backend(s) design (if that's fixable in the backend(s) alone), i'd surely be pleased.
Or is the only way to encrypt it yourself, and then transfer it?
With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,vud,vmdk,vmsn,vmsd,hdd,vdi,vmwarevm,nvram,vmx,vmem,iso,dmg,sparseimage,sys,cab,exe,msi,dll,dl_,wim,ost,o,log,m4v" files. They also don't want to backup your /applications, /library, /etc, and so on locations. They also make it clear that backing up a NAS is not the target case for their service.
I can live with that - because, honestly, it's $4/month, and my goal is to keep my working files backed up. System Image backups, I've been using Super Duper to a $50 external hard drive.
Glacier + a product like http://www.haystacksoftware.com/arq/ means I get the benefit of both worlds - Amazon will be fine with me dropping my entire 256 Gigabyte Drive onto Glacier (total cost - $2.56/month) and I get the benefit of off site backup.
The world is about to get a whole lot simpler (and inexpensive) for backups.
However, you can also secure the encryption with a password not associated with the account. Or even provide your own 448-bit key. If you do either of these options, CrashPlan support will not be able to help you
This setup allows CrashPlan to easily help non-technical home users, while allowing technically savvy users to securely hang themselves with their own encryption.
Furthermore, you can use the encryption key + a custom password, or your own encryption key with a passphrase. In this case, it is encyrpted locally and the key is not sent to Crashplan[1].
[1] http://support.crashplan.com/doku.php/recipe/change_security...
Again, if you're not typing the password in every time, a local compromise is almost certainly game-over. Apple's keychain helps reduce the damage if the data's not actively used but for something like CrashPlan which is always running the attacker is probably going to be lucky.
Passwords and key are usually shown in an obscured form, usually with asterisks, and stored in the user or system keychain. You are absolutely right that the security value of these standard practices should not be overvalued but still … what else within the security framework of CrashPlan is not done in accordance with best practice?
(I assume, BTW, that CrashPlan does not use the system or user keychain on the Mac because it is not a real Mac citizen but a Java-based app. Firefox and Wuala – the latter Java-based too – don't use the user or system keychain either.)
A home user is fine with a 3.5-4 hour window before their backup becomes available for download (as it will probably take them days to download it anyway).
In a corporate environment, I don't want to wait around for 3.5-4 hours before my data even becomes available for restore in a disaster recovery situation.
Seems good for archive-only in a corporate environment (as the name implies).
Does this satisfy the legal requirements you mention? I assume there are further requirements on what constitutes sufficient encryption?
In a true disaster recovery (Building burned down or is otherwise unavailable) - it usually takes most businesses a week or so just to find new office facilities.
But - agreed, there will be some customers for whom Glacier wouldn't work well for all use cases.
Now - a blended S3/Glacier offering might be very attractive.
In that case, 3-4 hours would be more than acceptable.
I am guessing you work in a company with a good IT department then, I am guessing this is not the average. Many companies I have worked for 4 hours would be a miracle with a 1-3 day operation minimum.
And don't forget that the X00MB type size limits many IT departments puts everywhere it not because getting a TB hard drive is cheap, but because all of the extra backups add to the cost of each new MB. Having another extremely cheap way they could backup large amounts of data (encrypted?) would help to reduce the cost of each extra GB.
ding!
It uses rsync's algorithm to figure out what has changed, and everything is encrypted locally.
(The site says it's in beta, but it's said that for years.)
Arq is the pinnacle of my rather large backup pyramid which also includes: Dropbox, Superduper, Crashplan, rsync, SVN/GIT and more.
That said, I am reminded that I should not forget about the Arq backups and do a few test restores sometime. :)
I will gladly pay more money for Arq again.
Arq is incremental backup. The data structure is similar to git's http://www.haystacksoftware.com/arq/s3_data_format.txt
Unless your files change a lot, keeping the latest backup version in S3 and previous versions in Glacier would mean that most of your backup data are still in S3 I think. Right?
http://www.haystacksoftware.com/support/arqforum/topic.php?i...
Already has Amazon support: http://git-annex.branchable.com/tips/using_Amazon_S3/
Very friendly with very large files!
Amazon simultaneously stands for ecommerce and web infrastructure depending on the context. e.g "Hey I want to host my server".. "Why don't you try Amazon". "Do you know where I can get a fair priced laptop?" "Check Amazon".
Is there any other brand that has done this successfully?
Edit: I should have specified internet brand.
Mitsubishi and Samsung springs to mind as two of the best known ones internationally where their brands are known in multiple markets internationally, though many of their businesses are less known outside Asia (e.g. Mitsubishi's bank is Japans largest). Any number of other Asian conglomerates.
ITT used to fall in that category back in the day: Fridges, PC's, hotels, insurance, schools,telecoms and lots more. I remember we at one point had both an ITT PC and fridge. The name was well known in many of its markets.
The large, sprawling, unfocused conglomerate have fallen a bit out of favor in Europe and the US. ITT was often criticized for their lack of focus even back in the 80's, and have since broken itself into more and more pieces and renamed and/or sold off many of them (e.g. the hotel group is now owned by Starwood).
The Japanese Keiretsu[2], from what I have read, is similar. The companies are somewhat loosely connected, but are connected nevertheless.
[1] http://en.wikipedia.org/wiki/Tata_Group
[2] http://en.wikipedia.org/wiki/Keiretsu
Virgin has been already mentioned, but is a really interesting example. It really is just a brand - there is no single controlling company.
Dropbox should work here, but it's simply too expensive. My photo library is 175GB. That isn't excessive when considering I store the digital negatives and this represents over a decade.
I don't mind not being able to access it for a few hours, I'm thinking disaster recovery of highly sentimental digital memories here.
If my flat burns down destroying my local copy, and my personal off-site backup (HDD at an old friends' house) is also destroyed... then nothing would be lost if Amazon have a copy.
In fact I very much doubt anyone I know that isn't a tech actually keeps all of their data backed-up even in that manner.
I find myself already wondering: My 12TB NAS, how much is used (4TB)... could I backup all of it remotely? It's crazy that this approaches being feasible. It's under 30GBP per month for Ireland storage for my all of the data on my NAS.
To be able to say, "All of my photos are safe for years and it's easy to append to the backup.". That would be something.
A service offering a simple consumer interface for this could really do well.
Packrat is a Dropbox Pro add-on feature that saves your file history indefinitely.
"In the coming months, Amazon Simple Storage Service (Amazon S3) plans to introduce an option that will allow you to seamlessly move data between Amazon S3 and Amazon Glacier using data lifecycle policies."
edit: emphasis formatting, niceifying.
That said, people DO use DropBox as backup.
If you take a walk around the British Library and asked every PhD student working there how they "Backup" their research and work in progress, I bet every single person who believes that they have a backup will say "Dropbox", and the only exceptions will be a few who don't really have a backup.
I know that because I ensured my girlfriend does have a real backup solution in place that is tested. Not one of her peers seems to.
DropBox is used for backup because they've made file sync so damn easy that most people can be convinced that if a file exists in many places, it is backed-up.
My whole point is that now storage for long term backup is priced in a way that is affordable to most, that consumer services may emerge that offer true backup to consumers and can successfully migrate people from lesser solutions (DropBox, stacks of CD-ROMs, etc).
One of the things about backup is that it needs to be easy. Currently the size and cost of backups make it expensive, and the only way to reduce the cost makes it difficult (HDD local copies stored at a friends' house for example).
By reducing the cost, perhaps we can finally increase the ease... and then a day may come in which most people have a real backup solution.
No offense, but this post doesn't quite match what you wrote originally. My sigh was in response to the phrase "Dropbox should work here, ...". You didn't state security as your concern as to why not use Dropbox, rather it was cost. This might lead someone who only needs <2GB backed up to believe that Dropbox is perfectly fine for that task.
"DropBox is used for backup because they've made file sync so damn easy that most people can be convinced that if a file exists in many places, it is backed-up."
And that's exactly what I'm scared about and why I'm saying it again and again that you shouldn't do it - if only one person listens and avoids potential data loss because of it I've already reached my goal.
"One of the things about backup is that it needs to be easy. Currently the size and cost of backups make it expensive, and the only way to reduce the cost makes it difficult (HDD local copies stored at a friends' house for example). By reducing the cost, perhaps we can finally increase the ease... and then a day may come in which most people have a real backup solution."
Agreed. IMO no consumer backup system is quite there yet. TimeMachine is very close, if only it would do better logging and have some more intelligence about warning messages.
A fair point.
In my case, Dropbox use is in addition to local RAID (scratch) NAS (network scratch, access of larger files) + off-site (backup).
I only use Dropbox for syncing and sharing.
The sync vs backup is an interesting one, simply because most consumers couldn't tell you the difference.
For example: Q: "Are your contacts backed up?". A: "Yes, they're sync'd to Google".
I did conflate my scenario with thinking about my girlfriend's peers in my post. And then reacted from my perspective again... my bad.
"Agreed. IMO no consumer backup system is quite there yet. TimeMachine is very close, if only it would do better logging and have some more intelligence about warning messages"
Vigorous agreement here too, except for the TimeMachine bit as that is Mac only and doesn't work for <insert any other system or device that isn't Apple Mac OSX).
Well yes, obviously it only works in a mac-only household/office. I still think it's a good solution for non technically experienced users who only have macs since it's so simple that you could literally explain your grandma how to setup. I don't think mobile devices are that important on the other hand. The important data on them should usually be synched to your computer and as long as that is backuped, you should be fine. On windows I think that the built-in backup since Windows 7 is finally decent, albeit not yet grandma-proof ;).
Also, please don't use uppercase for emphasis, as mentioned in the guidelines [1]. If you want to emphasize a word or phrase, put asterisks around it and it will get italicized.
[1]: http://ycombinator.com/newsguidelines.html
Can you explain how dropbox has lost history for you in the past?
I've never had any complete data losses until now, but I've had this situation where the Dropbox cache was the only place to retrieve files twice and it showed me that it can't be trusted this way. I'm certainly not gonna wait until the real deal happens just to have a personal story on how Dropbox can go horribly wrong. I've been shown the possibility and that's enough for me.
As for specifics, the case I mentioned is the one I saw where it can go wrong. IMO the only really save way to use Dropbox without additional backups is if you never share folders and only ever use one system with write-access at the same time - which is not the usual use-case of that product.
So usually there is some user behaviour involved, yes. But the whole point of a backup system is that you can rely on it, even when the user behaves stupidly up to a certain degree. Sync is for day to day collaboration and data management, it's not for backup.
That said, we still keep multiple backups.
For $50 in software (Arq+SuperDuper), $100 for an external HD, and less than $25/month ($4-backblaze, $10 Dropbox, $10 Glacier) you have a backup system that is next to air tight for a Terabyte of Data and a working set (on dropbox) of 100 Gigabytes.
In case you happen to read with show-dead on: it's a "superfecta."
I have my MacBook and a Linux server linked to my Dropbox account. So changes in my documents are synced to my Linux.
My Linux run three cron-jobs. One daily, one weekly and another monthly. The command is.
s3cmd sync --delete-removed ~/Dropbox/documents/ s3://backup-daily/
There are buckets for weekly and monthly too.
Note: the command is not exactly like that, check the man page.
That way I have backed up all my documents very cheap.
If you are going to the effort of having dual-backup systems, may as well try and find something that can't be impacted by a single disaster.
And yes, I might have daily-monday / daily/tuesday.... and so on, and it would work better, but this way works for me. There is a lot of room for improvement, and it is not hard to implement with cron-jobs and more buckets.
Let's say your data loss occurs on the Sunday, 31th of a month at 23:55 and then gets synched across all your S3 backups (or it could occur at some point before that but you don't notice it). And poof goes your data.
Why don't Dropbox enable this use-case. It would surely be very easy to implement. I guess that pack-rat does do this in a way but seems like overkill.
I'd like something along the lines of duplicating all files but requiring confirmation of deletions and overwrites.
> The US Standard Region automatically routes requests to facilities in Northern Virginia or the Pacific Northwest using network maps.
I guess this means that the data is automatically geographically sharded?