Ask HN: How would you store 5TB of data for 50 years, untouched?
Let's say you were thinking of putting digital data in a time capsule where it couldn't be touched for 50 years. How would you store it? How would you ensure it was readable at the end of the 50 years?
177 comments
[ 0.23 ms ] story [ 274 ms ] threadA4 page, which has 210x297mm, say a border of 10mm all around non-printable area, gives 190x277.
Say we print at a resolution of 0.5mm, and in 16 shades, so we get 4x4 = 16 bits per mm, or 2 bytes. That gives 105,260 bytes per page. Probably we can squeeze more bytes in than this, but let's allocate those to redundancy and error correcting codes.
So for 5TB (in hard drive size, 5e12) would take about 23.75 million sheets of paper printed on both sides. At 5g per sheet, that's about 119 metric tons.
1 ream (500 pages) of the 75g/m^2 A4 paper I have beside my printer here is about 50mm thick. Say 215x305x55mm including some slack for packaging, is a little over 3.6 litres in volume; total volume for 23.75 million sheets is 171,285 litres, or 171.285 cubic metres.
A room with a ceiling height of 3 metres would need to be only 8 meters on each side to store this. Of course, the room shouldn't have any windows and should be at the appropriate humidity, etc.
The cost of the paper, assuming $5 per 500 pages, is less than a quarter of a million. Much more forbidding is the labour and temporary acquisition of printers required to transcribe the data to paper. A good printer @50ppm would take nearly 2 years, assuming zero downtime and very quick paper and toner changes. To do it in more reasonable time and with more reliability, you'd want a bunch more printers; and of course, you'd need to hire the people to do the work of shuffling paper and carting it around, but I'd bet you could probably do it for less than the cost of the paper, particularly if you did it in a cheap labour country.
size of one pixel: 0.5mm * 0.5mm= 0.25 mm2
1 page can then hold: 52630/0.25 = 210520 pixel
1 pixel is 2 byte thus 1 page is 421040 byte is about 0.5 MB/page (or 1 MB if you print double sided)
5 TB= 5000 GB = 5,000,000 MB = 5 Million pages
this is 5000 books with 1000 pages each.
I intentionally chose a pretty big pixel for redundancy reasons rather than trying to be clever and working out a code etc., but the link in the reply to my post looks more worked out.
use 1200 dpi printer to print b/w dots this gives: 8.3 * 11.7 * 1200 * 1200 bits/page = 17 MB/page
5 TB are 142 Books with 1000 pages double sided each (size of a small personal library)
case closed.
I'd include the definition of a simple machine, and the text of a program written in opcodes for the machine, for decoding.
Alternatively burn a 100 or so Blu-rays and get two Blu-ray readers (one on a mobile device) other an external reader that you will attach to the aforementioned desktop.
Or who knows, maybe holographic storage (http://en.wikipedia.org/wiki/Holographic_data_storage) will come around finally and store the 5TB in a toothpick sized gizmo (which might probably run 512 cores of Googple's Chip (in my alternate reality, Google and Apple merge and buy Intel)...
A quick Google of HDD MTBF suggests that 1 million hours (over a century) is wildly optimistic, and typical failure rate is 2-4% per year, possibly as high as 13%. If e is the failure rate (as a fraction of 1), and assuming a constant and independent failure rate over time, then the survival chances for any one drive are:
So the chance that any one will fail is: With n mirrors (assuming a reliable checksum to verify data in the event of only a single copy of a mirror surviving), the chances of all failing, f, are: So, for a reliability of 99.999%, and hoping you can keep the individual yearly failure rate at 3%, so f=0.00001 and e=0.03, n would need to be at least 47.I don't know if the bits eventually lose their magnetism over time, so if they do, you may need to spin up the drives every so often and copy to and from drives to make sure the data is still "fresh", but I seriously doubt they'd need to be left on and spinning for the entire 50 year span.
Removing that constraint and completely ignoring cost I'd also setup a low-risk savings account with $1M in it and put the data on S3 and Rackspace Cloud. I'd store access credentials in the capsule. Odds are pretty good one of those 2 will be around in 50 years (and you'll have a chunk of money left over in interest).
Try to keep everything ASCII, with really good text descriptions of data formats.
Realistically 50 years is not a long time: I would bet we'll still have legacy access to USB, SATA, and probably ext3 & NTFS (though probably not IDE). Tons of computer folk who used these technologies will still be alive to work them. English will still be the primary language in the US.
An interesting problem is what to do when the timescale allows these things to change. What if nobody remembers USB, or what spinning platters are. Or the English language?
Some people claim than MO media and DVD-RAM can guarantee 30 years, but this still is an estimate, they have not been around long enough to actually know.
The only "reliable" way to store digital data for more than five years known today is to copy them to new media well in advance of the old media loosing them, and even that is difficult if the amount of data is growing faster than the the storage technologies get faster. (I don't know if I should trust Eric Schmidt, but a few days ago he claimed that currently humanity generates as much data every two days as it did up until 2003, http://techcrunch.com/2010/08/04/schmidt-data )
[update]
Spansion quotes 20 years minimum. Under optimum conditions (whatever they may be) and with an adequate amount of redundancy, 50 years should be achievable.
Spansion single-bit-per-cell floating-gate flash devices are designed to provide 20 years of data retention after initial programming when exposed to a 125° C environment. Spansion two-bit-per-cell MirrorBit flash devices are designed to provide 20 years of data retention after initial programming when exposed to an 85° C environment. Both MirrorBit flash and floating gate flash are guaranteed to provide 20 years of data retention after initial programming when exposed to a 55° C environment. MirrorBit flash is guaranteed to retain data for up to the minimum guaranteed cycles (10,000). [F]loating gate is guaranteed to retain data after the guaranteed minimum of 100,000 cycles.
http://www.spansion.com/Support/AppNotes/EndureRetentn_AN_A0...
If I was to store on magnetic media, I'd do it in a way that allows for some data loss (like usenet does with .par2 files). If you can stand to lose some of it, just pad it with enough redundancy for recovery and you'll be fine.
I think that this is luck; I found a batch of 8-year-old floppies a few years ago, and more than 2/3 of them were unreadable.
Edit: answered my own question. There are several kinds of toner (I had never heard of liquid toner), but some kinds are just fine:
"Toners composed of stable resin materials and a stable pigment such as carbon black are capable of strong bonding to the paper surface. Copies using these toners and printed onto permanent or archival quality paper can be considered permanent and suitable for long-term storage." -- National Archives Australia
Ideally some kind of artificial intelligence would come about sometime in the future to assume the role of data keeper - hiring people to do any work it couldn't do from within the computer and running off some kind of fund that had been set up. Maybe one day there will be a market for creating intelligent services like this, I hope I have something to do with them.
Start by looking at these guys: http://www.falconrak.com/pro_archival_cd-r_gold_ep.html
Even if it didn't, if you compare the costs of many other storage techniques, they're probably equivalent to jerry-rigging a CD player to read back these disks. The difference is the cost is shifted to the reading and not the storage.
Your main risk is the shonky assumptions of the "archival" CD manufacturer. Not that I know what those shonky assumptions are, but I have a vivid memory of the hosts of Tomorrow's World demonstrating the durability of CDs by spreading jam on one, then wiping it off.
Also I'd claim that your 286 has higher build quality and it's less sensitive than modern computers.
I've heard that the LDS church / Vatican have both been interested in the archival media, and they have a pretty good long perspective, so might be worth checking with technologists in that realm.
Optical media is the only way to guarantee whichever future "creatures" encounter the data, can actually figure out how to access it.
If you are a blind alien (so that you do not understand the _concept of light_) you can potentially still have measuring equipment that can sense the pits in the media and make sense out of the binary data.
If so, you could maybe give use some more information on the constraints involved(although i must admit thinking about it without any constraints is fun, too)
The constraints were chosen in order to remove the easiest answers (file sizes, period of time, etc).
Ultimately I think it's an unsolved problem that will become more important over time. My family can has photo albums from over 50 years ago but that doesn't have the kind of bandwidth we need for larger datasets (audio, video, etc).
So I guess it's just a thought experiment I thought was interesting.
Assuming there is no cost limit here, I would go one step further and say use some form of metal. Say stainless steel, aluminum, gold, or titanium. Some metal that is very stable over time and does not interact with the atmosphere readily. Again, use micro-abrasion / carving technology to write data to the materials.
The next question is what format for the data? It depends on whats being stored. The biggest issue is that of "formats".
Lets look at things that last a long time. The English(or any language) is unlikely to change that much in 50, 100, or even 200 years. Words and their meanings will change, but for the most part a native English speaker 200 years from now could read what we write now. Whether or not they understand the usage of the language is a different question. So if its a textual document you're saving, write it in plain English. No abbreviations, etc.
What about media? That gets complicated. If its a static image, perhaps keeping it simple is best. In plain English, write that the following section of data is an image. Each group of three numbers starts at 0 through 255. In procession from left-to-right, the values represent Red, Green, and Blue. Each group of 3 numbers is what we called a "pixel". The image is 300 pixels wide, and 800 pixels heigh(arbitrary numbers for this argument).
For moving images, further expand on the single image description and say every 24 images should be spaced equally and viewed over the course of 1 second to achieve animation.
Sound is something I don't know anything about from a data format perspective, but I would again find the simplest mechanical way to produce a sound and store it in that format with ample verbage describing how to handle it.
Edit After reading other responses that came in while I was writing this, I want to add some more thoughts.
Remember that our technology is ephemeral. We don't really use much tech from 50 years ago, hardly any from 100 years ago, and it just gets worse from there.
Things like microfiche, ssd's, cd-roms, blue-ray, etc are all the more bad ideas for long-term backup. Paper books are a better option than any of these for near-term storage for time periods up to 50 years.
If we want to actually store data in a meaningful way for long periods of time, say over 100 years, we have to keep it simple. Your devices will probably not last 100 years, even if kept in storage under the most secure environment. But in 100 years people will still have eyes, ears, and hands.
We have to look back over history and look at the material types that survive long periods. Stone, and metal to an extent, are very good long-term materials. Cloth and paper are not*. DNA is potentially a usable data store, but is corruptible. Plus you can't read DNA patterns with the eye.
But as you say, stone and metal as well as acid-free paper made from rags or mulberry fibres have a proven track record for longevity.
Printing presses, microfiche, film, radio, telephones, television, computers, light bulbs, electrical sockets, toasters, cars, bicycles, electric stoves, speakers, microphones, projectors, etc.
Technology is not nearly as fickle as you'd think. Remember, 50 years ago is only 1960. VHS tape technology is almost 40 years old. Sure, things are moving faster now, but things that work tend to stick around.
Actually that alone is no proof that clay tablets are very durable. Who knows, there might have been billions of them in circulation, and only a few of them survived. That would be a rather bad track record.
If 1MB is too much, try 1KB, and repeat for 5 billion people. There should still be some people left for systems maintenance.
In other words, the reason we don't use (say) the MD5 hash system is not because computers are able to break the 32-bit hash system, but because people have discovered flaws in the MD5 algorithm that means it doesn't give '32 bits of strength'. In this case it's not the hardware (CPU clock speed) that gets better, but the software (programmes that break MD5) that gets better.
Hence, you could use Moore's Law to predict what computers in 50 years time will be like, but you can't know what mathematical techniques will be like in 10 yet alone 50 years. Your encryption system you use might get broken in 10 years time.
But even when it was published, people were saying a 56-bit key was too small, which they aren't saying of modern cryptosystems (to my knowledge).
http://en.wikipedia.org/wiki/Data_Encryption_Standard
For preservation, encode the data in the DNA of a bacterium, replicate it massively and put them in some kind of suspended animation (hand waving wildly).
Getting the meaning of the data is another matter. I wonder whether it will be possible to create species of bacteria that can decode the above DNA and present it directly accessible to human senses - ex: bacteria that change color, form shapes, etc.
http://www.utheguru.com/fun-science-how-many-megabytes-in-th...
I also think things like NASA datasets, other govt agency datasets, etc, should be placed on torrents for anyone who wants to make a copy. Let the self-replicating nature of the internet serve as the backup backup plan.
If you put those Apollo datasets online, it's a guaranteed certainty that some hacker somewhere will have them in 50 years.
How would you store an ever-increasing amount of digital data indefinitely?
[1] Charles Stross describes it: "Memory diamond is quite simple: at any given position in the rigid carbon lattice, a carbon-12 followed by a carbon-13 means zero, and a carbon-13 followed by a carbon-12 means one. To rewrite a zero to a one, you swap the positions of the two atoms, and vice versa." See http://www.antipope.org/charlie/blog-static/2007/05/shaping_...
1. Get a 9600 bps modem. Use it to encode your data, and record the output as an audio file.
2. Take this audio file, and split it up into 60 minute segments.
3. Record these 60 minute segments onto two-sided vinyl LPs, 30 minutes per side. This will take about a million LPs.
4. Print on acid-free paper, using ink that will survive 50 years too, instructions on how a 9600 bps modem works. Describe the encoding in detail, sufficient so that someone using the equivalent of MATLAB or Mathematica or something 50 years from now on the computers they will have then could easily write a program to decode a modem signal.
5. Also print and include instructions for making a record player. As with the modem, the important part is describing how the signal is encoded on the LP. They'll have no trouble building a record player 50 years from now. (Assuming they don't just photograph the LPs with the 3D terapixel camera on their iPhone 54, and then write an app to extract the signal from the photo...)
5. Store all of this somewhere. LPs will last 50 years easily in a typical office environment, so you probably don't have to resort to something like a hermetically sealed vault buried in an old salt mine or anything extreme like that.
http://en.wikipedia.org/wiki/Pete_Shelley
It is software which, when loaded into the Speccy's audio In port, does funny lightshows in time to the album's songs.
BTW, the LP speed doesn't matter, the Speccy picks it up anyway.
Pretty innovative. And the best thing is you can now get the LP in a TAP-style emulator format! So it survived over 25 years.
I don't think 'use rackspace /and/ use amazon' is a good strategy either, just 'cause if market conditions change enough that one goes under it's likely the other will to... and it's also likely that they could be bought by the same entity. you'd want to put one copy on s3, and then use some completely different storage method, like microfiche or or archival CD roms or something in a safety deposit box.