Ask HN: What's the most stable form of digital storage?
I wrote a program which I'm proud of having done and would like to keep it for posterity. What's a good storage medium where I can keep and load again in the future? Requirements are: size < 1GB, must keep for at least 3 decades, must be easily transportable (for moves between houses and such) and can sit on a shelf. Bonus points for suggestions on an equally stable storage type that some computer will still be able to understand in the future.
238 comments
[ 5.4 ms ] story [ 306 ms ] threadUSB flash? HDD? SSD?
> must keep for at least 3 decades
Most storage mediums won't have problems lasting that long if they're stored well. Particularly flash and SSD storage shouldn't take issue: tape and HDD has a small chance of mechanical failure even if you don't use them.
> must be easily transportable (for moves between houses and such) and can sit on a shelf
I mean, I think most of our current formats do just fine in that regard.
> Bonus points for suggestions on an equally stable storage type that some computer will still be able to understand in the future.
Almost everyone is still using SATA. There is unlikely to be a world in the future that is unable to "understand" serial ATA storage.
I expect similar problems looking forward. Hard disks made today aren't rated for 30 years, NAND flash retention is much less than 30 years, and Blu-ray drives may be hard to find.
Hard failures aside, I assume very slow charge decay might be somewhat mitigated by periodically copying files flash-to-same-flash (let alone another) -- or perhaps even rewriting them in place.
The best answer is "host a NAS that you actively maintain and has an offsite mirror".
I personally host an all-flash NAS for the performance and MTTF benefits over HDD. Since it's always on and performs regular scrubs there is no concern of bit-rot. The data I put there will be there as long as I'm around to take care of the box and pay the B2 bill.
That said - I'm at a loss to understand what software is ultimately worth using without modification after 30y....
I wrote a video game for my child’s birthday. Does that count?
For your child 100% so if that´s the goal why not?
Modification is a big word. Old cars can still be valueble in capital and make you feel much more alive then a brand new car.
might not spin up. I recently booted a Sun 3/80 from 1990 and while the battery backed RAM that told it how to boot was long dead, the HD was not a problem at all. Once I replaced and re-coded the RAM, the machine booted right up. And this isn't just a one off story. I've got a bunch of machines from that era, and the HDs are not usually an issue (although occasionally they are).
About half of them still spun up and were readable. Probably could have gotten more with heroic measures.
Most 3.5 floppies were likewise readable.
> Most storage mediums won't have problems lasting that long if they're stored well. Particularly flash and SSD storage shouldn't take issue: tape and HDD has a small chance of mechanical failure even if you don't use them.
I think you're overstating the durability of these systems.
I saved most every HDD I've used dating back to the early 90s. I wanted to get rid of the box of drives, so I got an adapter and tried to see what I could salvage from them before throwing them out. Ages varied from about 8 to 28 years. None of the ancient drives (>15 years) worked. I could usually hear them spin up, but I couldn't get anything off them. Several had the dreaded click of death present. I got some of the newer (<12 year old drives) to work. Even the newer 8 year old drives had really high failure rates. They were all working when I pull them out of service.
My old USB flash drives have also had high failure rates for me.
From my experience, expecting an HDD that isn't in use, to go for more than ~5 years totally idle is asking to lose the data on it. The data might still technically be there, but the mechanical+electrical system that makes up an HDD does not have high longevity.
To the original question. I actually just researched this topic extensively to make purchases for my back up systems and my conclusion is: If I really want data to go 30+ years. I'm using archive grade Blu-ray discs (M-disc and LTH). I burn multiple of the data I want, and I store them in dark places at separate locations.
If you must use flash, keep it cool. We have some deer cameras in direct sunlight. After six months or so, the SD cards become unmountable due to media read errors.
When I got married (20+ years ago) I burned a CD full of music for the DJ. Songs that were not likely in his/her collection.
I found the CD in a box about a year ago, and it is currently in my car stereo. All the songs are in tact.
Oh.. My xbox also reads and plays the music.
My main comment has some hard numbers based on my own experience: https://news.ycombinator.com/item?id=30889474
In short: 77% of the discs I ripped after ~15-18 years had minor issues as demonstrated through read errors (when ripping with ddrescue) and checksums (in png, zip, and gz files; I did not manually checksum files back in the day). I was able to rip 99.88% on the worst disk, and I estimate that about 97% of the files are intact on that disk. CDs and optical discs more broadly seem to be reasonable, but I'm not really sure they'll be in good shape after 30 years. I'd consider them part of a backup strategy, not an entire backup strategy. I personally use a combination of external (magnetic) hard drives, flash drives, and optical discs for backups. Replacing the optical discs after 10 years or so seems prudent, as is redundancy and/or par2 files.
I go with simple external hard drives. I get a new one every couple years and copy everything to it as my main backup. The likelihood of my in-machine drive and fairly young backup drive failing at the same time is pretty low. Even if that happens, the previous backup drive will have a good portion of it. And my important docs folder also goes to Carbonite online backup.
If I had to save 1 TB I'd probably save 1 copy with a specialized storage provider, 1 hdd in a safety deposit box and 1 hdd at home. You could also use tapes if you don't trust the HDD hardware to last.
https://brianhenryie.s3.amazonaws.com/2022/kodak-gold-cdr.jp...
35 mm film is also interesting but probably costs a fortune: https://www.piql.com/services/long-term-data-storage/
https://www.monperrus.net/martin/store-data-paper
I mostly jest. It would require a lot of paper. But it should be stable storage for potentially centuries.
For a single copy long term paper or etched metal is probably the most reliable.
Now what is the most density you can get on standard paper? That’s a more interesting question.
Probably some collection of QR coded, multiple copies maybe.
Real talk: every 6 mos when checking fire alarm batteries check your storage (and as necessary migrate it to copies on new cloud systems etc).
I wonder if printing microfiche is something you can find easily.
My strategy changed, though, with the arrival of cloud storage. Now I have all of my files—about a terabyte and a half—synced to multiple computers through Dropbox, and I back up each of those computers to an offline hard disk every ten days or so. My main computer is also backed up continuously to an external hard disk.
File formats are a different problem. Until around the end of the 1990s, I used a series of different word processor, spreadsheet, page layout, email, and other programs, all for the Macintosh, and I can no longer open a lot of those files. In more recent years, file formats seem to have become more stable, and I have tried to be careful to use formats that seem likely to survive as long as I will—docx, odt, mp3, pdf, jpeg, and especially txt.
Most of my oldest documents were Wordperfect 5, so they were still readable. I have converted them to txt (accessibilty) and pdf (preserve formatting).
My biggest miscalculations were using CD-RW media during a period of time in the early 2000s (the media often cannot be read by CD drives the disc was not created on) as well as my choice to use cheap CD-R media which over time the plastic became cloudy, or the reflective layer oxidizing and then flaking over time.
It costs the Internet Archive $2/GB to store content in perpetuity, maybe create an account, upload your code as an item, donate $5 to them, and call it a day. Digitally sign the uploaded objects so you can prove provenance in the future (if you so desire); you could also sign your git commits with GPG and bundle the git repo up as a zip for upload.
EDIT: @JZL003
The Internet Archive has their own storage system. I would assume it caps out because they're operating under Moore's Law assumption that cost of storage will continue to decrease into the future (and most of their other costs are fixed). Of course, don't abuse the privilege. There are real costs behind the upload requests, and donating is cheap and frictionless.
https://help.archive.org/help/archive-org-information/
> What are your fees?
> At this time we have no fees for uploading and preserving materials. We estimate that permanent storage costs us approximately $2.00US per gigabyte. While there are no fees we always appreciate donations to offset these costs.
> How long will you store files?
> As an archive our intention is to store and make materials in perpetuity.
https://archive.org/web/petabox.php
Quick envelope math, if they were using backblaze pricing, $5 would give a gb 83 years of storage. But it's unclear if backblaze is actually regionally duplicated
EDIT: Understood, thank you for the reply.
The market for tape is much smaller than consumer markets (in units sold)
You'll probably be able to read CDs in fifty years, just because there are so many of them.
If you have a fifty year old tape, even in perfect condition, you probably won't be able find hardware to read it.
If you have a fifty year old tape, even in perfect condition, you probably won't be able find hardware to read it.
As I found out when migrating a pre-LTO AS/400 tape, there are rental places known to IBM partners that rent the drives that can hook to modern equipment.
They’re special DVD and Blu-ray discs designed for long-term storage. DVD and Blu-ray are so widely used, it seems likely you’d be able to find some equipment in 30 years that could still read them.
I think the disc players are the weak link. I can definitely imagine them going away nigh-entirely in a decade or three.
Yes.
There are too many use cases for physical and immutable long-term offline storage for this niche to go unfilled, but the niche is too small (at present) to prompt the development of a replacement medium and format, so while I am sure that the materials and read/write hardware will continue to evolve (better data longevity guarantees, read/write speed, physical durability, etc.) the implementations will remain compatible, or at least the reading ones will.
"The material, a nanostructured glass disc, also has an estimated lifetime of 13.8 billion years (roughly the current age of the universe) at elevated temperature of 462 K (190 C), and a capacity of 360 TB. It has been hailed as a particularly significant invention, as no other existing storage medium can so safely ensure that data will be accessible by future generations."
https://www.guinnessworldrecords.com/world-records/412399-mo...
https://www.arweave.org/
https://egyptianmuseum.catalogaccess.com/search?search=contr...
Though some are the only remaining replicas from RAID-1 (RAIT?) groups.
Consider adding LDPC or something while you're at it.
https://sketchfab.com/3d-models/ancient-mesopotamian-stars-p...
The cost per bit of this data is mind boggling at this point.
https://en.wikipedia.org/wiki/HD-Rosetta
The only true solution is a living one, where you have make sure you have the ability to get your data from an old format to a new one periodically. More importantly, you should look into the idea of 3-2-1 Backups. Anything that you intend to keep indefinitely is subject random events, ie fire, flood, tornado, theft, etc. Having multiple archives in separate systems is more import than worrying trying to ensure a single copy will last a long time.
Storing less than a gigabyte is very cheap to do in multiple formats, such as USB flash drive, external hard drive, CD, BlueRay disc, etc. You can hedge against data corruption with PAR2 files. Also, consider storing a copy on the cloud, ie Backblack B2, AS S3, etc. Again, I suggest creating PAR2 files and/or using an archive format that can resist damage.
Just create calendar events to check periodically the integrity of your archives. Having problems reading a CD, use the hard drive backup to burn a new one. This also a good time to consider if one or more your formats is no longer viable.
Finally, realize that a program runs within an environment and those get replaced over time. You need to no only back up your program, but probably want to store the operating system and tools around it.
Coding is not the problem, the hardware is.
Use mainstream technology media formats for physical storage. It's trivial to get a USB floppy drive for reading floppies from the early 80's, but getting hold of a new drive to read LS-120 disks from the late 90's/early 00's is pretty much impossible. BluRay is probably the best bet for physical media for the next 20 years. I've done some trials with SD cards but they seem less reliable than BluRay.
Then come back to it at least once a year to run it again and make sure it still works.
At present we are lucky enough that Windows programs from 1990s still run under Windows 10 to some degree. Thank the folks at Microsoft for maintaining their operating system as a digital museum of archaic bug-compatible APIs.
Something to keep on the shelf would mean you ignore it for too long and it stops working.
Even something like a Python script may stop working due to changes in the language, and old versions of the interpreter no longer being maintained.
Yeah, you can do archival diamond, or archival ink on archival paper. But will you be able to read it 1,000 years hence?
I doubt there's a market for a printer, except as a novelty device. I wonder what kind of bit density you can achieve, in clay?
That said, etched metal or ceramic would likely be better. A CNC with a dremel would would let you digitally generate your tablets at ~$200 + tablet materials.
Cheap ceramic tiles are $0.50-$1 per sq ft.
But at the face value, it's hard not to wonder whether you'd like to preserve a functional program, or its source code, or its architectural and design ideas?
Either way, your current perception of the program is likely tied to the current technology or perhaps even whole ecosystem around it.
So to preserve something like that, you'd need more than just storage.
If it's just for the source code, as in text, then as golden rule of backup goes - keep many copies in distributed but known locations. In other words - diversify and distribute. Whatever storage digital, or analog, or organic, as in human memory (storytelling is a type of storage too).
Though, likely, you mean the functional program. Thus you'd need to preserve the platform too, along with build tools. So, at least some system specs need to be preserved or a VM image for more or less stable virtualization envs.
These ideas, as well as the appropriate external context, can be captured using any medium, be it analog or digital, since no one came up with a better way to convey it than text and possibly some diagrams.
It is part of the reason I consider well-written and well-structured high-level documentation more critical than tests or even functioning program itself. The latter are lossy, narrower representations.
[0] If there is anything novel as far as specific patterns and algorithms involved, they can be appropriately formalized and documented in isolation of the larger program and also in a language-agnostic way.
I had a teacher who was known as a master printer. Students would ask, "How can I make a truly archival print?" His answer: "Show me something you make worth keeping, and I'll tell you."
For another expensive approach, I suspect gold, like the gold voyager record would surpass it.
I imagine you could encode a lot of data in something like a QR code stamped in clay.
In other words, clay tablets get re-purposed too.
Bronze tablets will not break. Unlike stone or ceramic, they are slowly corroded, but we have well preserved bronze tablets which have survived at least 2200 years.
A bronze tablet, or better a stainless steel sheet, can be engraved with text using a computer-controlled mill.
Then again, selecting a physical carrier isn't going to ensure the longevity of data stored.
Far more poignant are questions whether data on those stone tables will be readable, let alone understandable. Ultimately, those tables will contain marks that - to us - signify 1's and 0's. If you want data to outlive us for just a couple of millennia, anyone far down the future will need to figure out which numerical system is used first. Then they have to understand the concept of binary representation of information. And then they have to understand the encoding.
Ironically, a JPEG picture of a cave painting stored on a stone tablet, is still far more brittle compared to actual cave paintings which lasted for tens of thousands of years.
Of course, other concerns such as curation and relevance of cultural objects stored as a digital representation also need to be factored in. Some stuff simply doesn't survive because later generations simply lose interest. The more time passes, the more longevity of objects solely hinges on simple happenstance and chance.
https://en.wikipedia.org/wiki/Long-term_nuclear_waste_warnin...
In which you end up with these types of proposals:
> The linguist Thomas Sebeok was a member of the Bechtel working group. Building on earlier suggestions made by Alvin Weinberg and Arsen Darnay he proposed the creation of an atomic priesthood, a panel of experts where members would be replaced through nominations by a council. Similar to the Catholic church — which has preserved and authorized its message for almost 2,000 years — the atomic priesthood would have to preserve the knowledge about locations and dangers of radioactive waste by creating rituals and myths. The priesthood would indicate off-limits areas and the consequences of disobedience
Meanwhile, there's Github's Archive Program which as this nice tidbit on it's roadmap:
> The GitHub Archive Program is partnering with Microsoft’s Project Silica to ultimately archive all active public repositories for over 10,000 years, by writing them into quartz glass platters using a femtosecond laser.
https://archiveprogram.github.com/approach/
Which sounds nice. But also unrealistic since technology alone isn't enough to ensure longevity. Embedding the longevity of data into culture could also involve founding a "digital priesthood" analoguous to Sebeok's "atomic priesthood". A group of people who safeguard and pass on the knowledge required to read and interpret artefacts containing digital representations of information.
This question could have an answer by now, but it looks like everybody is optimizing for cost/GB.