Ask HN: How do you plan to archive all your digital content?

45 points by ezconnect ↗ HN
I have been wondering on how I could preserve some of my digital photos so my son could access them when I am long gone. I currently have a mix of iCloud and OneDrive but those requires account logins and I couldn't find a convenient way of like hey son here is all your baby pictures and all of our adventure when we where still living together. Any suggestions?

72 comments

[ 0.18 ms ] story [ 122 ms ] thread
I have two pocket SSDs that I archive to on an irregular basis (basically quarterly) of selected documents and photos. Each is in a small faraday bag, whichever one is in the house is in a safe, the other is in a safe deposit box at my bank. The drives themselves are unencrypted but some of the documents are encrypted (e.g. I export my 1Password vaults quarterly and encrypt those). I don’t have any kids nor plans to have any but have lost about 30% of the digital stuff I thought I was keeping safe (through bit rot, fading CDRs, regrettable document retention policies I was unaware of).

My advice would be to cut selected items to an SSD on a routine basis and either stash it in the family safe or wherever you keep important files. Use an existing filesystem, not whatever the latest newfangled vendor specific one, and use a fairly well documented format for images. Store documents as text or rtf.

Just be aware that SSDs are not stable long-term stores - they need to be periodically powered up or they will eventually lose data.
Am hoping that monthly/quarterly use is enough but that’s a good point to research.
From my own experience that's fine - I mentioned it more as a general note for anyone thinking of using SSDs for archival. I know someone who left a computer unplugged for a couple of years who lost all their data; I myself have successfully stored SSDs for a year or so without issue.

Anyway as you say, worth doing your own research.

> ...Each is in a small faraday bag...

Now that is a great idea! I do something similar to you with the exception of using faraday bags. Thanks for sharing!

I don’t know that it would shield from another Carrington event but figure it can’t hurt. Am assuming I will have more important problems in the event of rapid dispersal of nuclear radiation.
I'm sure you are right, but it's an easy extra step. Thanks again for the idea!
E-Books:I use zotero to store all my books and notes. Using webdav to back up data to nas.

Photos: I will manually copy the photos from the drone and DSLR to the nas and format the TF card after copying.

Phone photos: I use the software provided by my nas service provider to back up my photos

Web pages: I run my own docker version hamsterbase on nas. Store all the pages I am interested in.

NAS and SSD, multiple copies. I don't know if I'll get to the point of archiving on Amazon Glacier or something similar but that's always an option. Having just gone through a funeral for my dad this summer I can say that having non-digital images (photo prints, negatives, slides) doesn't always yield the most durable image. Compared to the possibility of storage devices failing I think digital images will be far more durable than previous analog formats... having those in a format and location where the next generation will know to find them is another matter. (For example: how long before the images I burned to archival quality DVDs cannot be viewed because no computers ship with optical drives and external optical drives are getting harder and harder to find?)
3-2-1 backups haven't let the world down.
Keep it simple. No raid or weird filesystems, no special NAS computers, no encryption, no remote services. I stuff every computer I assemble with hard drives until it can support no more then mirror important data from every computer to every other computer with rsync.
But, are all those computers in the same location? Just one fire away from losing everything? I have all my photos and videos on local storage, but it is all mirrored to a secure cloud storage service that costs hardly anything. I’ve pulled the backups from that cloud service (you can’t access the files directly) to make sure it works properly.
> a secure cloud storage service that costs hardly anything

Which is...?

Crashplan

I’ve also heard good things about Backblaze.

I'm really curious to learn which cloud service that you use which "costs hardly anything". Would you kindly share?
Crashplan.

I’ve also heard good things about Backblaze.

Thanks for sharing! I don't know why i thought crashplan went out of business (i guess i must be thinking of another provider). Now, looking at their pricing, its really great! Thanks again!
Well… they used to provide services for individuals, but then they decided to drop that and only support businesses. Since I was under a multi year contract, they moved me over to a small business account and they have let me stay there ever since.
I think the best way to hand that stuff off to your children is to wait until they get to a point where they hand off can happen, when they are adults, and then hand them a highly curated set of photos on a USB drive and let them do with it what they will.

The key is in the curation though, it is unlikely that your child will want to sort through thousands of blurry, duplicate, or unrelated photos. 100 really good and special photos/videos are better than a mixed up mass of 5,000.

If you are concerned with passing away before your son is of age, just make sure there are multiple backups, and the sign in information for anything that isn't local is communicated to his guardian.

> The key is in the curation though, it is unlikely that your child will want to sort through thousands of blurry, duplicate, or unrelated photos. 100 really good and special photos/videos are better than a mixed up mass of 5,000.

Honestly, I would really enjoy sifting through 5,000 photos from my grandparents or parents. (Sometimes the best photos are the embarrassing or private photos, like the time my dad asked me to help back up photos and I saw my mother trying to recreate a famous album cover.)

I appreciate the curation too.

I think the act of developing film meant that photographs had a higher signal to noise/duplicate/blurry factor. It's trivial to take 100 pictures in an afternoon, with only 1-2 being worth saving.
By letting it go. I used to horde, multiple formats and offsite... then I just let it all go. I've got a local copy of files on my machines plus a backblaze copy, that's it, I don't need tens of terabytes of data I'm almost certainly never going to use.
I use PhotoSync on my iPhone to copy the original photos from iCloud to my Mac.

Then I have two 5TB LaCie Rugged drives that I just attach to the Mac. Finally, using ChronoSync to schedule copying contents from the Mac to both drives as mirrors.

I simply print the photos I want to keep, and put them in photo albums like my parents and grandparents did before me.

I don't expect my children to be able to explore my digital archives when I'm dead in 50 years or so, so anything I want to be found then I print or otherwise place on a physical medium that's readable without computers. The less important stuff can stay archived on hard disks as plain files on a FAT32 filesystem. With a physical label on the disk describing the contents.

Yeah and I think people overestimate how much interest their kids will have in old photos of their ancestors.

I cleaned out boxes of photos from my late parents house. Lots of pics from my childhood years, somewhat sparser selection of older photos. Never look at any of them. I have a framed photo of my parents on a shelf; that's enough.

I also basically stopped taking photos of stuff about 15 years ago when I realized I never go back and look at them again.

Bottom line, I don't have much digital content to archive, and I plan to keep it that way.

Pro tip to parents out there:

Your kids will want to see other people in those childhood photos, more than they'll want to see 100,000 photos of their own faces. You, when you were younger; their now-gone grandparents; that friend they had for two grades before one of them moved ("OMG I totally forgot about them!"), that kind of thing. You'll get way more than enough photos of your kids, so far as your kids themselves are concerned, without even trying—it's the others you may neglect.

Now, for your own purposes, yeah, you probably want the photos of your kids more than anyone else who might get in the shots—but try to think of them and snap one or two of other people in the room, when you're taking e.g. birthday photos, or pan around from time to time when shooting videos of them. They'll eventually get more enjoyment out of the stuff, if you do.

Counterpoint, your grandchildren will appreciate pictures of their parents as kids.
Another (much more obvious) tip: pictures with people in them are almost always more interesting than pictures of nature/buildings/objects/etc.
yes yes yes! not only that, the graduation ceremony won’t be as interesting as a casual day at home. What are the kids playing with? what’s on TV? or on the computer screen or phone (that’s going to be vintage in 20 years). Cars on the street and signs on shop or on buses. The dull daily stuff can become super fun to look back on.

source: me looking at family videos from the 90s

Yep. I could take a million pictures per day of my cats because they're cute, but those aren't going to be interesting even to me in a couple of years.

I actually try to take what others might consider 'bad' pictures. Someone mowing the lawn, cleaning the cabinets, sweeping the floor, parking at the grocery store, working on some hobby or another, etc. These kind of pictures show "what were we actually up to in those days".

I've installed iCloud on my Windows box, with iCloud photos enabled. Then I right-click the iCloud Photos folder and select "Always keep on device." It takes a bit but everything gets downloaded. I then have a script that makes a zip of all photos monthly with a timestamp for the title.
I have all my private data backed up appropriately on a NAS, in the cloud, etc. When I'm gone, whoever has access to my accounts, inherits my stuff, buys my property an estate sale, etc. can do with it whatever they please.

However, for the digital content I have published, such as our podcast, I plan to form and fund some sort of trust that will be tasked with keeping it online and publicly available for as long as possible. This may just mean making sure it is archived in at least one library, the Internet Archive, etc.

I am pretty simple when it comes to archiving data. I archive my blog posts and writings using archive.ph. For storing other type of content (images, etc.), I use Icedrive[0], which provides enough storage for me to store important images and other data.

[0]: https://icedrive.net

resilio sync (aka BTSync) on my iPhone to send all pictures to a dedicated server in the cloud, where I store a copy of all my stuff.

Format: a simple folder tree /Pictures/YYYY/YYYY-MM-DD-Album-name

I recently zipped all my photos ever into one ~400gb file (along with a .txt file list). I have the photos 'online' at home and a copy of the zip archive at home, but the cloud copy is in AWS where I pay about $0.40 a month to keep it there in Glacier Deep Archive. i plan to repeat the exercise every 2 or 3 years, building up a set of zip files. At that price i can treat it like a 'fire and forget' dustbin of stuff that I'll probably never have to access...
Printing photos is the best way, I think (if stored in a dark place).

There are also M-DISC disks and drives (archival quality BluRay with some specialized layers chemistry). Since they are positioned for long-term archival, I expect there will be some need to read them in the future, so some sort of drives will remain available.

Im surprised more clouds dont let folks put credit into their accounts. I used a smallish openstsck based provider LunaNode for a little while that had a "wallet" one had to pre-pay into... conceivably one could just pay-in-advance the estimated rate for the next couple hundred year. You'd have to find a provider you trust to stay around.

We can iterate from there. Wouldnt it be sweet to be able to let other people dump more credit into your provider wallet, so it could keep going? Maybe we could standardize a "end of service" webhook/notification, and your website & content could jump ship & re-instamce itself on another provider automatically if your current one declares it's going under.

Dont forget to setup after-life DNS too!

And what happens when LunaNode goes out of business or gets acquired and the CEO posts an “Our Amazing Journey” letter on their website?
I just keep them on a local fileserver, and do backups of that as normal.
Local, as in your own home? What happens when the home burns down and you aren’t there to grad the backups?
You lose them?
I call my buddy who lives 20 minutes away and tell him I need my backup drive back that I keep in his safe and swap every few months.
My backups are not in my home.
Same local NAS, where backup = encrypted in the cloud, and also on M-Disc blu-rays kept in my office (where I also keep two BD units which I test like once a year). But I love the other idea of printing the best out, I'll get to that for sure.
Homelab with ZFS and ~60tb racked. It backs up to a second box in the house. I occasionally swap drives with a nearby friend.

Homelab runs photoprism (open source google photos alternative), plex, code-server, etc.

And what happens if your house catches fire?
> I occasionally swap drives with a nearby friend
On this topic, does anyone have recommendations for good and relatively affordable home NAS hardware? A lot of the ones you can buy with all the bells and whistles included are quite expensive and the cheaper ones out there look kind of shoddy. I bought a little kit for turning a Raspberry Pi into a NAS device using a SATA hat but the connection seems dodgy or something as it doesn't recognise the hard drives persistently. Shame because it looked quite good.
You can check out the data hoarder subreddit, which often has good suggestions.

My current setup is Ryzen on an AsRock motherboard (let's me use Unbuffered ECC RAM), and you use LSI HBA cards (see art of server on EBAY) which gives you more SATA ports.

Still in the range of $500-$1000 (no HDD) depending on how you spec it out, but far more powerful and expandable for anything even close to that price range.

If you are going for something in the $200-300 range, you can likely take the same path, but will make compromises in e.g. motherboard choice that won't scale well over time (but can still do a basic 32TB file server with ease).

Would you mind posting your AsRock hardware?

I was considering AsRock x300 mini desktop, but it supports only 2.5” HDDs. I am looking for two 3.5” HDDs, one NVMe and ECC RAM, low power CPU, around $600 or so.

I am using the B550 Steel Legend (6 full speed SATA ports, which is perfect for raidz2 in ZFS), a Ryzen 5800x (you can't use the g variants with Unbuffered ECC), and my RAM is Kingston ECC Unbuffered DDR4 3200 Model KSM32ES8/16ME.

You will need a dedicated graphics card, thankfully they are going down in price, if you are going to run Plex/any kind of transcoding, I recommend looking up "NVENC Encoder Matrix" and finding a card that support h265 and B-Frames (Turing or later).

Biggest cost is easily the CPU, a 5600x will run you $190. You could try a 3600 for $150 but I have not tested that with the ECC RAM, which is completely undocumented what kind of support is there it seems.

You probably want the 5600x anyways, I was able to overclock the ECC RAM up to 3866MHz and set my infinity fabric FCLK to 1933MHz (the 3600 can't reach that really at all). It's only like 5% more performance though.

Might need to look at used parts (Mobo, CPU, GPU) to get it for $600, but I haven't looked up prices and tallied it up.

Also as a side note, you can't run anything headless (on consumer motherboards), it needs a GPU to post.
I have a RAID5 over a set of 3 internal SATA drives, then an external enclosure with 5 drives (RAID5) for backup connected to the same machine via eSATA, where an rsync job kicks off every night to sync between primary/backup volumes. No off-site backup, though, which I know is a single point of failure which I hope to remedy at some point.
I've been trying a different way of recording for several years now. Have no way of knowing whether it will be better or worse than photos, but ... Honestly don't care because it is fun for me.

When some interesting thing happens with the kids, I shoot an email to their accounts in which I write the conversation, the way I felt, how we admired what they did, what we learnt from them and such things.

Ardrive is interesting in this space: distributed storage with the promise of one-time payment sufficient to cover storage cost permanently. Every other solution I’ve seen requires ongoing rent, or ongoing stakeholder interest+capability for preservation without financial incentive
I was sad to learn that many of my IG stories (and presumably applies to reels too) have broken music links (even though the item is still in the library) . That means I cannot rely on IG to adequately safeguard my memories.
You shouldn't rely on any company to adequately safeguard your important data in the long term.
I have media going back 50 years. A common theme is all of it becomes unreadable after a while - not because the media degrades, but because you can't get a device that can read them (or it becomes very expensive to obtain such a device). Punch cards, paper tape, magtapes, 8" floppies, 5.25" floppies, PC hard drives, cartridge tapes, DECtapes, zip discs, hard drives, etc.

The easiest by far solution is to simply copy them forward to new media once every year. Have an offsite copy.

I don't trust any cloud backup. They can go dark at any time for any reason, you can lose your password, you can get "canceled", a hacker can ransomware it, etc.

I periodically check that ransomware hasn't infected my backups by seeing if an unrelated device, like a media server box, can read the files on it.

I have no faith in "archival quality" media, as if you can't find a device to read it, it's worthless. Have you noticed that computers increasingly don't come with CD drives anymore? Why would anyone want them, since a 64Gb USB stick goes for $6? Soon there will be no way to read those "archival quality" CDs.

In going through my Dad's papers, I found a listing of the first game I wrote (in DECSystem 10 Basic). I thought that was lost in the mists of time. It was fun reading the ancient code. I scanned it and put it on my backup drive.

I threw away my old punch cards, paper tape, magtape, DECtape, etc., as it wasn't worth the effort to find a machine that could read them.

3-2-1 seems to be a reasonable strategy for keeping the data relatively safe over the long time and maintaining some offsite backup saves it in the event of house fires and the cloud is convenient for that it just can't be the only storage. But the bigger issue is often conversion of data, there is often a window in which a format can be converted from an old format to a newer or more popular one and you often have to assess that yearly.
I tend to archive things in the most common formats, and haven't had much trouble with it.

For example, in the 80s I'd use various compression programs. When I eventually archived them, I uncompressed them with the old compression programs.

I store pdfs with a .txt version of their contents, too.

I would assume that outboard CD/DVD drives (USB, maybe others) will be available for a long time to come.

3.5 inch USB floppy drives are still routine. 5.25 inch, try the used market. Who knows what happens in 10 years?

That's for recovery. I wouldn't use floppies as a primary medium these days. CD/DVD might be on OK choice.

5.25 drives were once so ubiquitous, I am astonished how difficult it is to read a floppy these days. First, you have to find a drive - ancient used ones only. Then, you have to find an interface to it, because nobody makes them for modern computers, and BIOS support has been dropped over a decade ago.

CDs only hold 800Mb of information. When you can get a 64G usb stick for $6, who is going to burn CDs anymore?

At this point 5.25 reading is a data recovery task.

Re USB:

What's the life expectancy of a USB stick? I've had friends get unpleasant surprises when USB sticks were unreadable after a few years.

Are some brands better than others? High capacity vs. low capacity.

A bigger question is indexing that content. I would pay good money for a software suite that would let me scan all my old family photos and attach both text and audio. I'd love to sit down with my mother and one-by-one go through them and let her record the story behind them. I don't understand why such a format doesn't exist.
Put them in your PowerPoint alternative of choice and record narration?
That doesn't let you organize them by topic or by the names of people present in the photos, though. If I wanted to find "every picture with grandma in it" I'd be SOL.