Ask HN: How do you manage your “family data warehouse”?

Safe deposit box at the bank for the physical records. A labeled manila envelope stored locally for each of us with the typical "identity" documents, SSN card, birth certificate, etc.

Everything is stored locally, unencrypted, and backed up on a local removable HDD, and on BackBlaze. BackBlaze has saved my bacon more than once.

As for physical objects, they're all temporarily mine, until I pass on. They're either tools, consumables, or tied to personal memories. Everything else can go.

The one significant thing to pass on to others is all the data in D:\source\yyyy\yyyymmdd*, which is all of my photos, including family events, short videos, etc. As best I can, all the faces have be tagged with XMP metadata. (Nobody wants to keep an old box of photos without knowing how those people are related)

     Total Files Listed:
           392934 File(s) 725,479,895,968 bytes
           16103 Dir(s)  1,790,571,331,584 bytes free
  
  D:\Source>

I need to discuss with my spouse and child what they want from there.

Fairly simple here. Everything goes in iCloud with offline backups via Time machine. Documents are scanned and disposed of where possible - collecting paper is a bad bad bad thing. Identity and required paper documents are kept in a locked drawer. Credentials are stored in Keepass. Photos and videos are all in Apple Photos. Instructions are left on how to deal with the offline backup should anyone need it.

Obligation wise it's good to keep all documents for whatever the longest record keeping period is in your region.

Important that you scan your identity documents and keep copies too. That has been incredibly useful when I'm abroad because a lot of people will retain passports etc if you go in a hospital or something so you can just show them the electronic copy and tell them you left it in the hotel.

Documents are in Subversion hosted on a local home Debian server which is also NAS.

Photos (hundreds of GBs) are also in Subversion hosted on same server. Photos are managed/viewed either via Digikam (on both Mac, Linux, Windows) as well as through a generated HTML/js gallery served on the same server.

Home server is backed up via restic to

* local harddrives that are rotated weekly and moved offsite. * to a rented physical Debian server in a different country * to a cloud storage provider

Server storage is fully encrypted using LUKS. Backups are encrypted via restic/ssh.

Various sensitive documents are separately encrypted inside the svn repos via 'age' or similar tools.

This setup has been working for almost two decades. I have yet to lose any data.

Edit: For documents, we have a personal repo for each family member. Typically, we use a folder structure based on https://johnnydecimal.com/

We also have a repo specifically for receiped in which receipts are just stored as /<year>/<yyyy-mm-dd>-<name>.pdf (or .jpg if it is a scan/photo).

Legal documents like wills, etc. are stored in a safe as well as a physical copy stored at a relative + digital versions stored in the svn repo.

All types of documents go into a paperless ngx. All documents are tagged properly so that I can find them again. Also the search worked quite well so far. I’m still archiving the previous years currently, as I set up everything this year.

Backups via restic end up on backblaze. Haven’t gotten around to set up the local backup. This is a good reminder to do so. :)

The photos end up in iCloud and Adobe with a backup in backblaze. Not so happy yet with this solution. But most alternatives I tried weren’t as comfortable.

Edit: The paperless ngx server is a low power machine that runs in my local network.

I live in a paperful country, and I have no scanner.

Important stuff has a photo taken, and it's likely backed up in my photo archives (Syncthing → Pi Storage → NAS backup).

My girlfriend still uses google and whatsapp, and I tend to send her important media in our chats, so we'd both have to be kicked out of our gmail and whatsapp accounts to truly lose everything there.

Dropbox for everything except photos, videos and security-related info.

I used to keep photos and videos on an external hard disk with a jwz-style backup mechanism and Lightroom 3 for organizing it (kept the light room database file on the hard disk itself)

Now just gave up and use iCloud Photos. Not as good as Lightroom but can't find anything better that's also easy.

Secure details go into 1Password with a printout of emergency keys in a wardrobe under lock.

I used to use Evernote but moved away from it soon after they started introducing chat and a bunch of crap no one cared about.

I keep copies of often-referenced items from Dropbox in Apple Notes but the master copy is a file in Dropbox.

I'm using the Shortcuts app on the phone to take a receipt from an online payment and file it away in the /receipts folder in Dropbox.

I used to have an IFTTT integration to take all the photos I posted on Facebook and keep a copy of them in a Dropbox folder as well. It probably still works but I don't really post stuff anywhere any more.

In Belgium, every citizen has access to a "digital vault" provided by the government, which is accessible using their ID card. Individuals can upload up to 1GB of documents into this vault. It's a well-designed system that ensures privacy during one's lifetime, but you can choose what happens with it after your passing.

https://www.izimi.be/en/

[dead]

I'm a bit shocked that all answers so far use cloud storage and not a single one proposes offline.

I store everything in the cloud, because it is unbeatably convenient except for the very things mentioned: health, financial and legal. I'm not 100% strict either, because if a document has been sent or received electronically or can be assumed to be stored in the cloud anyways I really can't do much. I still assume that every online storage will be breached one day and keep some documents deliberately offline.

To give you some chills, check out "Who's been pwned" [1] and this is only the tip of the iceberg that has surfaced and that fulfills HIBP's inclusion criteria.

[1] https://haveibeenpwned.com/PwnedWebsites

Apple solves this problem for me entirely. No need to tag documents etc, just iCloud everything -> everything is searchable.

Fortunately for me, most of the official documents here are digitized, and stored on government servers, so in the event of my death, my family can request access to relevant documents. That takes care of most of the official stuff. Also, everything legal i cosigned by my wife, so either one of us has equal access to it, but like official documents, these documents are also stored in government databases.

As for products, i have aimed to buy services that are easily transferrable, i.e. we use iCloud family sharing, meaning every user has their own storage, but they piggy back on my paid subscription. If/when i stop paying, they can chose to continue paying by themselves, and things will just "magically" keep working, but otherwise it's a tricky game.

As for memories, i store everything in the cloud. Most is encrypted by Cryptomator. I have a small server at home synchronizing data home in real time, and the server backs up data hourly to a local target, and a couple of times per day to another cloud.

I have thought long and hard about decentralizing this to let each users laptop do this, but with a family photo album of 3.5TB that would mean i lose deduplication.

It would also put much more demand on the client hardware, as it would need to be able to hold the entire photo library to back it up. I have LONG REQUESTED a way to backup iCloud data without the need to keep it all local, i.e. download upon use for backup, and delete again, but sadly that's not how it works. If your data is "cloud only", your backup will only backup file pointers, and not the actual photos.

Once every year i make Blu-Ray M-disc archives of the previous years photos, and to some lesser extent also documents. I make identical copies and store them in geographically separate locations. I use no encryption or compression on these archive discs. I also have a set of identical external USB drives that i update with the entire archive yearly and then rotate between locations.

I am fully aware that optical media may not be around for much longer, but that's a problem for future me. For now it is the best and most affordable "long term, no maintenance" storage technology available to consumers. Until an official "end of life" is reported, i will keep using it. Once it's EOL i can easily migrate my archive to "something more better", even if that ends up being just ordering photo copies.

I have a messy drawer where I dump all of this stuff. It’s basically write-only, once I throw something in there I know I’ll never look for it again.

For digital stuff it’s mostly the same idea. If it’s email (which is almost always the case), I archive and forget about it. If it’s something else, I’ll throw it in my virtual “messy drawer” which is just a folder in Google Drive.

You are getting quite a few clever ideas in this thread, but I suspect the vast majority of people out there in the real world do more or less the same as I do (of course, I’m biased).

My “philosophy” (overstatement) is that 99.99% of the time I’ll never need it. If I ever do, it might be a little painful to dig around, but that doesn’t justify having a whole system in place. I would rather keep my mess.

A Fuji Snapscan S1500 + folder of PDFs on my laptop + full text search. Mylio for photos. I sometimes purge and file, but not very often.

There's always a bit too much that is attached to emails, I try to remember to save stuff out.

A Synology NAS in the attic which backs up the above.

No encryption.

An £50 "fireproof" box for passports, citizenship certs, things where the original is important.

I assembled a ridiculous old server to do tape backups but couldn't make a very practical system - I wish I could attach an LTO drive to something smaller!

TrueNAS Scale (formerly FreeNAS) server with 6 x 4TB ZFS storage in a Raidz2 (or was it z3?) configuration.

I rsync my picture and document library from my MBP to the NAS. The MBP also has a Backblaze backup. 2-4 times a year I sync my pics, docs, and backups to a 6TB drive that’s offline.

Laptop documents folder and outlook inbox

A Synology NAS running Portainer (https://www.portainer.io/) running Paperless NGX (https://github.com/paperless-ngx/paperless-ngx)

This works better than I can possibly tell you.

I have an Epson WorkForce ES-580W that I bought when my mother passed away to bulk scan documents and it scans everything, double-sided if required, multi-page PDFs if required, at very high speed and uploads everything to OneDrive, at which point I drag and drop everything into Paperless.

I could, thinking about it, have the scanner email stuff to Paperless. Might investigate that today.

Paperless will OCR it and make it all searchable. This setup is amazing, I love living in the future.

OneDrive and a fire resistant box. I suppose I actually store far more in my emails than I should!

I've been doing the data storage tango for thirty years now, and the one constant is that everything requires maintenance or has to be moved to a different platform and software eventually. Today, everything is in Google Drive and Google Photos; about 600 GB worth of data and media. Maintenance consists of an annual Takeout ordeal with copies going on two different removable drives, one of which is stored in a so-called fireproof bag.

The fireproof bag also has my account/password list (printed out once a year), birth certificates, titles, and passports.

If you are the tech person in your family, think about what happens if you die. Will your family be able to figure out and maintain your system? Mine is far from perfect, but most of my family can figure out Google Drive.

We scan every document and put it in a folder synced with SyncThing. We throw the original away unless it is something official like a birth certificate.

We use SyncThing because it is free, doesn't require user accounts, and works across Mac, Windows, Linux, iOS (Möbius) and Android.

We have a bunch of high level folders (eg. Health, School, etc). Inside that folder every document is named starting with a date, eg: "2023-09-11 Invoice Dentist Kid1.pdf".

So far I've always found every document.

Having built complicated storage systems during work hours I had a strong urge to build a home NAS. What stops me is that I have this Documents directory on my laptop, which was salvaged from another disk that failed about three years ago, and served for 11 years prior to that, and some documents were copied to it from a PC I owned before then... and going farther back doesn't really make sense as none of the documents would be really relevant / authorities at the time couldn't deal with digital copies of documents, so, storing them made no sense.

So, in the end, I just keep them in the Documents directory. Some, most commonly used are also in my email. That's it.

Really important documents are paper and stored in a filing cabinet into labeled folders. Every so often we go through these and discard the papers we no longer need (old tax records beyond 7 years, utility bills more than 6 months old, etc). If our house burns down or floods badly, we'll potentially lose these documents, and that's a risk we're willing to accept. Pretty much all of these documents are actually replaceable with effort, time, and a little money. I can get new notarized copies of birth certificates, old tax documents, will, passports, etc if I really need to but it would be quite annoying. The risk of losing them and the annoyance even each other out, it's a very small risk. If I die, no one needs to know how to use a computer in order to access my will, taxes, etc.

Important digital files are all on our family computer which is backed up to a cloud service which will mail me a disk to do a restore if needed. This includes all our photos and videos from since we've had digital cameras. If both my wife and I both die, no one will have any clue how to deal with this and that's OK.

Photos we really like are printed and in photo albums organized by date. We'll lose these in a house fire and that's a risk we're willing to accept. The ability to easily pull a physical album and see pictures from a past time grouped together and curated is something we really like.

I use https://www.trustworthy.com/ and Dropbox and am very happy with this combination.

For me and now by extension the missus and kids: An organised chaos of diverse storage, for docs and photos. On a NAS, Google Docs, Dropbox, Flickr, variuos other local photo apps now offline (coppermine etc). All newer photos are on icloud though also backed up offline. A lot of older paper documents in physical folders, some organised but most just bungled together. Old physical photos are hopefully somewhere in the attic.

Most recent years all letters are however photographed on a phone and put into relevant shared photoalbums, then shredded. Albums by year and category e.g. "School docs", "Kids School work", "kids art", bills, statements, manuals and instructions, receipts, etc. Still have a big chore to go through all the old folders to scan and shred so we can really be paperless.

A few times I have had to dig out old bank statements or proof of address on a recent bill. However they all just wanted a photo of it, so storing it digitally was fine.

A few folks have talked about digitizing paper documents, so one of my favorite tricks for iPhone havers:

iPhones have very nice scanning tools built in, but it’s buried in the Notes app. Create a new note, click the camera icon, then “scan documents” and it will create a very nice, usually well cropped and OCRd scan that’s saved as a PDF you can then export elsewhere. Wish it was a standalone app because it works so well, and this is from someone who helps digitize and preserve paper documents for a living.

Apple solves this problem well! Note that I use iCloud family which allows me to share the storage with my family. But for going down the list:

A) Mint for most financials. iCloud for storage of documents.

B) Apple Health for keeping track of my records. For ad-hoc stuff I use iCloud to store the documents.

C) Product and service info I use iCloud for. Logins and credentials to them are stored in my family's 1Password vault.

D) All legal documents stored on iCloud.

E) For photos and videos I use iCloud Shared Albums. It's unlimited free online storage of photos. Otherwise it's a mix of iCloud storage and the Notes app on Apple devices.

Obviously you can tell I'm deep in the iCloud ecosystem but personally I feel it's worth it. The synchronization and ease of use for my family is well worth it. For what it's worth I'm on the 200GB plan.