Ask HN: How do you store photos and videos?
Problems:
0. I'm using a ton of local HD space to store them. I want to free up a bunch of space.
1. But if I delete photos from my computer, Arq will eventually delete the backups that contained them. Could be months or years but if they're not local, they'll be gone when the cost to store them exceeds the monthly budget I've set with Arq.
2. Additionally, Shutterfly apparently will let you upload videos up to 2GB. But I've randomly found a couple of videos well under that size that just don't get backed up.
I think this is what a good storage situation looks like:
1. Photos and videos are automatically backed up to the cloud from my devices (nice to have)
2. Photos are retained forever in the cloud even if they're deleted from my devices. I have to go to cloud storage to delete something.
3. Very nice to have: photos are encrypted locally before getting sent to the cloud like with Arq.
4. Bonus points: the UI for browsing cloud photos is nice (Shutterfly bought the company I was using, which had the best UI I'd found).
What do you use? Have you solved any of the problems I've laid out with my approach?
edit: thanks for all the great replies. I'm in the middle of a lengthy process of moving off of Google services, so Google Photos sounds great but isn't something I'm into. Will look at iCloud and other solutions mentioned below.
103 comments
[ 3.2 ms ] story [ 255 ms ] threadAlso it have a nice plus: being physical need a bit of attention on your parts, so you probably end up in conserving only really valuable stuff instead a monster-collection of thing you do not really value...
i have my family use Google photos. I have old family videos on there, and most of have Google Android phones that have data storage boasts.
No privacy guarantees, few guarantees of never having a sunset.
It works very well if you're all in in the Apple ecosystem.
https://support.apple.com/en-us/HT204264
But it really does Just Work. I’ve got basically every photo I’ve taken since ~2002 in a single archive, easily scrollable by date on any of my devices, searchable (on a map!). It works perfectly for me.
One day I hope Apple improves delta/library reconciliation, but for now, it's the best I've been able to find. It Just Works (for the most part).
I try to download the Photos data regularly via their takeout tool, but the process is cumbersome. I wish there was an automated way of doing that. I'd even pay for that.
I moved from Gmail to Fastmail, but I still haven't found a good replacement for Google Photos.
Storing all my photos online would require for me to get a very expensive plan. I do have offline storage of multiple external HDD where I regularly sync together.
I own a small photography business. Anything I really care about—almost everything captured on my DSLR or drone—starts life in Lightroom. After images are edited, and uploaded to appropriate online sites (e.g. Flickr), they are moved to an external 2TB drive.
The external drive is cloned to a separate drive to ensure I have a physical backup. My laptop and the primary external drive are backed up to CrashPlan.
It cost me $2700 to learn that one physical backup is not enough. When I bought my two drives a few years ago, they cost $99 each. That’s nothing relative to the value of my time/work.
Family photos are additionally rsynced nightly [one-way] to a Synology server (just because I had it before I built the FreeNAS storage and it has a reasonable picture browser app while on the home network).
zfs gives me copy-on-write (protection against crypto-locker, though I'd lose effective backup while crypto-locked) and availability even in the face of loss of 2 drives (RAID-Z2 ~= RAID-6).
I'm posting not because my solution is perfect, but because I'd like a better answer to client-side encryption for backup of legal docs, a better solution for video backup, and a better solution for browsing/tagging/auto-albuming accessible from mobile devices. I don't object to the current costs, though probably wished I'd skipped the Synology and went straight to FreeNAS.
[1] http://duplicity.nongnu.org/
[1] https://github.com/gilbertchen/duplicacy
https://duply.net/
To your bonus points: Google photos web UI is truly impressive even ignoring the awesome search capabilities. They wrote an article[0] on how they built it and since then I've been keeping an eye out to use similar techniques in different applications.
[0]: https://medium.com/google-design/google-photos-45b714dfbed1
I don't typically take photos of really private things, so the convenience of quick automatic backups from my phone and the availability of the search feature outweigh any privacy concerns I might otherwise have. I do also take local backups occasionally to my external hard drives in case Google somehow suffers major data loss.
1. Create a family gmail account (ex. house-yourlastname@gmail.com)
2. Install Google Photos on the both you and your spouse’s phone and set the automatic photo upload to upload to the house-yourlastname@gmail.com account.
3. Create a new account on your local home server (ex. house-yourlastname)
4. Install and login to the google drive client for the house@yourlastname.com account. DON’T do any of their photo upload nonsense. Just use the app to sync your google drive contents.
5. Go to https://photos.google.com and set the "Sync photos & videos from Google Drive” to “true”.
6. Go to https://drive.google.com and set the "Automatically put your Google Photos into a folder in My Drive” to true.
7. Install hazel on your home server and have it monitor the automatically created “Google Photos” folder that shows up in your Google drive (this folder shows up as a result of step 5). This folder is where all auto-uploaded photos from your phone(s) end up by default.
8. Have hazel move all auto-uploaded photos out of the “Google Photos” folder in Google Drive and into a folder structure of your choosing in Google Drive (Ex. “Google Drive/Photos/2018/11/2018-11-25-1234.jpg”). Because of step 4 the photos we move into our own folder structure will show up in https://photos.google.com.
9. Copy fancy camera photos into the same folder you are monitoring with hazel. The same hazel rules will rename and place photos into the same Google Drive directory structure.
10. Done - you have a pretty good shared family library. All family members can view all photos using the google photos app OR by visiting https://photos.google.com. And, any management (deletes, etc) are propagated to Google Drive. For example, I’ll triage on https://photos.google.com and all deleted photos end up n the trash can of my iMac by the time I happen to check.
Some general reasons why I landed on this:
1. I have a family and whatever I chose had to work for the whole family without jumping through manual copying at home. This basically nixes Apple photos, as they haven’t yet figured out we have families.
2. The only graceful way I’ve found to have a family library is to have spouses share an account. You could do that with an iCloud account, but there are lots of downsides to that. Google apps support multiple account gracefully, which allows you to be logged into a different account for Google Photos vs. Gmail (or whatever).
3. Most photo management solutions require you go sit down on the “shared family computer” to manage them. Google Photos bootstrapped off of using Google Drive lets you manage photos from anywhere (your phone, your spouse’s phone, the website, etc). The changes are synced back to Google Drive and back to your “source of truth” on your home server, Synology, etc.
4. The above solution only relies on the home server for sake of moving photos around to the folder structure of your liking. Even if your machine is turned off for a week, all your photos end up in the family library pretty quickly. Whenever you happen to start the server again, all the photos sitting in the “Google Photos” folder will then get organized into your preferred folder structure. This is appealing, as I didn’t like any solution I tried that relied on a server being up 24/7 to have all photos show up in the family library.
Switched to using google photos as primary backup mechanism. Working great so far. I also like their search feature, auto-create and reminder of previous year photos.
https://github.com/ptoomey3/family-photo-library/blob/master...
I have 2 TB+ mainly of kids/family photo/videos for ~15 years.
Right now, it is on a 4TB HDD with backup. Looks for good sharing solutions.
What about concerns of other people or do you mostly do inanimate objects or selfies?
I suspect there are many people like me who feel coerced into allowing our private life being stored in public clouds by other people without ever being explicitly asked for our consent. Worst, I feel in many situations it isn't easy to modify behavior of others because of the fear of appearing "needlessly fussy" and and because we value relationships with other people more than the abstract, intangible and hard to quantify concept of loss of privacy.
They've got that decent photo library app called Moments, which is available on mobile devices to upload your pictures from your phone directly. They also have another app called Drive, on mobile and desktop to upload photos and files. It's basically a closed source Google drive.
The NAS itself can very easily be plugged to Amazon S3 for encrypted incremental backup (I do a weekly backups).
The Hyper Backup is a wonderful tool (comes with Synology) that you can use to create multiple backup strategy, to another portable hdd, various cloud providers, any host with rsync etc, with or without backup version and rotation. The user experience is pretty neat.
Since my wife and I both have our own desktops it made sense to build the media pc to have a central place for photos and videos.
Unfortunately, it did not go to production: the majority of people are ok with sharing their digital artifacts with google, dropbox, advertising and insurance companies, and any other entities who manage to have an agreement with of hack into the cloud provider. (See, for example, how many people talk about google photos in this thread). The market for a truly secure digital archive is, in my opinion, too small.
A prototype can be still seen at [0].
0: https://goryachev.com/products/secure-archive/index.html
Sftp can't touch the snapshots which protects against malicious access (cryptolockers and such)
[1] - https://tinyvpn.org/sftp/#lftp
Also in term of "safety" (or encryption) many kind of performance depend on how you encrypt your data, and how you compress your transfer because they are pretty CPU-intesinve things...
CPU overhead isn't really a limiting factor these days, assuming your computer is less than a decade old.
With sftp chroot, I can let people upload files without executing files, while preventing eavesdropping. rsync does not support TLS without complex stunnel wrappers on each side.
> The lftp mirror subsystem allows you to break up batches of files, or even a single file using p-get into multiple streams.
I do not see any particular benefit from that... Network reliability is still a bit needed or you still have to restart...
On restricting rsync on the other end there are many guide so again I do not see much benefit and that's not true that you can't encrypt the transfer otherwise with classic ssh transport this is an example
https://sixohthree.com/1458/locking-down-rsync-using-ssh
but you can do many more things, for instance with cryptcat or ncat in the middle to transport... And they normally run faster than ssh tunnels...
What I suspect you will find is that where a single rsync stream might reach a couple hundred mb/s, lftp can saturate the link if you wish. At least, that is what most people find that try it out.
On setup cryptcat and ncat require almost ZERO setup, and you still need to have ssh on the over side unless you are restricted by some common crappy webui so...
Also, I see no reason to saturate the link apart of try to milk more performance at the expense of all others, servers and network hardware are not there to be abused but to be used... It's the same story with people that abuse http range to start multiple downloads of a single big file trying to milk a bit of performance...
rsync performance are not much rsync related, they depend on network, encryption and compression you use, disk I/O, ... per se it's pretty efficient. Certainly if you can incremental send a zfs snap via mbuffer is far more efficient but that's far more efficient than any kind of logic file transfer...
AWS recently announced Glacier deep-archive at $1/TB-month (standard Glacier being $4/TB-month).
I have also been eyeing a project called photoprism to serve as a front end for my media, and perhaps Filebrowser as well as a secondary interface to view them as "files".
https://github.com/photoprism/photoprism
https://filebrowser.github.io
Another project in the same "Google photos alternative" space that I have bookmarked is Ownphotos.
https://github.com/hooram/ownphotos
Some day, I will find time to set it all up ;-)
When I have some free time I'm going to set up the pi to run the sync via cron so that when I copy files via my MacBook over to the NAS, I can close it and let the pi do the syncing when I am out of the house.
0. I may empathize BUT if you value data on them... Well you ask something like how I can have my wife drunk and my barrel full (ancient Italian proverb). Sometimes you can "optimize" your storage for instance lowering a bit quality of images that you may do not need at super-high resolution, stripping metatada, change formats, ... Depending on the case you may save big size of storage.
1. Why not having a more flexible backup then? RClone is a nice tool if you need to use someone else computer and you may use various vendors, also rsync+something you can mount as a local filesystem may also enlarge market offers list...
2. No proprietary tool can be consider trustworthy, no service will take care of your stuff better then yourself...
To answer more directly: I do not feel the need of buy someone else resources for storage, however consider a thing: local backups may not be super reliable, may be stolen, lost etc. however if you properly store and use these kind of event are not much more likely to happen than a "disaster in the cloud" especially from cheap services that in turn themselves re-sell someone else resources. IOW my own personal suggestion is assembly/buy a decent PC, with enough PCIe ports to support many sata port, fill it with reasonably big plate disks at the best price/Gb ratio you are able to find, install a GNU/Linux distro you prefer or pay someone to do so for you, I suggest using NixOS because being declarative if something goes wrong recovery will be quicker. Create a proper raid structure and use that as a personal cloud.
File transfer from various device may vary depending on the source device, from Syncthing to rsync+webdav you may have many options. It's automation again highly depend on "sources"...
To serve your content GNU MediaGoblin may be interesting, simply locally mount your server storage via nfs and use any local tool you want may also be an option.
I have a terabyte of photos and videos. I use git annex (without the assistant) to store and manage them. I push a copy to my desktop computer at work and to the NAS at home.
I'm not much of a believer in using the cloud for archival and backups. Hard drive space is much cheaper than the cloud, even with lots of redundancy.
I just purchased a cheap external drive which I'm going to use for offline backups in the fire safe.
I also have a friend who is willing to set up a backup-swap with me, but I need to sneaker-net the first backup to him.
Git annex will push to the cloud as well, I don't use that because I don't want to pay for the storage space.
For the photos on my phone I'm using resilio sync plus a cron job to copy them off and then free up space on the phone.
For organizing them I use a react/rails web app that I've been developing on and off for a few years. It's similar to google photos (without the AI). It's at https://github.com/jewel/hypercheese.