Ask HN: Which file storage service would you recommend for the long term?
[EDITED]
I have about 1TB of files right now, and I expect that to grow by 0.5TB every year.
My goal is to
* use a service for the next 50 years
* lose none of my files
* be able to add to my files every day
* be able to access any of my files any time
Which service would you recommend?
The ones I know of: [Dropbox, Google Drive, Microsoft OneDrive, iCloud, S3]
I'm not really tied to a platform. I have a Mac, a PC, and a Google account.
Right now I use Dropbox, fwiw.
51 comments
[ 4.6 ms ] story [ 88.9 ms ] threadEDIT: It's Glacier I haven't heard of, not Amazon.
In any case, I think the criteria of "I haven't heard of it" is a little strict, given you've made an open question asking for advice on what to use...you should be open to the possibility that there are people here with expertise who know of things that you haven't heard of. For example, you don't seem to have heard of Amazon S3, which Dropbox is based off of. Amazon is also the company behind Glacier, and while Amazon is far from being a sure bet of lasting 50 years, it's a good as bet as any company.
I don't think you can guarantee 50 years, regardless of number of services used.
But you'll have plenty of time to offload them before any of these big names shut down.
Also, I'd guess this is your best bet for > 50 years.
http://docs.aws.amazon.com/amazonglacier/latest/dev/data-ret...
For the former, I'd recommend using Amazon's Glacier or Google's Nearline storage.
If you intend to retrieve your files often, Nearline will do better as the costs are lower and retrieval is faster. You'll likely want to wait until it's out of beta first though, which might take a while.
If you don't intend to retrieve them often, Amazon's Glacier might be a good choice.
There are also services like CrashPlan and Backblaze that are definitely worth looking into. They're definitely easier to use and CrashPlan at least gives you the option to encrypt.
For the latter use case, I'd recommend Google Drive. You can get a ton of storage through Apps for Work Unlimited and there's very little chance of it disappearing in the near future with Google's apparent goal of moving all your data into their cloud.
EDIT: same with iCloud (or MobileMe or whatever it was before that)
Synology gives me my own private cloud (all major OS and mobile clients) and loads of other benefits as well.
I'm very happy with it.
I haven't used it for accessing individual files from backup, just whole drive backup.
You can set it up to sync to Glacier/S3/Dropbox/GDrive/OneDrive/Local NAS/FTP server or any combination of them if you want to have multiple copies.
I get 1TB of OneDrive free with Office365 so backup to that but also to S3 so that if I was to stop using Microsoft I'm not tied down.
Has unlimited upload rate; never deletes files; good restore client; can be bought with unlimited upgrades for $60 and at 1TB (without taking into account accessing the data) on S3 you'd pay $30/month | $24/month reduced redundancy or $10/month on glacier for less frequently accessed data.
Because it's a bit more expensive I only tend to put absolute essentials (like photos) on S3. Arq does also compress to help save some cost here.
Worth mentioning that Arq also encrypts everything client side before transmitting so everything is stored encrypted as well. Might be important for you.
You lose Arq, you lose your backups.
https://www.arqbackup.com/s3_data_format.txt
And there are tools to help:
http://sreitshamer.github.io/arq_restore/
I would be happier with a more bulletproof app that only backed up to one provider.
Disclaimer - I work for Riverbed, who used to have a lovely product called WhiteWater, subsequently renamed SteelStore and sold to NetApp. This is a backup / archive system that uses whatever 'cloud' (not a word I'm fond of) backend you prefer. Data is de-duplicated at ingress, and encrypted at rest. It would require a local VM, but that's a small matter these days. No idea on pricing.
If you're at all inclined, and primarily to address the '50+ years' requirement, you could look at rolling your own solution. Architecture would be (initially) something like a RPi with a 3TB drive - bought and installed at (at least) two friend's places. Perhaps a contra-deal arrangement with them. VPN for security & convenience. LUKS for at-rest encryption. Something sitting on top of rsync (over SSH) for in-transit encryption + file-level deduplication -- I use dirvish[1] for archives that are too large / too binary blobby for git. I've got two family members already doing rsync (using unison[2]) backups over the net to VMs I host that then get dirvished.
[1] http://www.dirvish.org [2] http://www.cis.upenn.edu/~bcpierce/unison/
Duplicates. Perhaps two or three online services, plus local disk array at home. Check data integrity every year...
Every decade you will have to swap service or two, as companies evolve.
https://leastauthority.com
Your 50 year requirement is the real tough problem here. I would have a hard time recommending any of the types of services your are referring to for more than a year at a time. To achieve anything beyond that you need to look for a service that is VERY standards based and portable.
You do not mention a need for "backup/restore" or "version history. So lets treat sync & cloud access separately from backup & versioning.
File Sync and Cloud Access
I think any thing that syncs local files to the cloud will do as long as you don't rely on special features of a specific service. This sort of means that iCloud and S3 are out of the running since neither of them is a cross-platform file directory syncing system.
You could keep copies on one or more systems that you control and using the service to provide sync and cloud access. Then if you decided that the service that you are currently use no longer meets your needs, or is out of business, you could easily switch to a different syncing/cloud service.
Today I would say that Dropbox is the best solution. Personally, I am beginning to replace Dropbox with BitTorrent Sync and keeping a close eye on syncthing.net. Although, BT Sync doesn't have a web interface, I consider that a plus for privacy, and BT Sync's mobile apps and built in sharing features replace most of what I need from Dropbox's web interface.
Backup/Versioning
This is where it gets really tricky looking out 50 years. I simply can not recommend thinking out that long term. I think you can look out 10 years at a time, and then make choices for the next 10 years.
Something like rdiff-backup with would last at least 10 years, but it doesn't have the easy to access interface. (last I check the radio-backup web interfaces where not very good).
Personally, I use a combination of diff-backup and Crashplan pointed at local and remote storage. CrashPlan gives me remote access via the web and apps, and a professionally managed data store. (I feel that while I don't trust others with protecting my data, I don't trust myself 100% either). I also use Arq (with now has a Windows client, and provide an open source code for doing a restore) and backup to a remote site.
https://www.crashplan.com
https://www.arqbackup.com
http://www.nongnu.org/rdiff-backup
https://syncthing.net
https://www.getsync.com
Everything points to my Linux NAS and than I backup to CrashPlan.
This is the key point to focus on: you don't actually have a backup plan if you don't regularly access the stored data – until then, it's just hope.
I like the CrashPlan + Dropbox approach but there's a key gap: you need something else to check strong signatures (SHA-256/512) on data to avoid scenarios where a bit flips on one of your computers, the file is updated and you don't notice it for a long time, which is really easy for data which is infrequently accessed or where the client programs typically cover minor flaws (most image/video/audio formats).
My favorite example is a JPEG-2000 file which had corruption in the middle of its tile hierarchy – either a thumbnail or full 1:1 zoom looked fine but something like a 50% zoom would show corruption. You'll probably never find something like that without automated checksums until it's too late to recover a copy from somewhere else.
What I'd really like is something like a mature large-file optimized Git repo where there's a strong, checksumed history and it's immediately obvious if a remote version is out of sync. https://github.com/bup/bup and https://git-annex.branchable.com/ are very interesting.
Archive Team is trying git-annex as a way to back up the Internet Archive globally, which will likely lead to changes helping that tool become solid for general usage: http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK
I may suggest deploying your own service using (owncloud)[https://owncloud.org/] on your own dedicated servers
Edit: in their favor, at least they do provide GPG signatures for their downloads.
Probably won't happen. Businesses come and go. Unless you want to be constantly shuffling file storage backends, such a plan will probably end in disaster.
If you want something with the requirements you describe, you'd be better off self-hosting. I'd set up two servers - one in your house or office or what have you, and a second in some further-away location (perhaps a relative's house). The "servers" could be as simple as ordinary desktops with a bunch of Western Digital Red drives. Pop your operating system of choice on them, set them up with some way to transfer files (I personally just SFTP important files to an OpenBSD box; once I find a suitable secondary location, I'll probably set up another one and have the first one pump the contents of its backup folder to the second one on a daily basis with something like rsync), and be done with it.
The downside is that now you have to manage the hardware side and the software side of things, but at least now you have some assurance that - so long as you actively maintain your backup server(s) - your data will be safely backed up.
For my servers, I've been wanting _something_ that I can use Amazon S3 with. Just did a search and found out about CloudBerry (http://www.cloudberrylab.com/). This is perfect since I also have a NAS and scripts that backup the server nightly. With this I could essentially just install and repoint the existing script to the new drive. Also, S3 has built in versioning which is really nice.
This really bummed me out because I was evaluating it before sending Google my money, but I have concerns about the quality of their client. Apparently this has been a problem for years.
Also google drive has become annoying (e.g. I upload a ripped DVD movie which I own and paid for and share it with my son who lives in the same house, google tells me that I'm sharing copyright content (wtf)) So they definitely look at content and behave like big brother.