HN Explorer

Advice needed for backing up and hosting large amount of files

171 points by KingZil ↗ HN
Some of you may know Hong Kong has been attacked by China and most of the pro democratic media companies have shut down.

A number of us have been working hard to backup whatever we can and some oversea universities thankfully would allow hosting on their private library domain.

However I also would like to host these files on a public domain, just to ensure more people can revisit on these events in the future.

I am considering to start a very basic category based (eg event based) website, with text and picture files, while video files (close to 6tb) will be uploaded to censorship resistant sites. And a torrent file that contains all files in that particular page.

It will be ran by my own fund/donation with monero.

What kind of option do I have? I am guessing it wouldn't be safe for me to self hosting, given the nature of these files.

If someone can give me a pointer, that will be great.

Thanks!

96 comments

[ 0.27 ms ] story [ 171 ms ] thread
jmathai
Have you looked at the Internet Archive or Archive Team?

The Internet Archive provides free storage (last I knew from ~2014). https://archive.org/services/docs/api/ias3.html

KingZil
thanks, this is actually what we used for the initial backup, but what we foresee in the future is how to "get" the information out, because in a few decades, these media name may get forgotten all together. We also need to host the video files, which can be tricky on there.
noduerme
if you want to stream the video... you need hosting in a neutral country like Switzerland and you will pay a lot for it, especially if it gets any attention at all and the unfriendly government starts to dDos it. You probably want to have it as a set of torrents, mirrors, and pop it up in different locations. Don't rely on a single locale. Again, 6Tb is not a lot. Make a site with nothing but links, hold the data in mirrors and torrents. Also, only pay for hosting in crypto and use fake EVERYTHING. fake name fake IP, fake browser when you set it up. Let paranoia guide you. If you think you're being too paranoid, you're not being paranoid enough.
[deleted]
(comment deleted)
herbst
> you need hosting in a neutral country like Switzerland and you will pay a lot for it

If you are actually interested in Swiss hosting for whatever reason, there are plenty of VPS & Dedicated offers these days, like Exoscale, that offer really fair pricing comparable to Vultr or DO

ChuckNorris89
>you need hosting in a neutral country like Switzerland

Switzerland is only neutral on paper and has a history of letting itself and it's tech companies influenced by powerful foreign nations (the German-US - Crypto AG scandal , ProtonMail sharing info on some of its French users to the French authorities, etc.)

You might get good protection if you're a Swiss citizen living in Switzerland, but if you think Switzerland is some bastion of digital safety for foreigners, think again. If you're a foreigner, the Swiss authorities will not hesitate to throw you under the bus if you're being targeted by another powerful nation state with influence in Switzerland.

noduerme
I don't think Swiss authorities would take down a site with video evidence of human rights abuses at the behest of the Chinese government. But hey, maybe times have changed.
m348e912
Encrypt the files and then use backblaze, reasonably priced cloud hosting. With encryption the nature of the files are irrelevant.
jazzyjackson
I'm not sure what to make of the monero mention, is it a requirement that you can pay anonymously? or do you have a fund that is able wire money? If you're avoiding the eye of China I suppose it's the former.

I believe there's services to "pin" content to the IPFS network, but I'd be surprised if there aren't torrent seedboxes you can pay for with crypto, but I'm afraid I'm short on specifics. Would the idea be, to keep the files available, there would be a wallet that anyone can pay into to pay for hosting ? It's what I assumed we would get with all these blockchain projects but I haven't really seen it yet, there's filecoin and arweave but I don't know if that they are trustworthy for the long haul what with the market crashing

EDIT: After reading some blogs on FileCoin it seems like they fit the bill, and I think my notion of "anyone can fund the maintainance of this specific dataset" is known as a DataDAO and maybe doesn't exist yet?

KingZil
Monero would allow donations to be anonymous, so even if all things go south, it should be only my payment to VPS/hosting that would be leaked.

I want to enforce this by only accepting monero, having privacy by default, at the cost of not getting a lot of donations.

pronoiac
Torrent seedboxes might also be useful, as they prioritize bandwidth and (maybe?) disk space. I'm not sure if they'd be more or less secure than the typical VPS hosting, though, and I'd default to pessimism.
BxGyw2
I'm interested in your project, have experience in this field and can give you details of privacy friendly hosting and perhaps guide you through setting this up anonymously+safely. Get a throwaway email/xmpp address and post it on site please
KingZil
Thank you! We are still in the "back up" stage, because some platforms are at risk of closing down too, now that the new chief executive is in power. I will post a throwaway mail here if we confirm to go ahead with this plan.
PaywallBuster
You can host it anywhere and just "proxy" it from a throwaway box

So in that case you can use any commercial storage option suitable for that like Backblaze/S3/etc and simply use a 5$ VPS (which you can rotate often) in front

uniqueuid
I work in communication science and I know some folks who would be interested to use this for research. If you have the bandwidth, ping me at p@atrifle.net and I can pass the data on. Regarding hosting, I probably can't offer that because public institutions are typically kind of averse to hosting data for everyone.
null0pointer
While I trust that the commenter above is genuine, OP seems to be highly concerned about their opsec. So I just want to post a friendly reminder for the OP to be careful about reaching out to people that comment on this post. For example, don’t reach out using an email that can be linked to your identity.
uniqueuid
Yes I thought about that. There are a couple of different threat models. Anyone living in HK right now is in great danger and should be extremely cautious. For them, even posting here may be too risky.

Anyone outside of China may still be at risk, especially if they have family / friends there.

But to add a bit of opsec, the email above is on Protonmail, and if OP wants to reach out they can create a burner account there and email me - that would stay within the service. (cue debate about TLA access to Protonmail, but at least it's probably not the CCP)

noduerme
consider this alternative.

Split it into parts. Create a torrent file for each part. Get a private (home) server in the US and seed it, e.g. I have 1Gb upstream and would happily donate that to it 12 hours a day. Then come back to HN and other places with a website pointing to the torrent files, asking for more people to seed it. Also, ask seeders to store mirrors and put them wherever they can. 6Tb is not that much.

sophacles
If you do a plan like this OP, also share the torrent links and explanation to r/datahoarder - those folks are always talking about interesting datasets that they think are useful to seed. Many of them explicitly do so to theoretically fight against the sort of censorship you are actually facing, and would jump at the opportunity to help.
Jiocus
About video,

MEGA.nz offers e.g. an 8TB storage tier for a reasonable price. I'm not bringing them up because of their security+privacy concept –which may suit you– but because of their take on how you can put the material to work (and do work) while hosted with them.

HTML embed videos from storage into your web content and they'll be decrypted on the fly as their watched. Link sharing for selected videos for discretionary views or downloads by receiving users, and of course team member co-op and granular access to said materials.

For starters though, prioritize getting material sent and backed up in relative safety before thinking about functionality. A simple store might fulfill your requirements in the end.

Feel free to drop me an email.

chmike
I get daily spam from mega.nz. They can go to hell
Jiocus
This was the first time I heared about your issue, so I checked what they had to say on the matter.

Complaints about email spam with a MEGA email address: https://help.mega.io/security/data-protection/complaints-abo...

TLDR hypothesis - You get daily spam from a third party, using spoofed email headers, but I'm sure you'd figured that out already

chmike
Sorry, that is correct. I can't delete my comment. The mega.nz appears only in the from field. I see they use SPF.
Akronymus
You can possibly try to contact dang via hn@ycombinator.com to request that comment to be deleted. As a general rule, replied to comments can't be deleted by a user.
bauruine
According to Kim Dotcom [0] The Chinese government has a backdoor to MEGA.nz.

[0]: https://nitter.net/KimDotcom/status/1539426611870986240

Akronymus
Huh. I just learned that nitter, but not twitter is blocked by the company firewall.

But yeah, I don't trust mega either.

Jiocus
If he could produce substantial evidence to back up that claim, I'd be very interested in learning more. Until then, he doesn't have the credibility to be trusted.

Based on the assumptions I have about MEGA, Chinese intelligence and their motivations I find it unlikely they have an active, mutual conspiracy (impractical, expensive).

I assume MEGA is as viable target for covert, low-cost try-your-luck attacks as much as any other western infrastructure/enterprise. In that sense and in this post-Snowden world we live in, it seems as likely that any capable nation or five-eyes member et.c. could have* such a "backdoor".

*edit: 'has' → 'could have'

stonenoodles
I'm planning to look into https://www.storj.io/pricing (150GB free if I understand this correctly) for my own backup needs. You could theoretically host your own node somewhere on a cheap VPS, get credits and pay for your own storage? Not sure - I myself have to do some research.

Then you'd need a fronend capable mounting Storj and they seem to have pretty good documentation.

Again, unaffiliated, and not tested (yet)

Another alternative - pay google (https://one.google.com/about/plans) via a VPN in Turkey and you have 10TB for about 20USD a year. Make the files shared and there you go.

Oh and: I'm very sorry about what's happening in HK the last few years. I still hope the process is reversible.

tehlike
How big of an archive are you talking about? Torrent + a few cloud providers in different regions, ipfs pinning can create sufficient redundancy?

Reddit also have data hoarders that might help

dewey
> (close to 6tb)
ilaksh
Internet archive and IPFS pinning on Filebase and web3.storage as well as your own IPFS cluster put together with volunteers. Also can look into Sia and Filecoin.

If you can get two other engineers to volunteer and about 5 or 6 volunteers for buying storage, you can cover all of the above in less than a month.

Easiest is probably Internet Archive though. With a torrent.

Look into CAR files. Once you get those it's easy to pin on IPFS with multiple providers. Some of them free or extremely cheap.

fsflover
I wonder why nobody mentioned torrents over I2P: https://geti2p.net. It's distributed, anonymous and anyone can help without risking to reveal their IP. It's slow of course but might be faster if many people join.
ajvs
What torrent clients offer native I2P support?
fsflover
HN audience should be able to install the I2P client and enter the address in browser to access the torrents.
janandonly
The real issue is not making a torrent file, but finding enough people who want to seed it for years to come, whether over clearnet, TOR or I2P.
rsync
Please allow me to offer you a free rsync.net account, in perpetuity, for the backup portion of your requirements.

We don’t do hosting of any kind so you’ll have to secure that elsewhere.

Just email info@rsync.net to discuss.

smcleod
It's lovely to see things like this.
[deleted]
(comment deleted)
stavros
Would it be against your ToS if the OP hosted a small $5 Hetzner server that mounted your FS with sshfs and hosted files with a simple proxy?
gattopalla
Don't use hetzner for shit like this. They're EXTREMELY strict on abuse.
[deleted]
(comment deleted)
elcritch
How is this abuse? You're paying hetzner for a VPS. If running a proxy server is an abuse of their TOS, then in my mind their service doesn't seem much more than a scam. I'm glad to know to avoid their service.
ev1
If this is HK/CN related stuff, I'd anticipate Hetzner to be drowned in torrential rains of fake abuse emails.
schroeding
Anecdata: Hetzner does not care what files are on their servers as long as you don't also serve them publicly from their servers. So if they only use it as a network storage for another server, it should be fine.
rsync
No.

I will say that that sounds fragile and non performant but … at the same time, the whole point of our service is to be a “dumb” primitive - in the classic Unix sense.

So, of course this is within our terms and we see plenty of folks doing things like this.

xrisk
Whoa that’s a cool trick!

It’s important to note, however, that sshfs has been archived by its creator. What this entails with regard to the future of the project is uncertain though.

RealStickman_
I'm not sure what protocols rsync.net exposes, but it's probably supported by rclone if there are others than ssh.
pluijzer
I am looking to wean off Dropbox. Do you know if Syncthing in combination with rsync is a good combo?
rsync
I forget - isn’t syncthing straight up SFTP ?

Should work perfectly…

Belphemur
Usually every node that share a folder need to have syncthing installed.

The only way to make it work with rsync.net would be to mount a sshfs on a machine running syncthing.

Not sure if the performance will be good.

Maybe with Rclone ?

cbsks
I just want to say, mysterious ’rsync’ person, your evangelism of your company is top notch. From this and other comments you have made on this site, rsync.net is a service that I will sign up for without hesitation the instant I have offsite backup needs.
KingOfCoders
Had been a happy rsync customers with several companies, this makes me happy.
Handytinge
Hi!

Is there a way to get a test account/download file to test bandwidth/latency? On top of this, are there plans for nodes closer to or preferably in Australia?

By the way, your personal websites certificate has expired today :)

zxcvbn4038
Keep a copy in AWS S3, they are extremely reliable for storage, you’ll sleep well knowing they are there. But that S3 bucket should be private and locked down - it’s your master copy - and AWS is the most expensive option for outgoing transit so you want to use it as little as possible. For routine usage keep a second copy in Backblaze B2 which is the cheapest storage you can get that isn’t running out of someone’s basement. I’d use Digital Ocean to serve files from that - doesn’t have to be anything major, just something running nginx. I’d front that with Cloudflare which is essentially free for content that doesn’t change frequently. Your DO instance should only respond to requests from Cloudflare’s backend IPs and only if those requests contain some magic header you inject. That makes it near impossible to find your DO host and access it directly, and it is the only thing that knows your backblaze B2 secrets so nobody is accessing B2 directly either.

If you want to add an obfuscation layer in front of Cloudflare - though maybe not needed in your case since the content itself isn’t illegal in most of the world - but if you want to serve through Tor to protect Cloudflare you can - DO is a good option again - and if you go that route your Cloudflare site should have some random unrelated name and should only serve files if the requests have a magic header. Tor is where your going to reverse all the money you save with the Cloudflare + DO + B2 setup, your going to pay mostly for network usage, Tor has a lot of overhead. You can scale Tor horizontally across multiple hidden services if you can afford it.

laurent123456
If using S3 don't forget to setup a lifecycle policy to transition to a cheaper storage like S3 Intelligent Tiering, and possibly Glacier. Maybe just avoid One-Zone IA for better durability.
ndtfaker
Please upload to https://ipfs-gateway.cloud/

1. This hosting is based on IPFS, which mean the data is immutable and no censorship can be apply, you can retrieve the data from another gateway in this list https://ipfs.github.io/public-gateway-checker/. You can find out more detail about IPFS here https://ipfs.io/

2. The owner of this hosting has donated his storage to the community and you can upload for free, this was original posted on Reddit at: https://www.reddit.com/r/ipfs/comments/v0bnd1/ipfsgatewayclo.... He has approximately 13.7PB of storage =)))

After upload, you might want to save all the CID of your data and share them with your people (just like a URL)

stavros
IPFS is a good solution in theory, but in my experience it works so poorly in practice that I wouldn't trust it as my only form of archival. Certainly add files to IPFS too if you want, but not exclusively.
akavel
Is Dat/hypercore or other similar alternative more reliable in practice? Are torrents better? (But wouldn't tracker hosting be a "chicken vs. egg" issue?)
stavros
Hmm, I remember Dat being a bit more reliable when I tried it, but I have used IPFS way more. Torrents are good too, but you can't update them, which might be an issue.
kkielhofner
I second this and for anything other than "need to fetch a file or two occasionally" it's essentially broken in my mind. Expect to wait anywhere from 3 - 30 seconds for first byte. When modern CDNs are doing first byte in 100ms (or whatever) over HTTPS that's an eternity for most users.

Virtually all IPFS storage and bandwidth from pinning and gateway providers is at least 2-3x what you'd expect to pay for S3, etc (because they just use S3 on the backend). The public IPFS gateways have such low request limits and especially poor performance they often struggle to load a website with more than a couple of IPFS hosted assets on it.

If you want to run your own node go-ipfs is extremely difficult if not impossible to use at anything more than toy scale. Eats RAM like crazy, uses a TON of bandwidth (not unexpected but still seems like a lot), garbage collection is broken, and much more...

Frankly the state of IPFS is embarrassing considering it's seven year old tech.

dannyobrien
For the delay do you mean via the web gateways, or directly over IPFS?

For storage, most of the Filecoin-backed serves (like http://web3.storage, https://nft.storage, and https://estuary.tech are either free or close to that for most use cases. Have you looked at those recently?

I hear you on go-ipfs, but there are now multiple implementations, include rust-ipfs, that are also getting there.

sneak
ipfs is transport, not storage.
koonsolo
> It will be ran by my own fund/donation with monero.

Well well, seems like cryptocurrencies have a valid use-case after all. ;)

mootzville
We've been using fpsync (http://www.fpart.org/fpsync/) to backup our production NAS. It is basically a scheduler wrapped around rsync. We use some 10g NICs for internal backups...it's fast enough that the SSD's are the bottleneck, so we are probably going to switch over to m.2's at somepoint.

Having said that, external transfers across the net will end up being your bottleneck, so you'll have to decide how much compression is worthwhile to reduce the time on the wire.

Another option, if available, is dump it all to drives, and snail mail it, but I'm guessing you already thought of that.

Good luck.

gerardnico
If you want to store a lot of data, the solution is known as an object storage file system.

Check this list: https://datacadamia.com/file/edge_storage

There is a lot of other pricing implied if you want or not retrieve your data but this is mostly one write, several read solution.

traceroute66
Everyone here has been jumping in and saying "use my great service" or "use this great service".

But could I be the voice of reason here and jump in and remind the OP and everyone else to avoid putting all their eggs in one basket.

Even if you can't afford to replicate 6TB on multiple services, you should at the very least seriously consider picking, say three hosting services (preferably in different jurisdictions) and putting 2TB in each one.

In terms of offline/non-live backup itself, 6TB is really not that much these days. You could just buy a bunch of high-capacity hard drives, create multiple copies and spread them out via old-school "sneakernet" to your collaborators in different jurisdictions. Sometimes KISS is the best solution.

sacrosanct
pCloud.com has some plans. You can even buy a 10TB lifetime plan. Not affiliated, just a happy customer. Just make sure if you have intellectual property or sensitive things to encrypt before you upload.