Ask HN: What is your favorite method of sending large files?
I just opened up a simple HTTP server to send someone a large file. Then I figured, I never gave this question proper thought.
But some of you have, and I figured they make for fun and interesting stories ;-)
So what's your favorite method to send large files, of at least 5GB or bigger? Though, I'm also curious on how you'd send 10TB or more.
361 comments
[ 2.7 ms ] story [ 257 ms ] threadBetween OSX, AirDrop works very well. I have sent >10GB files between Macs, quite quick as well.
I have never send a 10TB file so I wouldn't know. None of my drives are that large yet :)
Edit: magic wormhole is awesome! Thanks! :D
Provided of course that you have a wifi card that supports active monitor mode.. and you are root on your machine.
https://github.com/seemoo-lab/opendrop
It _imitates_ the AirDrop UI, but is not compatible.
Use case is you open up snapdrop.net on both devices (nearby? same wifi?) and you can send files between, but I believe it all goes through a central server. Great for sending a few photos from phone to laptop or linux device when AirDrop isn't an option.
[0]: https://github.com/schollz/croc
Partially out of my annoyance with this I implemented a port of Magic Wormhole in Go that fully interoperates with the official python version: https://github.com/psanford/wormhole-william
I chose not to copy the features of magic wormhole because it seemed to me (at the time) to be very complicated and lacking in features crucial to myself (namely restarting transfers [1], sending folders without zipping them).
This has been to my benefit, I think, because I can iterate on croc quickly without having to conform to the wormhole spec and I've since added features still not available in wormhole (like restarting transfers [1], and ipv6 support [2]).
That being said, I really like magic wormhole and consider it my inspiration. And I'm glad that you ported it to Go!
[1]: https://github.com/warner/magic-wormhole/issues/88
[2]: https://github.com/warner/magic-wormhole/issues/354
In any case, I encourage everyone to setup their own relay. Its easy. Make sure ports 9009-9014 are open and then run `croc relay`.
Connecting to your relay is as easy as `croc --relay ADDRESS:9009 send X`. And if you add the `--remember` flag you only have to tell it which relay to use once.
Just curious: pun intended?
As I mentioned in another thread installing Magic Wormhole is _easy_, just "brew install magic-wormhole" or "sudo apt-get install magic-wormhole" (or distro equivalent). Windows might be tricky but don't see how croc does it much better. I hadn't heard of Scoop but seems like you'd need to install _that_ first (which has its own dependencies). That said I haven't used Windows in many many years, maybe Scoop is the standard nowadays like Homebrew is for Mac? In which case I'd give a point to croc.
Also, with a Go app you could just share/distribute the binary, but then you also need to know where to put it. And you could do that with Magic Wormhole as well.
Or a step better like using something like PyInstaller (wonder why author hasn't done that?)..
I agree installation for magic wormhole is easy for Mac/Linux, but I think its not easy on Windows. Windows is very common (77% of the market share for desktops [1]) so I wanted to keep croc easy to install for Windows.
In fact, if you are just receiving a file on Windows, all you have to do is download the binary from releases [2], unzip it, and double click on the binary - no terminal experience required. (Sending a file does require using a terminal, but all CLI apps have that hurdle...).
[1]: https://gs.statcounter.com/os-market-share/desktop/worldwide
[2]: https://github.com/schollz/croc/releases
I use MWH often for sending stuff like URL's, passwords, API keys... having to create a file for that is quite annoying (and remembering to delete it afterwards).
"brew install magic-wormhole" or "sudo apt-get install magic-wormhole" is easy enough and I don't hang around with people who use Windows. ;)
Resumable file transfers sound like a great feature though. Not sure it's implementable with MWH the way it now works.
[1]: https://github.com/schollz/croc/releases/tag/v8.2.0
Tab completion is something I'll miss from MWH (I'm already converted!) but I can live with it, and my non-technical friends aren't even aware of the concept.
Simple. Works.
But for something like 10 TB or more, I'd see a torrent as the only way. My uploads speeds are too slow for anything else, the connection would be reset. The torrent also helps prevent corruption.
https://cloud.google.com/transfer-appliance/
There's an amusing infographic towards the bottom. Only 42 days!
"Dude, I'm maximizing my data transfer speed"
https://github.com/FiloSottile/age
Encrypt:
Decrypt:Others have said it better than I can. See e.g. https://latacora.micro.blog/2019/07/16/the-pgp-problem.html
With that said, Magic Wormhole is also a very good tool for transferring files. It will encrypt in transit. So for many files, using a separate encryption tool is not necessary. (So far I haven’t tried it for large files.)
The author really wants to dislike PGP, but the reason everyone trusts PGP is because it's been around forever. Yeah there've been deficiencies, just like there've been deficiencies in OpenSSL, but that doesn't make it a bad tool. I could go on but this xkcd sums it up: https://xkcd.com/2347/
>Absurd Complexity / Swiss Army Knife Design
Git is complex, yet effectively every project ever uses it. The reason is you're fine to avoid the edge-cases and just focus on the main functionality, but that one time you need to do something ridiculously hacky, there's a tool to do it, instead of having to roll your own solution.
>Backwards Compatibility
Would you rather your software not have backwards compat? GPG has sane defaults, and everyone you talk to using modern versions will be secure by default. Not sure what the author is going on about with weak default password encryption:
>Obnoxious UXI don't really find GPG any harder to use than Git. http://irtfweb.ifa.hawaii.edu/~lockhart/gpg/
>Long-Term Secrets
By default, GPG keys expire in a year.
>Broken Authentication
I've never heard of any of this. Sign and encrypt, by default you get AES256 encryption and a SHA512 digest:
>Incoherent IdentityHave an identity. Have other people verify it. Trust based on that. It's the same way the PKI works, you know, that thing that runs the entire internet. Except you don't need to trust CAs anymore.
>Leaks Metadata
He's not wrong about this one, normally you can see who's ID a message is encrypted for. If you're trying to be sneaky just use symmetric encryption I guess, it feels like a different use case.
>No Forward Secrecy
Definitely a different use case. There's no case where I want to decrypt a packet from the middle of a TLS conversation a few years later. But an encrypted attachment in an old email?
>Clumsy Keys
How are GPG keys harder to handle than SSH keys? Both are just blocks of base64 (gpg --export-secret-keys -a)... one is 80 lines while the other is 50, but does it really matter?
>Negotiation
Same argument as Backwards Compatibility, I think.
>Janky Code
The page he linked has 27 CVEs. Over the last 15 years. For comparison, OpenSSL has over 200.
Just for posterity, the tool linked above that everyone is discussing here was written by the Go Lead Dev for cryptography and security at Google.
One obvious guess is contradicting Latacora. 'tptacek is well known here, his name alone give significant weight to anything he writes.
In any case, they have valid points. PGP was written at a time we didn't understand cryptography as well as we do now. We can do better. Have done better, if half of what I've heard about Age is true.
Absurd complexity: We can definitely do simpler than PGP, at no loss of functionality.
Swiss Army Knife design: I think I disagree with Latacora there. Doing many things doesn't mean you have to do them poorly. There's no material difference between having 3 programs, and having one programs with 3 options, at least on the command line. If PGP does anything poorly, it's for other reasons.
Mired In Backwards Compatibility: well that depends. It makes sense that PGP can decrypt old obsolete ciphers & formats. Ability to encrypt to those same old stuff wouldn't. For instance, PGP longer be able to generate RSA keys at all. Then, one year later, once all RSA keys have expired, new PGP versions should no longer be able to encrypt to RSA keys at all. (In an ideal world. More realistically, we should wait a couple more years.) Only the ability to decrypt old messages should be kept until the end of times.
Obnoxious UX: I don't know enough to have an opinion.
Long term secrets: Sure they're bad, but I don't think we can avoid them. People need your public key to send you anything, so it can't be too short lived. My guess here is that Latacora is attacking the whole file encryption + web of trust thing, not PGP in particular.
Broken Authentication: If attackers can trick PGP decoders into decrypting forged messages, that's fairly critical, and should be fixed even if it breaks backwards compatibility (we could have an optional `-legacy` flag or something to compensate). Now if you go sign and encrypt… well there are two possibilities: if you sign then encrypt, you run into the cryptographic doom principle: the decoder will decrypt then verify, which creates the temptation to process unauthenticated data. Many vulnerabilities have been caused by such errors. If you encrypt then sign, you reveal to the entire world that you signed this particular ciphertext. Not the kind of data most people would like to leak. In my opinion what we really want is authenticated key exchange followed by AEAD. With the Noise X pattern, you'd even hide your identity from snoopers.
Incoherent Identity: Okay, they're clearly attacking the very notion of web of trust, not PGP specifically. They say it doesn't work, but I'd like to know why. First, I'm not sure I want to take their word for it, and second, the causes might be fixable.
Leaks Metadata: that one is clearly avoidable. Noise X for instance uses an ephemeral key to encrypt the transmitted public keys, and the recipient's key is implicit. Can't know who the message is for (nor from) without the recipient's private key.
No Forward Secrecy: Different use case indeed. Again, Latacora is attacking the very notion of file encryption, not PGP specifically.
Clumsy Keys: I'm with Latacora on this one. The 50 lines SSH keys are clearly RSA based, and as such obsolete. Modern keys use elliptic curves, and those take one line, which is more easily copy & pasted in various contexts. Arguably a detail, though.
Negotiation: It's more than just backwards compatibility. Backwards compatibility can be achieved with a simple version number. If instead we have a range of algorithms to chose from, things get more complicated. Now, you can't avoid the need for different kinds of encryption: you can send against a public key, or you can encrypt with a password. Possibly both. Beyond that however it's s...
This speaks to the article's complaint that GPG is usually the wrong tool for the job. For example if you just need to transfer a file securely (and have a fast, reliable internet connection on both ends and don't need to worry about active tracking of metadata), you can use Magic Wormhole (or a similar PAKE system) to do it. Imagine two scenarios: one with GPG and one with PAKE, and in both cases an adversary captures a ciphertext. With GPG if they can get your private key 6 months later, you're screwed. With PAKE the keys used to exchange the data are ephemeral, and so this isn't even a possibility.
> > Broken Authentication
> I've never heard of any of this.
I believe this is referring to authenticated encryption (AEAD), which is definitely valuable and GPG does not provide. AGE does.
Most of the other stuff you mention also falls under "wrong tool for the job". If you want a better argument, I'd talk about GPG having a web of trust system built in. On the other hand I think it's an open question whether this has ever brought real value to anyone. We have enough other secure messaging systems that it's no longer necessary for a single program to get there on its own. Usually you can count on some other mechanism to confirm your contact's identity.
AFAIK this isn't really true. If the adversary captures the initial key exchange plus all the data (ie the full transaction), then later discovers your PSK, they'll be able to decrypt. The only case where this helps you is if they capture some packets out of the middle without the initial handshake.
>authenticated encryption
It doesn't matter if your ciphertext is authenticated if it's both signed and encrypted, which is what I was getting at. In a normal TLS-like encrypted conversation, yes AEAD is very useful. But it's not applicable here.
Someone can correct me if I'm wrong, but I believe the idea behind a PAKE is that the password only authenticates the key exchange and doesn't contribute to it. So if you record all transmitted data you still need to break the key exchange which should have used a bunch of random bytes from both parties that are thrown away after use. The password is only there to prevent MITM, not to derive keys.
I believe magic wormhole uses SPAKE2, which has perfect forward secrecy. When using passwords to secure transmitted files it's really important to have forward secrecy otherwise you risk the transmission being recorded and the password being attacked offline which depending on your password strength might lead to trivially decrypting the data.
That's right. From memory, the passwords are just used to do a DH key exchange. The key is entirely ephemeral. Even if the entire ciphertext is captured, and even if the adversary then gets your password, they can't decrypt. To decrypt you'd have to MITM the key exchange, which would require knowing the password before the file is exchanged.
With trackers, you're revealing the fact-of-transmission, transmission-size, & endpoints to any number of unknown remote parties. Potentially, attackers not even on the privileged network-path from origin to destination could tee off a copy of your encrypted data for offline analysis.
With Magic Wormhole's rendezvous-server, only one server, run by the same person whose code you're trusting (& can audit), briefly relays encrypted control-messages. (It might even be limited in its ability to deduce the size of the transfer – I'm not sure.) And if that's still too much, you can run your own rendezvous server.
It seems to me the amount of information leaked in the BT Tracker approach is strictly (& perhaps massively) more, to more entities, than that leaked in using the Wormhole author's server.
Of course the nice thing about Magic Wormhole is that its security does not depend on server components to be trustworthy. Use the default servers, use your own, use a different third party server, it doesn't matter, your data is still secure.
Edit: If you are worried about privacy magic-wormhole supports transit over tor.
There are many tools which are way older than gpg, but still in the wide use. For example, I have not heard any arguments that people should stop using things like "rsync" or "curl".
But if you like GPG then feel free to continue using it.
Using Magic Wormhole would seem to be far more secure than gpg and bittorrent since it has less risk of third party interception and uses SPAKE2 which has perfect forward secrecy.
If I had to use bittorrent to transfer a sensitive file I would generating a random key and encrypt, then share the key over a different channel.
In your browser, start downloading a test file, like [1]. Turn off your network card for a few seconds, then turn it back on. See what your browser does.
Next, start downloading a large file over Bittorrent, like [2]. Turn off your network card for a few seconds, then turn it back on. See what the client does.
[1] http://speedtest.tele2.net/100GB.zip
[2] https://cdimage.debian.org/debian-cd/current/amd64/bt-dvd/de...
a raven with a sufficient large micro-sd card ;)
What about using "Firefox Send"? (I never used it so far)
https://support.mozilla.org/en-US/kb/send-files-anyone-secur...
I read that the limit is 1-2.5GBs => maybe you could break down the file and upload it in multiple pieces... .
EDIT: oopps, Firefox Send doesn seem to be available anymore - https://support.mozilla.org/en-US/kb/what-happened-firefox-s...
For my 12TiB of data I use Syncthing when I need to sync them more often, rsync.
I used rsync several times for billions of smaller files totalling to 300GiB, but really all depending on how I connect nodes. I prefer syncthing, but when only ssh is available, then rsync is good too.
Currently largest synced directory by syncthing (that shares usage stats) is over 61384 GiB :)) https://data.syncthing.net/
https://docs.syncthing.net/users/faq.html#why-is-there-no-io...
If it's too big for cloud storage, ship a hard drive in the mail.
However no-one mentioned a super simple service: https://wetransfer.com/ - Simple as drag & drop, enter recipient address, SEND. Pretty simple if you want non techie to send you something.
A dead simple UI and a link is nicely emailed to the receiver.