Ask HN: What is your favorite method of sending large files?

319 points by mettamage ↗ HN
I just opened up a simple HTTP server to send someone a large file. Then I figured, I never gave this question proper thought.

But some of you have, and I figured they make for fun and interesting stories ;-)

So what's your favorite method to send large files, of at least 5GB or bigger? Though, I'm also curious on how you'd send 10TB or more.

361 comments

[ 2.7 ms ] story [ 257 ms ] thread
Magic Wormhole (https://magic-wormhole.readthedocs.io/en/latest/) if it's between OSes.

Between OSX, AirDrop works very well. I have sent >10GB files between Macs, quite quick as well.

I have never send a 10TB file so I wouldn't know. None of my drives are that large yet :)

I use the magic of AirDrop as well :) Unfortunately, sometimes I'm on my Linux machine (in which case python3 -m http.server works just fine), or need to send a file to someone else and then I'd like to have at least some form of monitoring/progress.

Edit: magic wormhole is awesome! Thanks! :D

OpenDrop works very well on linux for airdrop between linux+macos.

Provided of course that you have a wifi card that supports active monitor mode.. and you are root on your machine.

https://github.com/seemoo-lab/opendrop

There's a moderately useful one I've used quite a few times: https://snapdrop.net/

It _imitates_ the AirDrop UI, but is not compatible.

Use case is you open up snapdrop.net on both devices (nearby? same wifi?) and you can send files between, but I believe it all goes through a central server. Great for sending a few photos from phone to laptop or linux device when AirDrop isn't an option.

I wrote github.com/spieglt/flyingcarpet because I needed AirDrop on my Linux and Windows machines, pretty much does the same thing as running a Python server but over ad hoc wifi.
Magic Wormhole is good. These days I use croc,[0] which I find even better.

[0]: https://github.com/schollz/croc

Oh this looks great! Thanks for this, checking it out.
I never understood why croc chose to copy the features of Magic Wormhole but implement them in an incompatible way.

Partially out of my annoyance with this I implemented a port of Magic Wormhole in Go that fully interoperates with the official python version: https://github.com/psanford/wormhole-william

I initiated the croc project. Others have asked me the same thing, so I'll bite:

I chose not to copy the features of magic wormhole because it seemed to me (at the time) to be very complicated and lacking in features crucial to myself (namely restarting transfers [1], sending folders without zipping them).

This has been to my benefit, I think, because I can iterate on croc quickly without having to conform to the wormhole spec and I've since added features still not available in wormhole (like restarting transfers [1], and ipv6 support [2]).

That being said, I really like magic wormhole and consider it my inspiration. And I'm glad that you ported it to Go!

[1]: https://github.com/warner/magic-wormhole/issues/88

[2]: https://github.com/warner/magic-wormhole/issues/354

Would it be possible to sniff the protocol on the wire and provide some sort of cross-compatible magic wormcroc service?
Definitely possible. I'll happily take any PR that does this.
Can you write what is that hadcoded relay in croc? Who maintains it, how does it handle the increase in demand or outages etc?
I maintain it. Its been up for about 2 years with pretty minimal downtime.

In any case, I encourage everyone to setup their own relay. Its easy. Make sure ports 9009-9014 are open and then run `croc relay`.

Connecting to your relay is as easy as `croc --relay ADDRESS:9009 send X`. And if you add the `--remember` flag you only have to tell it which relay to use once.

> so I'll bite

Just curious: pun intended?

I didn't see this one until now :)
Thank you. Croc is fantastic because it solves a problem that no other tool does. Magic Wormhole stumbles at the first hurdle (installability).
I'm sorry but I don't understand the hurdle?

As I mentioned in another thread installing Magic Wormhole is _easy_, just "brew install magic-wormhole" or "sudo apt-get install magic-wormhole" (or distro equivalent). Windows might be tricky but don't see how croc does it much better. I hadn't heard of Scoop but seems like you'd need to install _that_ first (which has its own dependencies). That said I haven't used Windows in many many years, maybe Scoop is the standard nowadays like Homebrew is for Mac? In which case I'd give a point to croc.

Also, with a Go app you could just share/distribute the binary, but then you also need to know where to put it. And you could do that with Magic Wormhole as well.

Or a step better like using something like PyInstaller (wonder why author hasn't done that?)..

As BusTrainBus says, "installability" is useful, and it was one of my motivations.

I agree installation for magic wormhole is easy for Mac/Linux, but I think its not easy on Windows. Windows is very common (77% of the market share for desktops [1]) so I wanted to keep croc easy to install for Windows.

In fact, if you are just receiving a file on Windows, all you have to do is download the binary from releases [2], unzip it, and double click on the binary - no terminal experience required. (Sending a file does require using a terminal, but all CLI apps have that hurdle...).

[1]: https://gs.statcounter.com/os-market-share/desktop/worldwide

[2]: https://github.com/schollz/croc/releases

I don't see an option to send text (browsed through README and Issues), is this not possible with croc? With Magic Wormhole it's the default to get a message prompt when running "wormhole send". Or you can do it as a oneliner with "wormhole send --text 'something'". Wormhole William can do this as well IIRC.

I use MWH often for sending stuff like URL's, passwords, API keys... having to create a file for that is quite annoying (and remembering to delete it afterwards).

"brew install magic-wormhole" or "sudo apt-get install magic-wormhole" is easy enough and I don't hang around with people who use Windows. ;)

Resumable file transfers sound like a great feature though. Not sure it's implementable with MWH the way it now works.

Not possible right now. This is a good idea, I can add it in :)
Just added it now. Send with croc send --text "hello, world" [1]

[1]: https://github.com/schollz/croc/releases/tag/v8.2.0

Haha, quick turnaround.
Haha wow, thanks! I'll give it a spin today!
Nice! I just tried it out now. I also like the fact that "croc" defaults to receive, where with MWH you need to type "wormhole receive". Saves a lot of keystrokes.

Tab completion is something I'll miss from MWH (I'm already converted!) but I can live with it, and my non-technical friends aren't even aware of the concept.

I once sent a huge file over airdrop and was shocked how much faster it was then sending through the local LAN.
some copy methods verify and check during the transfer operation, others do not, so the final speeds are very different between them
The major difference is that Airdrop establishes a direct WiFi connection, bypassing the router completely.
Just had a 41GB set of files sent to me in one transfer using magic-wormhole. Worked first try without a hitch. Pretty awesome from the receiving end.
This is local lan only right? Or are you port forwarding to expose it?
Resilio Sync (formerly bittorent sync) is my go-to for any file larger than a few hundred megabytes.
Same here. I use this to deploy large files to my VPS
I third this suggestion, if you have to send the file to multiple parties they can all share among themselves (it's BitTorrent after all) speeding up the process even further.
I send public links from the Seafile instance running on my home server. For 10+ TB, I think the only practical option would be to mail a hard drive or two.
I like gofile.io, as it's private and has no limits.

But for something like 10 TB or more, I'd see a torrent as the only way. My uploads speeds are too slow for anything else, the connection would be reset. The torrent also helps prevent corruption.

A station wagon full of tapes hurtling down the highway...
No one makes station wagons anymore, except luxury european makers :(
There's plenty of used ones available. Nice and cheap.
Aside from the already-mentioned Subaru Outback, there's the Buick Regal TourX, the Mini Cooper Clubman, and I'll contend that the Dodge Journey is more stationwagon than SUV; it's on the Avenger sedan platform.
Some exchanges still sell historical tick data by posting over large amounts of HDD/SSD with the data preloaded.
"Why do you need to buy an Audi RS6?"

"Dude, I'm maximizing my data transfer speed"

Related: If you need to transfer sensitive data over Bittorent, Age is a good tool for encrypting it before transmission.

https://github.com/FiloSottile/age

Why a new tool? How is this better than gpg symmetric encryption, considering gpg is installed/available effectively everywhere?

Encrypt:

    gpg --symmetric file.dat
    (enter a password)
Decrypt:

    gpg --decrypt file.dat.gpg > file.dat
    (enter the password)
IMO, GPG is simply not a good tool and it should be replaced.

Others have said it better than I can. See e.g. https://latacora.micro.blog/2019/07/16/the-pgp-problem.html

That's said often about any tool which has been around long enough. People without experience come around and think they can replace an old tool with a better one, but it's usually only ignorance of either the complexity of the task, knowledge of using the tool properly, or both.
GPG is just not a very good tool. I think 'tptacek explained it quite well in the article I linked.

With that said, Magic Wormhole is also a very good tool for transferring files. It will encrypt in transit. So for many files, using a separate encryption tool is not necessary. (So far I haven’t tried it for large files.)

I want to comment on that article you keep referring to, but I don't want to clutter up the top of the thread so I'll do it here.

The author really wants to dislike PGP, but the reason everyone trusts PGP is because it's been around forever. Yeah there've been deficiencies, just like there've been deficiencies in OpenSSL, but that doesn't make it a bad tool. I could go on but this xkcd sums it up: https://xkcd.com/2347/

>Absurd Complexity / Swiss Army Knife Design

Git is complex, yet effectively every project ever uses it. The reason is you're fine to avoid the edge-cases and just focus on the main functionality, but that one time you need to do something ridiculously hacky, there's a tool to do it, instead of having to roll your own solution.

>Backwards Compatibility

Would you rather your software not have backwards compat? GPG has sane defaults, and everyone you talk to using modern versions will be secure by default. Not sure what the author is going on about with weak default password encryption:

    $ gpg -vv --symmetric test.txt 
    ...
    gpg: using cipher AES256
    gpg: writing to 'test.txt.gpg'
>Obnoxious UX

I don't really find GPG any harder to use than Git. http://irtfweb.ifa.hawaii.edu/~lockhart/gpg/

>Long-Term Secrets

By default, GPG keys expire in a year.

>Broken Authentication

I've never heard of any of this. Sign and encrypt, by default you get AES256 encryption and a SHA512 digest:

    $ gpg --sign --encrypt test.txt
    ...
    $ gpg -vv -o /dev/null --decrypt test.txt.gpg
    ...
    gpg: encrypted with 3072-bit RSA key, ID 74588E74DDD483BC, created 2020-09-02
      "test"
    gpg: AES256 encrypted data
    gpg: binary signature, digest algorithm SHA512, key algorithm rsa3072
>Incoherent Identity

Have an identity. Have other people verify it. Trust based on that. It's the same way the PKI works, you know, that thing that runs the entire internet. Except you don't need to trust CAs anymore.

>Leaks Metadata

He's not wrong about this one, normally you can see who's ID a message is encrypted for. If you're trying to be sneaky just use symmetric encryption I guess, it feels like a different use case.

>No Forward Secrecy

Definitely a different use case. There's no case where I want to decrypt a packet from the middle of a TLS conversation a few years later. But an encrypted attachment in an old email?

>Clumsy Keys

How are GPG keys harder to handle than SSH keys? Both are just blocks of base64 (gpg --export-secret-keys -a)... one is 80 lines while the other is 50, but does it really matter?

>Negotiation

Same argument as Backwards Compatibility, I think.

>Janky Code

The page he linked has 27 CVEs. Over the last 15 years. For comparison, OpenSSL has over 200.

I wish the downvoters would explain what's wrong with the parent comment, as the counter-arguments raised by parliament32 seem completely reasonable to me. Perhaps it's just the length of the comment, and they would prefer it written up as a blog post somewhere.
Don't worry, this is normal for HN. As a commenter higher up said, "People without experience come around and think they can replace an old tool with a better one". People tend to like thinking along the lines of "surely old things must be bad" and you'll get met with disagree-downvotes anytime you try to explain why you want old software in fields like crypto or internet routing. Try defending BGP in any of the DDOS or hijack threads and you'll be met with the same fate.
> People without experience come around and think they can replace an old tool with a better one

Just for posterity, the tool linked above that everyone is discussing here was written by the Go Lead Dev for cryptography and security at Google.

> I wish the downvoters would explain what's wrong with the parent comment,

One obvious guess is contradicting Latacora. 'tptacek is well known here, his name alone give significant weight to anything he writes.

In any case, they have valid points. PGP was written at a time we didn't understand cryptography as well as we do now. We can do better. Have done better, if half of what I've heard about Age is true.

Absurd complexity: We can definitely do simpler than PGP, at no loss of functionality.

Swiss Army Knife design: I think I disagree with Latacora there. Doing many things doesn't mean you have to do them poorly. There's no material difference between having 3 programs, and having one programs with 3 options, at least on the command line. If PGP does anything poorly, it's for other reasons.

Mired In Backwards Compatibility: well that depends. It makes sense that PGP can decrypt old obsolete ciphers & formats. Ability to encrypt to those same old stuff wouldn't. For instance, PGP longer be able to generate RSA keys at all. Then, one year later, once all RSA keys have expired, new PGP versions should no longer be able to encrypt to RSA keys at all. (In an ideal world. More realistically, we should wait a couple more years.) Only the ability to decrypt old messages should be kept until the end of times.

Obnoxious UX: I don't know enough to have an opinion.

Long term secrets: Sure they're bad, but I don't think we can avoid them. People need your public key to send you anything, so it can't be too short lived. My guess here is that Latacora is attacking the whole file encryption + web of trust thing, not PGP in particular.

Broken Authentication: If attackers can trick PGP decoders into decrypting forged messages, that's fairly critical, and should be fixed even if it breaks backwards compatibility (we could have an optional `-legacy` flag or something to compensate). Now if you go sign and encrypt… well there are two possibilities: if you sign then encrypt, you run into the cryptographic doom principle: the decoder will decrypt then verify, which creates the temptation to process unauthenticated data. Many vulnerabilities have been caused by such errors. If you encrypt then sign, you reveal to the entire world that you signed this particular ciphertext. Not the kind of data most people would like to leak. In my opinion what we really want is authenticated key exchange followed by AEAD. With the Noise X pattern, you'd even hide your identity from snoopers.

Incoherent Identity: Okay, they're clearly attacking the very notion of web of trust, not PGP specifically. They say it doesn't work, but I'd like to know why. First, I'm not sure I want to take their word for it, and second, the causes might be fixable.

Leaks Metadata: that one is clearly avoidable. Noise X for instance uses an ephemeral key to encrypt the transmitted public keys, and the recipient's key is implicit. Can't know who the message is for (nor from) without the recipient's private key.

No Forward Secrecy: Different use case indeed. Again, Latacora is attacking the very notion of file encryption, not PGP specifically.

Clumsy Keys: I'm with Latacora on this one. The 50 lines SSH keys are clearly RSA based, and as such obsolete. Modern keys use elliptic curves, and those take one line, which is more easily copy & pasted in various contexts. Arguably a detail, though.

Negotiation: It's more than just backwards compatibility. Backwards compatibility can be achieved with a simple version number. If instead we have a range of algorithms to chose from, things get more complicated. Now, you can't avoid the need for different kinds of encryption: you can send against a public key, or you can encrypt with a password. Possibly both. Beyond that however it's s...

> By default, GPG keys expire in a year.

This speaks to the article's complaint that GPG is usually the wrong tool for the job. For example if you just need to transfer a file securely (and have a fast, reliable internet connection on both ends and don't need to worry about active tracking of metadata), you can use Magic Wormhole (or a similar PAKE system) to do it. Imagine two scenarios: one with GPG and one with PAKE, and in both cases an adversary captures a ciphertext. With GPG if they can get your private key 6 months later, you're screwed. With PAKE the keys used to exchange the data are ephemeral, and so this isn't even a possibility.

> > Broken Authentication

> I've never heard of any of this.

I believe this is referring to authenticated encryption (AEAD), which is definitely valuable and GPG does not provide. AGE does.

Most of the other stuff you mention also falls under "wrong tool for the job". If you want a better argument, I'd talk about GPG having a web of trust system built in. On the other hand I think it's an open question whether this has ever brought real value to anyone. We have enough other secure messaging systems that it's no longer necessary for a single program to get there on its own. Usually you can count on some other mechanism to confirm your contact's identity.

>With PAKE the keys used to exchange the data are ephemeral, and so this isn't even a possibility.

AFAIK this isn't really true. If the adversary captures the initial key exchange plus all the data (ie the full transaction), then later discovers your PSK, they'll be able to decrypt. The only case where this helps you is if they capture some packets out of the middle without the initial handshake.

>authenticated encryption

It doesn't matter if your ciphertext is authenticated if it's both signed and encrypted, which is what I was getting at. In a normal TLS-like encrypted conversation, yes AEAD is very useful. But it's not applicable here.

> AFAIK this isn't really true. If the adversary captures the initial key exchange plus all the data (ie the full transaction), then later discovers your PSK, they'll be able to decrypt. The only case where this helps you is if they capture some packets out of the middle without the initial handshake.

Someone can correct me if I'm wrong, but I believe the idea behind a PAKE is that the password only authenticates the key exchange and doesn't contribute to it. So if you record all transmitted data you still need to break the key exchange which should have used a bunch of random bytes from both parties that are thrown away after use. The password is only there to prevent MITM, not to derive keys.

I believe magic wormhole uses SPAKE2, which has perfect forward secrecy. When using passwords to secure transmitted files it's really important to have forward secrecy otherwise you risk the transmission being recorded and the password being attacked offline which depending on your password strength might lead to trivially decrypting the data.

> Someone can correct me if I'm wrong, but I believe the idea behind a PAKE is that the password only authenticates the key exchange and doesn't contribute to it.

That's right. From memory, the passwords are just used to do a DH key exchange. The key is entirely ephemeral. Even if the entire ciphertext is captured, and even if the adversary then gets your password, they can't decrypt. To decrypt you'd have to MITM the key exchange, which would require knowing the password before the file is exchanged.

What no one seems to mention is that Magic Wormhole depends on the maintainer's server to negotiate transmission between clients. I don't like that requirement. Better to GPG encrypt and use bittorrent for a direct transfer. At least then we're using public trackers instead of some private server.
How would public BitTorrent tracker servers be better than a single rendezvous-server run by the tool's author?

With trackers, you're revealing the fact-of-transmission, transmission-size, & endpoints to any number of unknown remote parties. Potentially, attackers not even on the privileged network-path from origin to destination could tee off a copy of your encrypted data for offline analysis.

With Magic Wormhole's rendezvous-server, only one server, run by the same person whose code you're trusting (& can audit), briefly relays encrypted control-messages. (It might even be limited in its ability to deduce the size of the transfer – I'm not sure.) And if that's still too much, you can run your own rendezvous server.

It seems to me the amount of information leaked in the BT Tracker approach is strictly (& perhaps massively) more, to more entities, than that leaked in using the Wormhole author's server.

What it more specifically that you’re concerned about with regards to the server?
Its not a requirement that you use the default Wormole rendezvous or transit server. Its all open source you could run your own private servers if you wanted to (and its easy).

Of course the nice thing about Magic Wormhole is that its security does not depend on server components to be trustworthy. Use the default servers, use your own, use a different third party server, it doesn't matter, your data is still secure.

Edit: If you are worried about privacy magic-wormhole supports transit over tor.

Disagree.

There are many tools which are way older than gpg, but still in the wide use. For example, I have not heard any arguments that people should stop using things like "rsync" or "curl".

I don't see how any of the points in the article are relevant to the symmetric encryption method I posted. Yes, GPG can do a lot, and yes, some parts of it are kinda ridiculous. But symmetric pre-shared-key encryption/decryption is a solved problem, and I'd much rather trust GPG than some random's git repo.
The author of Age is definitely not some rando.

But if you like GPG then feel free to continue using it.

gpg symmetric encryption just derives a key from the password so theres no forward secrecy. Using bittorrent in particular opens you up to the risk of third parties getting ahold of your encrypted file and without forward secrecy the can attack the password offline which is typically much smaller that 256 bits.

Using Magic Wormhole would seem to be far more secure than gpg and bittorrent since it has less risk of third party interception and uses SPAKE2 which has perfect forward secrecy.

If I had to use bittorrent to transfer a sensitive file I would generating a random key and encrypt, then share the key over a different channel.

One reason age is beneficial is that it will generate a sufficiently long password for you when encrypting a file, This is a good idea if you're using a transmission protocol without a handshake. I don't think gpg has this as an option?
SFTP Chroot server. lftp client using mirror subsystem + sftp. It is multi-threaded, even for a single file and supports rsync like behavior even in sftp chroot. I can max out any internet link using lftp (client) + sftp (protocol).
I self-host a Nextcloud instance. If it's too big for a browser download, Bittorrent is the way to go.
This is interesting. What conditions would make a file too big for a browser download? If the server supports range requests (I'm almost certain Nextcloud does), then the browser can download in chunks and just retry any failed chunks.
It's technically possible, but most browsers don't auto-resume, and most webservers will timeout a request after a certain amount of time has elapsed. You can get around all this with browser extensions, but it's honestly easier to use a more robust protocol (this is in the context of, say, a 10TB file that'll take you a few days to download).
I don't see how anything would be much more robust than HTTP. Any network based protocol can fail at any time, so you're going to have to have some concept of tracking your progress and retrying failed requests. Both are easy to do with HTTP. Can you give a concrete example of something that would work better than HTTP?
Here's an exercise for you:

In your browser, start downloading a test file, like [1]. Turn off your network card for a few seconds, then turn it back on. See what your browser does.

Next, start downloading a large file over Bittorrent, like [2]. Turn off your network card for a few seconds, then turn it back on. See what the client does.

[1] http://speedtest.tele2.net/100GB.zip

[2] https://cdimage.debian.org/debian-cd/current/amd64/bt-dvd/de...

This has nothing to do with HTTP. It's just poor behavior on the part of the browser.
as a game of thrones fan:

a raven with a sufficient large micro-sd card ;)

Sending a 10TB file on my internet connection would take 2.5 years of constant uploading. Shipping a hard drive is cheaper and quicker.
8x10,240,000Mbits /2.5/365/86400 = 1.03Mbs?
That's slightly faster than my Silicon Valley upload speed.
Google cloud storage. Ui is actually nice, permissions are flexible and has both a robust web uploader as well as a cli one. Not to mention it’s cheap and only billed by the GB/hour or something.
Personally I would use my own Nextcloud instance for up to 20-30GBs. Not sure about TBs.

What about using "Firefox Send"? (I never used it so far)

https://support.mozilla.org/en-US/kb/send-files-anyone-secur...

I read that the limit is 1-2.5GBs => maybe you could break down the file and upload it in multiple pieces... .

EDIT: oopps, Firefox Send doesn seem to be available anymore - https://support.mozilla.org/en-US/kb/what-happened-firefox-s...

People were using Firefox Send to "send" malware and other bad things to people. That's what happens if you let anyone anonymously host files on your domain for free.
And how other platforms, like wetransfer, are dealing with that?
python -m http.server
upload to s3 bucket, make public, send link
ngrok.io and similar reverse tunnels
>I'm also curious on how you'd send 10TB or more.

For my 12TiB of data I use Syncthing when I need to sync them more often, rsync.

I used rsync several times for billions of smaller files totalling to 300GiB, but really all depending on how I connect nodes. I prefer syncthing, but when only ssh is available, then rsync is good too.

Currently largest synced directory by syncthing (that shares usage stats) is over 61384 GiB :)) https://data.syncthing.net/

Syncthing works pretty well in my experience, even across NAT. I recently shared 10GB worth of photos with a family member and ~30min later it was done.
Upload to cloud storage then have the other party download it.

If it's too big for cloud storage, ship a hard drive in the mail.

Private Nextcloud. Or some free tier OneDrive/alternative is enought for me.

However no-one mentioned a super simple service: https://wetransfer.com/ - Simple as drag & drop, enter recipient address, SEND. Pretty simple if you want non techie to send you something.

+1 for WeTransfer.

A dead simple UI and a link is nicely emailed to the receiver.

Also emails the sender when the receiver downloads it.
I can recommend WeTransfer as well. Super simple interface, no sign-up required, free.
I work with geneticists and they all seem to use wetransfer for big data files.
Wetransfer is brilliant. I’ve used their white label option in the past to transfer files to customers and allow them to send in files. It’s ace.