It's obviously not magic. In practice, I find this feature difficult to use due to the distribution of ssh keys necessary to be practical.
Actually, scp got quite old. Rsync typically makes a better job, especially with many files or unreliable networks. Streaming tar.gz over ssh is again typically faster for many files.
rsync however does not work across two remote hosts, at least not without trickery that one cannot remember. scp at leasts supports easy invocation of a two-remote-hosts scenario but not a direct host-to-host communication, you will always be limited by the client bandwidth.
My favourite magic trick is dumping db on one host, gzipping on the fly, tunneling through ssh to other host, then ungzipping straight into other database. Useful when you don't have enough place on disk for local copies.
Unless you can compress at a higher throughout than your network speed, don’t do this. I’m not familiar with too many machines that can gzip at greater than 120 Mbytes/sec, which is the network bandwidth of a switched gigabit network. There’s no point in making your task take longer with unnecessary compression steps. This also applies to encryption but that’s a much more context dependent decision.
That doesn't seem to be the case. A relatively fast machine can't gzip anywhere near 120 MB/s unless the data ultra compressible. Zstd is multithreaded, but single threaded it's speed isn't that fast either.
$ lscpu | grep name
Model name: Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
$ time dd if=/dev/zero of=/dev/null bs=10240 count=102400
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB) copied, 0.145168 s, 7.2 GB/s
real 0m0.147s
user 0m0.036s
sys 0m0.111s
$ time dd if=/dev/zero bs=10240 count=102400 | gzip > /dev/null
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB) copied, 5.36061 s, 196 MB/s
real 0m5.364s
user 0m5.317s
sys 0m0.472s
$ time dd if=/dev/urandom bs=10240 count=102400 | gzip > /dev/null
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB) copied, 27.3478 s, 38.3 MB/s
real 0m27.352s
user 0m27.169s
sys 0m10.119s
$ time dd if=biggestwords bs=10240 count=102400 | gzip > /dev/null
10642+1 records in
10642+1 records out
108981378 bytes (109 MB) copied, 6.66068 s, 16.4 MB/s
real 0m6.669s
user 0m6.667s
sys 0m0.249s
$ time dd if=biggestwords bs=10240 count=102400 | gzip -1 > /dev/null
10642+1 records in
10642+1 records out
108981378 bytes (109 MB) copied, 1.53539 s, 71.0 MB/s
real 0m1.540s
user 0m1.522s
sys 0m0.138s
$ time dd if=biggestwords of=/dev/null bs=10240 count=102400
10642+1 records in
10642+1 records out
108981378 bytes (109 MB) copied, 0.0407698 s, 2.7 GB/s
real 0m0.043s
user 0m0.007s
sys 0m0.036s
Biggestwords is simply /usr/share/dict/words concatenated to itself 22 times so as to provide ~100 MB of data.
If you can get good enough compression it might still be worthwhile to compress. But it's pretty rare in my experience.
Re: managing the SSH keys being annoying—I’d suggest setting up an OpenSSH `AuthorizedKeysCommand` in your instances (e.g. using a cloud-init script), that looks at some central repository of SSH keys.
This isn’t me telling you to implement some scary arcane wire-protocol, either. All an AuthorizedKeysCommand script/binary has to do is spit out a stream of SSH pubkeys considered valid to authenticate a login attempt on a given username. The “implicit default” AuthorizedKeysCommand is `cat “/home/$1/.ssh/authorized_keys”`. Your implementation just needs to do the same thing that that command does: take “pubkeys that mean this user could log into this box” from some source, and spit them at stdout. (Don’t want that user logging in to that box? Don’t write anything to stdout.)
I wrote my own little AuthorizedKeysCommand binary a while back (https://github.com/tsutsu/github-auth3) for using Github’s user-SSH-keys and user-org-membership APIs as such a repository. You tell it a Github org name, and it’ll stream out the username’s equivalent Github user’s SSH pubkeys, if-and-only-if that Github user is a member of the specified Github org. (The code is, like, 20 lines. Read it yourself!)
——
This is also one case where specific ecosystems can really smooth things over, though.
If you’re running in Google Cloud, for example, you don’t need to do any of the above. Instead, just configure “OS Login” (https://cloud.google.com/compute/docs/oslogin/), and then log into the gcloud(1) tool on your workstation as an IAM user that you’ve granted the “OS Login” role. Then it’s as simple as:
On first use, IAM will generate a new SSH keypair for the OS Login user. Your gcloud(1) keystore will grab the private half when it needs it; Compute Engine instances will hit IAM to ask for the public half when anyone tries to log in (probably using an SSH `AuthorizedKeysCommand` stanza; I haven’t looked into it.) The user account (`me_example_com` if your IAM user is `me@example.com`) will be created on the instance if it doesn’t already exist (but you can set up your instance template such that it does already exist, if you like, or set up roaming profiles using Filestore as an NFS host, or whatever.)
You can also log into your OS Login account through regular direct SSH/SCP to the instance’s external IP (this isn’t some custom proprietary version of SSH or anything), but you won’t have access to the IAM-held private key if you’re not going through gcloud(1), so you’d want to configure your IAM user with a custom authorized pubkey first (which your instances will also automatically see.)
Is it a bad practice to have the private key left unencrypted? It seems easily stoleable. On the other hand, if I had to type the decryption password each time I wanted to connect, I wouldn't see the benefits of the hassle (not even talking about the ability to easily change user password on the server if needed for security purposes)
[1] You need to add your authentication subkey's keygrip which you can get using: gpg --list-keys --with-keygrip or the like. Separate the keygrips with a newline, and IIRC you need to have a newline at the end of the file as well.
[2] For example: "sign_and_send_pubkey: signing failed: agent refused operation".
The benefit is having a much more secure "password" (the ssh key) than the password that you have to remember. The password you use to encrypt your key doesn't need to be a really strong one since it's only proupose is to give you enough time to change your key if your pc gets stolen or the key is leaked in some other way.
I went in expecting some kind of Tavis Ormandy type magic to copy files without authentication.
"You can authenticate with keys" wasn't the punchline I was expecting but perhaps if I had noticed the domain I may have set my expectations differently.
Seems like it is written for Windows users. They would use tools like filezilla etc for copying files. Besides it could be IBM effect, they used to publish fine articles on their developerworks website.
There is one piece of information, I think, that isn't obvious and needs to be communicated. Since I am anonymous, I am not embarrassed to ask even on HN.
If you copy a file using this method from host B to host C by issuing an scp command on host A, do the bits go directly from host B to host C? or do they go through host A on their way to host C ?
I suspect the latter is what happens on Windows network fileshares, because it takes significantly longer to perform a file transfer from \\hostB to \\hostC while in \\hostA than just doing it direct from \\hostB or \\hostC.
What happens in linux behind the scenes for such 3-party scp transfers?
scp has a -3 option so you decide whether the data travels through your local system or not
-3 Copies between two remote hosts are transferred
through the local host. Without this option the
data is copied directly between the two remote
hosts. Note that this option disables the
progress meter.
My spider sense when reading the headline was “this is going to be some reinvented BS that ultimately could be replaced by just using scp” so I’m also disappointed but simultaneously pleasantly surprised it’s just good ol scp.
Any technology demonstrated to a sufficiently unsophisticated audience is indistinguishable from sufficiently advanced technology, which is indistinguishable from magic.
If SCP/SFTP is not an option and you've got netcat installed on the target you can definitely perform some magic. now of course this does require you to log in on the destination though
Note: you can do the opposite as well to achieve the same thing, e.g. listen on the "sending system", in which case when you connect, you get the file.
> The scp protocol is outdated, inflexible and not readily fixed. We
recommend the use of more modern protocols like sftp and rsync for
file transfer instead.
I remember reading about this also, but never really received a good explanation. Implementation-wise SCP is a lot cleaner to use especially in code. It doesn't support some things like resuming a file, but it could, and in any case it's less of a pain than SFTP.
There was a recent CVE about writing to wherever the server specified but that does not necessarily reflect poorly on the protocol.
Modern rsync (err, 3.0+, so maybe not that modern any longer) can scan the directory tree concurrently with sending the files.
(I'm not sure how it's done, presumably interleaving readdir() calls with non-blocking sendfile, allowing page cache prefetching as well as the actual sending to be done in parallel by the kernel.)
Definitely, but mostly due to compression and change tracking. The canonical replacement for scp is sftp which is to me a nonstarter, I'm either using scp in code or for single files, or rsync for a lot of files. I would use rsync everywhere, but I need to make up a good alias, it needs too many flags by default.
> It doesn't support some things like resuming a file, but it could
... and if pigs could fly.. It could, but it doesn't, and openssh developers have made it quite clear they view scp as fundamentally broken and have no interest in improving it.
Scripting sftp is, while doable, tedious compared to scp. Then again, rsync is scriptable as well and offers much more than scp.
I wouldn't call sftp more modern. They're from the same time. One is interactive, and the other isn't. Nothing more. Nothing less.
So, if you're using it to load code for net booting, scp is the obvious choice since you want to automate the retrieval of a specific file from a specific place, to a specific destination. And you don't want to have to babysit it every time.
If you're copying some specific files with lots of randomish names from your livingroom entertainment server's Downloads directory to your laptop's Documents directory, sftp is your friend.
One doesn't supersede the other in any way whatsoever. There are so many things wrong with the idea that rsync is more modern.
Isn't rsync older by quite a bit? And since it is lacking encryption, isn't scp the more obvious tool to use?
You can't use rsync the way that you use sftp. Think of it this way:
* Secure direct: scp <-- encrypted
* Secure interactive: sftp <-- encrypted
* Insecure direct: rsync <-- clear text
* Insecure interactive: ftp <-- clear text
Use clear text if the network is absolutely known to be secure (good luck on that in any productive environment) and stick to scp and sftp or equivalent secure protocols.
Toss ftp and rsync to the dogs and pretend you've never heard of them except in dire emergencies where significantly better tools (like scp and sftp) aren't available.
> I wouldn't call sftp more modern. They're from the same time.
No. scp was included in Tatu Ylönens original ssh in 1995. sftp is significantly newer, e.g. openssh gained both client and server implementation with the 2.5 release in 2001, and is dependent on the ssh 2.0 protocol.
> One is interactive, and the other isn't. Nothing more. Nothing less.
No. They are different network protocols. And per CVE-2019-6111 mentioned in the link I provided in the previous post, the scp protocol has some questionable semantics, which is why the openssh developers are recommending people to switch to better designed protocols such as sftp or rsync.
> So, if you're using it to load code for net booting, scp is the obvious choice since you want to automate the retrieval of a specific file from a specific place, to a specific destination. And you don't want to have to babysit it every time.
You can automate sftp as well. Fetching files is as simple as with scp, but uploading is more tedious, but doable if, for instance, you don't have rsync available. Something like "sftp -b - user@server:/path <<< 'put filename'".
> There are so many things wrong with the idea that rsync is more modern.
rsync is not a fundamentally broken network protocol like scp. And as another answer mentioned, you can tunnel rsync over ssh, which nowadays even is the default if you don't specify otherwise.
When it comes to 4096 the only viable benefit seems to be the key size and performance. While I certainly agree with that and I wish I could use ed25519, I can't use it with my security keys.
Yubikey currently supports 4096 bit RSA keys as a part of GPG applet. There's a PIV module, however in terms of elliptic curves it only has ec256 and ec384 and I still can't get ssh and pkcs11 to pick up anything other than built-in 2048 RSA keys.
The article shows how to copy a file from a remote host to another remote host. How do you do that with rsync? I feel like there might be a way but I couldn't find it
[EDIT: That link gives basic instructions for rsync between your current machine and a remote one. I haven't confirmed that both endpoints can be remote.]
I was specifically wondering if one could do it between remote hosts.
Also in those instructions `-e ssh` might not be needed unless specifying additional ssh options or a different port, as later rsync versions ssh is often the default connection method.
I see people knocking it as, "it's too simple" but consider that there are those who are just learning all these things might not know that you can copy a file from one remote host to another remote host. That's a nifty trick.
There should really be a "things you should know about SSH" article. Then an inventory of similar articles like the awesome-thing type lists.
I'm guessing it already exists and I don't know what it's called? (Maybe it's basically cheat-sheets? But they tend to be cues for syntax, rather than concepts).
scp manpage alludes to the feature of copying files between hosts without logging in, but I've never got it to work or seen it work until now. This functionality must've been introduced recently.
69 comments
[ 3.9 ms ] story [ 139 ms ] threadActually, scp got quite old. Rsync typically makes a better job, especially with many files or unreliable networks. Streaming tar.gz over ssh is again typically faster for many files.
[0] https://coderwall.com/p/bmv0nw/dump-postgres-databases-throu...
# from host1:
cd $DIRECTORY && tar -jcf - . | ssh host2 "cd DESTINATION_DIR && tar jxvf - "
If you can get good enough compression it might still be worthwhile to compress. But it's pretty rare in my experience.
This isn’t me telling you to implement some scary arcane wire-protocol, either. All an AuthorizedKeysCommand script/binary has to do is spit out a stream of SSH pubkeys considered valid to authenticate a login attempt on a given username. The “implicit default” AuthorizedKeysCommand is `cat “/home/$1/.ssh/authorized_keys”`. Your implementation just needs to do the same thing that that command does: take “pubkeys that mean this user could log into this box” from some source, and spit them at stdout. (Don’t want that user logging in to that box? Don’t write anything to stdout.)
I wrote my own little AuthorizedKeysCommand binary a while back (https://github.com/tsutsu/github-auth3) for using Github’s user-SSH-keys and user-org-membership APIs as such a repository. You tell it a Github org name, and it’ll stream out the username’s equivalent Github user’s SSH pubkeys, if-and-only-if that Github user is a member of the specified Github org. (The code is, like, 20 lines. Read it yourself!)
——
This is also one case where specific ecosystems can really smooth things over, though.
If you’re running in Google Cloud, for example, you don’t need to do any of the above. Instead, just configure “OS Login” (https://cloud.google.com/compute/docs/oslogin/), and then log into the gcloud(1) tool on your workstation as an IAM user that you’ve granted the “OS Login” role. Then it’s as simple as:
SSH: `gcloud compute ssh instanceName`
SCP: `gcloud compute scp ./f.txt instanceName:` (or vice-versa)
On first use, IAM will generate a new SSH keypair for the OS Login user. Your gcloud(1) keystore will grab the private half when it needs it; Compute Engine instances will hit IAM to ask for the public half when anyone tries to log in (probably using an SSH `AuthorizedKeysCommand` stanza; I haven’t looked into it.) The user account (`me_example_com` if your IAM user is `me@example.com`) will be created on the instance if it doesn’t already exist (but you can set up your instance template such that it does already exist, if you like, or set up roaming profiles using Filestore as an NFS host, or whatever.)
You can also log into your OS Login account through regular direct SSH/SCP to the instance’s external IP (this isn’t some custom proprietary version of SSH or anything), but you won’t have access to the IAM-held private key if you’re not going through gcloud(1), so you’d want to configure your IAM user with a custom authorized pubkey first (which your instances will also automatically see.)
And 'scp -c' will automatically compress and decompress your file, and 'scp -R' will walk a file tree for you.
alias upload='ssh username@server'
Then when you need to upload something, just type: upload 'something'
s/ssh/scp/
[1] You need to add your authentication subkey's keygrip which you can get using: gpg --list-keys --with-keygrip or the like. Separate the keygrips with a newline, and IIRC you need to have a newline at the end of the file as well.
[2] For example: "sign_and_send_pubkey: signing failed: agent refused operation".
"You can authenticate with keys" wasn't the punchline I was expecting but perhaps if I had noticed the domain I may have set my expectations differently.
If you copy a file using this method from host B to host C by issuing an scp command on host A, do the bits go directly from host B to host C? or do they go through host A on their way to host C ?
I suspect the latter is what happens on Windows network fileshares, because it takes significantly longer to perform a file transfer from \\hostB to \\hostC while in \\hostA than just doing it direct from \\hostB or \\hostC.
What happens in linux behind the scenes for such 3-party scp transfers?
Any technology demonstrated to a sufficiently unsophisticated audience is indistinguishable from sufficiently advanced technology, which is indistinguishable from magic.
~~ Receiving system `# nc -l -p $arbitraryPort > /path/to/file `
~~ Sending system `# cat /path/to/file | nc $receivingSystemIP -p $arbitraryPort `
eg: `# nc -l -p 9669 > /home/user/staging/httpd.conf ` `# cat /etc/httpd/httpd.conf | nc 10.50.50.1 -p 9669 `
This method is a useful feather to have in one's cap when used in combination with tar and Pipeviewer `pv`.
I've used this several times doing a backup of a large database and when I don't have enough space locally to hold the copy.
https://www.openssh.com/txt/release-8.0
> The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead.
There was a recent CVE about writing to wherever the server specified but that does not necessarily reflect poorly on the protocol.
When copying a large number of small files rsync over ssh is much faster.
(I'm not sure how it's done, presumably interleaving readdir() calls with non-blocking sendfile, allowing page cache prefetching as well as the actual sending to be done in parallel by the kernel.)
... and if pigs could fly.. It could, but it doesn't, and openssh developers have made it quite clear they view scp as fundamentally broken and have no interest in improving it.
Scripting sftp is, while doable, tedious compared to scp. Then again, rsync is scriptable as well and offers much more than scp.
Is set on `sshd_config`.
So, if you're using it to load code for net booting, scp is the obvious choice since you want to automate the retrieval of a specific file from a specific place, to a specific destination. And you don't want to have to babysit it every time.
If you're copying some specific files with lots of randomish names from your livingroom entertainment server's Downloads directory to your laptop's Documents directory, sftp is your friend.
One doesn't supersede the other in any way whatsoever. There are so many things wrong with the idea that rsync is more modern.
Isn't rsync older by quite a bit? And since it is lacking encryption, isn't scp the more obvious tool to use?
You can't use rsync the way that you use sftp. Think of it this way:
* Secure direct: scp <-- encrypted
* Secure interactive: sftp <-- encrypted
* Insecure direct: rsync <-- clear text
* Insecure interactive: ftp <-- clear text
Use clear text if the network is absolutely known to be secure (good luck on that in any productive environment) and stick to scp and sftp or equivalent secure protocols.
Toss ftp and rsync to the dogs and pretend you've never heard of them except in dire emergencies where significantly better tools (like scp and sftp) aren't available.
rsync lately, and depending how it is compiled, defaults to using ssh, so it shouldn't be clear text in that case.
No. scp was included in Tatu Ylönens original ssh in 1995. sftp is significantly newer, e.g. openssh gained both client and server implementation with the 2.5 release in 2001, and is dependent on the ssh 2.0 protocol.
> One is interactive, and the other isn't. Nothing more. Nothing less.
No. They are different network protocols. And per CVE-2019-6111 mentioned in the link I provided in the previous post, the scp protocol has some questionable semantics, which is why the openssh developers are recommending people to switch to better designed protocols such as sftp or rsync.
> So, if you're using it to load code for net booting, scp is the obvious choice since you want to automate the retrieval of a specific file from a specific place, to a specific destination. And you don't want to have to babysit it every time.
You can automate sftp as well. Fetching files is as simple as with scp, but uploading is more tedious, but doable if, for instance, you don't have rsync available. Something like "sftp -b - user@server:/path <<< 'put filename'".
> There are so many things wrong with the idea that rsync is more modern.
rsync is not a fundamentally broken network protocol like scp. And as another answer mentioned, you can tunnel rsync over ssh, which nowadays even is the default if you don't specify otherwise.
[0] http://johnbokma.com/blog/2019/05/27/ssh-public-key-authenti...
Yubikey currently supports 4096 bit RSA keys as a part of GPG applet. There's a PIV module, however in terms of elliptic curves it only has ec256 and ec384 and I still can't get ssh and pkcs11 to pick up anything other than built-in 2048 RSA keys.
[EDIT: That link gives basic instructions for rsync between your current machine and a remote one. I haven't confirmed that both endpoints can be remote.]
Also in those instructions `-e ssh` might not be needed unless specifying additional ssh options or a different port, as later rsync versions ssh is often the default connection method.
alias bcp='rsync -avzu --progress'
I'm guessing it already exists and I don't know what it's called? (Maybe it's basically cheat-sheets? But they tend to be cues for syntax, rather than concepts).
Like zmodem back in the old days (which should work with SSH, but I didn't get it to ..?)
Anyone have a suggestion?