This is a problem I used to solve with 'zmodem' which you can install on both ends and use to perform text transfers, solely over the interactive channel.
I think the binaries were named 'zs' and 'zr'. It's been a long time since I needed to do this anywhere...
Zmodem, xmodem, kermit. Don't know which one's better etc.
Last time I used one of these (don't remember which) was when my ISP booted me off cause of not paying bill, but minicom still worked, and gave me a shell. This was around 2005.
You're close on the names of the binaries. They're 'sz' and 'rz'.
I used the crap out of them dialing-in to systems back in the day. I had TELNET and FTP access to the Internet out of a dial-in to a shell on a Unix host in the mid 90s (no SLIP, PPP, or SLiRP from the provider at the time. It was great to be able to ftp down files to my home directory then 'sz' them back home.
I wish terminal emulators still supported zmodem natively like some of the better emulators many years ago. It's not too difficult to configure iTerm2 to invoke rz when detecting the escape sequence[1], but it's still too much friction. sz/rz is something I would be using everyday, but I prefer sticking to Terminal.app on macOS--I prefer zero-config environments because of all the various systems, local and remote, I interact with on a daily basis.
scp and sftp are great, but the nice thing about in-band file transfers is that your shell session already has the current context for simplifying the transfer. For example, you often want to send to or receive from the current working directory, or something nearby, and interactive file globbing is just a single tab key away, without having to switch sessions, login all over again, copy paths, etc.
[1] I don't miss automatic printing without confirmation, though. The prank of `cat /vmlinuz | write user` wasn't that funny when the service desk hands you an entire ream of garbage along with a reprimand.
The instructions in this article should work, but they're more complicated than necessary maybe. If you don't have scp, you can usually just do something like this:
ssh user@host cat remote_file > local_file
to copy the contents of "remote_file" on the server to "local_file" on your local machine.
Scp works fine, I use it all the time. It's less complicated than sftp, and works basically just like the regular 'cp' command. It also doesn't require an sftp server to be present on the remote machine.
I'm not 100% sure I buy those "it's less secure" arguments in that Stack-exchange post, but even if true - lots of people use SSH to move stuff around inside an intranet, where both ends of the connection are trusted. In that kind of context, scp works great.
Which parts of the facts listed do you not "buy" ?
The fact that a remote server gets to silently respond to the request "Hey, I'd like funny-cat-noises.mp3" by writing two files, funny-cat-noises.mp3 and also .security/overrides.conf ?
Maybe your understanding of the protocol was that you allow a remote server to write whatever files it wants to your disk. That seems like a pretty crazy thing to want, but OK.
Most people assume it does what it seems like it does. You name one or more files on one machine, and they're copied to the other machine. SFTP can deliver without nasty surprises, but SCP can't really do so, although the OpenSSH team have worked pretty hard to make the worst surprises less likely to bite you.
> Maybe not an issue if you trust the machine you're copying to/from.
Maybe you shouldn't. After all, the remote machine doesn't have root on your machine. Even if you are root on the remote machine, how can you know the server isn't compromised? You can't.
> The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead. [1]
> From The Jargon File (version 4.4.7, 29 Dec 2003) [jargon]:
> deprecated
adj.
> Said of a program or feature that is considered obsolescent and in the
process of being phased out, usually in favor of a specified replacement.
[..]
> The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead.
I'm not sure what the misunderstanding is here.. The above text says there are better protocols and recommends their use over scp, but doesn't say scp is being phased out or discontinued in the future.
On *ix things stay around for pretty long. You still have ed and ftp on Ubuntu 20.04!
I have a convoluted method in my blog post due to the constraints I was working with. The SSH connection to the host goes via an SSH gateway that forbids the execution of remote commands without a login shell. I have laid out this constraint in the first paragraph of my blog post.
Without this constraint, yes, I agree what you mention would be the obvious straightforward solution.
I've found the base64/copy-paste trick useful when I can't do a conventional SSH. Usually this is when one of the machines is accessed via a serial console.
In that scenario, it can help to compress the data before base64-ing it:
They mention that you can’t run commands without an interactive shell. I believe that knocks out your proposal because then rsync+ssh would work without all the complexity you have in your post with all the same benefits. Could be wrong though.
I do wonder if there's a workaround for that, though. "bash" has the "-l" option for "Make bash act as if it had been invoked as a login shell", and also "-i" forces interactive mode. For ssh, "-t" is "Force pseudo-terminal allocation"; would that work?
> Usually this is when one of the machines is accessed via a serial console.
I've found zssh can be quite handy for that, it layers an extra pty over the SSH connection, for file transfer. By default it uses the ZMODEM protocol via rz/sz, but any pair of programs can be plugged in instead - useful if the connection isn't 8-bit clean.
ssh hostname tar cvjf - /path/to/folder | tar xjf -
Basically I ask ssh to execute tar on the remote host to create a compressed archive. ssh will output the archive contents on the local host; this data is then passed on to a local tar for extraction.
I do mention in my blog post that the SSH gateway in between forbids execution of remote commands without a login shell, so this rules this out as the solution. Otherwise, yes, this would have been a fine solution.
> Honestly, I find it distasteful to have to spend time working around somebody's incompetence at securing systems.
> Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time.
While it may be distasteful, what's the alternative? Refusing on principle to use a system configured in a way you don't like is way more likely to hurt you than it is to hurt anyone else, especially the person who configured the system.
Even if you are doing work for someone else (which, I think, is not indicated in the post), so that that person will be affected by your principled refusal, there's no guarantee that they're the ones who misconfigured the environment in which you're operating.
Your patience in (re-)explaining this constraint to everyone in this thread who thinks you don't know your way around standard Unix tools is impressive. :-)
Probably want to prepend a prefix so you can extract only those lines and filter banners, motd's, etc, like:
"... base64 | sed -e 's/^/~/'" | sed -ne 's/^~//p' | base64 ...`
BTW, base64 isn't a standard command (nonexistent on OpenBSD), and the options can vary based on implementation even when it exists (decode is -d for GNU coreutils base64, but -D for the macOS command). You can whip up an alternative using od(1) and printf(1):
#!/bin/sh
# encode as backslash-escaped octal (\0NNN)
export LC_ALL=C
od -An -to1 -v \
| sed -ne 's/\([01234567][01234567]*\)/\\0\1/gp' \
| tr -cd '\\01234567\n'
#!/bin/sh
# decode backslash-escaped octal
export LC_ALL=C
while read -r L; do
printf "%b" "$L"
done
It might not be strictly POSIX compliant, but I don't think there are any environments where printf isn't 8-bit clean in the C locale. I've tested something similar (see below) on various versions of bash and d?ash, as well as /bin/sh on AIX, FreeBSD, macOS, OpenBSD, NetBSD, and Solaris.
You can implement base64 as a shell script using the above techniques, and do so without relying on awk, which isn't always available. If od(1) isn't available you can use dd(1) to extract stdin byte-by-byte and convert with printf(1), but that gets really slow. I've written an implementation that only needs an 8-bit-clean but otherwise POSIX compliant shell, printf, and either dd or od. It doesn't need tr, but it's even slower without it. It just converts the bytes to decimal and uses plain shell arithmetic for the primitive encoding and decoding, similar to how it would be done in any language.
It certainly isn't a complete cross platform solution but to address the inconsistent options problem and some of the availability issues openssl could be used instead.
This seems like a CTF trick. If I were faced with this problem, I would do an SSH local port forward, and then just wget the file, hosted locally with python -m SimpleHTTPServer
Or you can open SSH back to your local port forwarded server from the remote SSH shell. At that point you can pipe whatever you want to/from your local server using the standard SSH commands.
I mentioned in my blog post that there is an SSH gateway in between. I tried this on my local system:
ssh -L 9000:127.0.0.1:8000 user@host
But when I visit http://127.0.0.1:9000/ on my local system, I get a connection reset by peer error. At the same time, in the terminal where I have set up the port forward with the ssh command above, I get this error:
open failed: unknown channel type
The SSH gateway is forbidding port forwarding as well.
ssh -W requires that the gateway support port forwarding, which is probably disabled if its enforcing interactive logins. (-W also implicitly sets -T, though I don't know if that would fail the login or not.) Likewise, -J is shorthand for a ProxyCommand that runs nc on the gateway, which also won't work if the gateway requires interactive (i.e. TTY interposed) logins. (EDIT: Looking at the OpenSSH source code, -J is shorthand for generating a ProxyCommand using ssh -W. Either it was refactored to use -W after -W was added, or I'm misremembering how -J used to work.)
I do something similar to move small files via copy-and-paste to/from hosts that I have to reach through an intermediate jump host.
Rather than copying the file to the jump host, then to the remote host, I'll just gzip and base64 encode it, then copy and paste it to the remote host.
I've done this to get files into VMs that have poor or no network connectivity, too. If I can get a console session w/ the ability to paste keystrokes I'm golden... >smile<
That's a litmus test for me when working with other technicians. If they need to transfer a file to me across a shared LAN and they offer to pipe the file over Netcat (and they just expect that I know what they mean) then they "rate". If I suggest Netcat and they understand they "rate". If they have no idea what I'm talking about or, worse, suggest a "cloud" service, then I file them appropriately.
Edit: Just to be clear, I'm talking about transient situations-- like when I'm working with another contractor on a day-long gig, etc. Obviously, for an established LAN piping files to each other over isn't a sensible or secure file transfer method.
I do not know why you are being down-voted. I would expect people to know what netcat is, and what it is used for. If all someone can come up with is related to cloud, then I would dismiss them, too. These "problems" have been solved a very long time ago, and we have great alternatives.
For myself I have written two shell functions; nc_send and nc_recv. It uses Nmap's ncat with SSL. It uses GPG and to verify data integrity, I use b2sum. Simple, yet perfect for the task. I use it often besides scp, but of course sometimes netcat is all I need.
I wouldn't necessarily "dismiss" somebody, but knowing about simple tools is a proxy in my mind for experience level. I can tailor my communication appropriately when I have some idea of experience level. If I know I can't say to somebody plugged-in to my laptop directly w/ a patch cable "Give me the last two octets of your APIPA address and start listening on port 31338 piped out to a file" then I know that there are a lot of other tasks that I'll need to break down in minute detail.
Okay, you are right. I would not dismiss them either for this alone, I would first perhaps ask them why they think that it is the best option, or what alternatives there are, and so forth.
> Give me the last two octets of your APIPA address
Is not APIPA Windows-only, by the way?
I can definitely give you two octets of an IP address through using awk, sed, or using C and bitwise operations. Do I pass? :D
APIPA was standardized as IPv4LL in RFC 3927[1]. I never remember the acronym IPv4LL-- I always remember APIPA instead. It was a Microsoft construct, but it has fairly broad OS support now, to my knowledge.
67 comments
[ 34.1 ms ] story [ 1334 ms ] threadI think the binaries were named 'zs' and 'zr'. It's been a long time since I needed to do this anywhere...
Wikipedia (https://en.wikipedia.org/wiki/ZMODEM) says that this came out of the BBS days. Boy, that takes me back.
Last time I used one of these (don't remember which) was when my ISP booted me off cause of not paying bill, but minicom still worked, and gave me a shell. This was around 2005.
I used the crap out of them dialing-in to systems back in the day. I had TELNET and FTP access to the Internet out of a dial-in to a shell on a Unix host in the mid 90s (no SLIP, PPP, or SLiRP from the provider at the time. It was great to be able to ftp down files to my home directory then 'sz' them back home.
scp and sftp are great, but the nice thing about in-band file transfers is that your shell session already has the current context for simplifying the transfer. For example, you often want to send to or receive from the current working directory, or something nearby, and interactive file globbing is just a single tab key away, without having to switch sessions, login all over again, copy paths, etc.
[1] I don't miss automatic printing without confirmation, though. The prank of `cat /vmlinuz | write user` wasn't that funny when the service desk hands you an entire ream of garbage along with a reprimand.
I’ve used it in datacenters where I’m ssh’d through three intermediate hosts, or when I’m inception-ing down into containers and vm’s
Tweeted about zmodem just last night! https://twitter.com/ryancnelson/status/1306444057602711553
scp is deprecated [1]; use sftp instead. At the very least, sftp should be your preference over scp.
[1] https://unix.stackexchange.com/questions/571293/is-scp-unsaf...
I'm not 100% sure I buy those "it's less secure" arguments in that Stack-exchange post, but even if true - lots of people use SSH to move stuff around inside an intranet, where both ends of the connection are trusted. In that kind of context, scp works great.
The fact that a remote server gets to silently respond to the request "Hey, I'd like funny-cat-noises.mp3" by writing two files, funny-cat-noises.mp3 and also .security/overrides.conf ?
Maybe your understanding of the protocol was that you allow a remote server to write whatever files it wants to your disk. That seems like a pretty crazy thing to want, but OK.
Most people assume it does what it seems like it does. You name one or more files on one machine, and they're copied to the other machine. SFTP can deliver without nasty surprises, but SCP can't really do so, although the OpenSSH team have worked pretty hard to make the worst surprises less likely to bite you.
Maybe not an issue if you trust the machine you're copying to/from.
Maybe you shouldn't. After all, the remote machine doesn't have root on your machine. Even if you are root on the remote machine, how can you know the server isn't compromised? You can't.
the sftp command has lousy ux, it's unlikely people will switch to it en masse.
> From The Jargon File (version 4.4.7, 29 Dec 2003) [jargon]:
> deprecated adj.
> Said of a program or feature that is considered obsolescent and in the process of being phased out, usually in favor of a specified replacement. [..]
[1] https://www.openssh.com/txt/release-8.0
On *ix things stay around for pretty long. You still have ed and ftp on Ubuntu 20.04!
Without this constraint, yes, I agree what you mention would be the obvious straightforward solution.
In that scenario, it can help to compress the data before base64-ing it:
I do wonder if there's a workaround for that, though. "bash" has the "-l" option for "Make bash act as if it had been invoked as a login shell", and also "-i" forces interactive mode. For ssh, "-t" is "Force pseudo-terminal allocation"; would that work?
I've found zssh can be quite handy for that, it layers an extra pty over the SSH connection, for file transfer. By default it uses the ZMODEM protocol via rz/sz, but any pair of programs can be plugged in instead - useful if the connection isn't 8-bit clean.
Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time.
> Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time.
While it may be distasteful, what's the alternative? Refusing on principle to use a system configured in a way you don't like is way more likely to hurt you than it is to hurt anyone else, especially the person who configured the system.
Even if you are doing work for someone else (which, I think, is not indicated in the post), so that that person will be affected by your principled refusal, there's no guarantee that they're the ones who misconfigured the environment in which you're operating.
Rather than pull via SSH I login to the remote interactive shell and dump the file onto my own server via SSH from that remote shell.
You can implement base64 as a shell script using the above techniques, and do so without relying on awk, which isn't always available. If od(1) isn't available you can use dd(1) to extract stdin byte-by-byte and convert with printf(1), but that gets really slow. I've written an implementation that only needs an 8-bit-clean but otherwise POSIX compliant shell, printf, and either dd or od. It doesn't need tr, but it's even slower without it. It just converts the bytes to decimal and uses plain shell arithmetic for the primitive encoding and decoding, similar to how it would be done in any language.
A possible workaround for this specific scenario would be to use 'sshutle' which runs a python script on the remote end to terminate the traffic.
Except in your specific scenario you'd have to somehow trick it to execute using a login shell like one of the other comments shows.
Having said all of that, I would just do what you did :)
Huh. I'm surpised this works, because $ is not escaped.
https://pubs.opengroup.org/onlinepubs/007904975/basedefs/xbd...
Rather than copying the file to the jump host, then to the remote host, I'll just gzip and base64 encode it, then copy and paste it to the remote host.
Edit: Just to be clear, I'm talking about transient situations-- like when I'm working with another contractor on a day-long gig, etc. Obviously, for an established LAN piping files to each other over isn't a sensible or secure file transfer method.
For myself I have written two shell functions; nc_send and nc_recv. It uses Nmap's ncat with SSL. It uses GPG and to verify data integrity, I use b2sum. Simple, yet perfect for the task. I use it often besides scp, but of course sometimes netcat is all I need.
> Give me the last two octets of your APIPA address
Is not APIPA Windows-only, by the way?
I can definitely give you two octets of an IP address through using awk, sed, or using C and bitwise operations. Do I pass? :D
[1] https://tools.ietf.org/html/rfc3927