37 comments

[ 1.3 ms ] story [ 63.6 ms ] thread
It’s amazing what you can learn by reading the manual.
It's amazing how badly written most manuals are.
When I see one of these with obvious AI tells at the top (sentences lacking a subject or verb), I ask myself:

Can’t I just open up a harness and prompt “Teach me how to do X?”

The article mentions bastions, but no jumphosting?

  ssh -J user1@bastion1,user2@bastion2 targetuser@targethost
Edit: Jumphosting was introduced in OpenSSH 7.3 2016-08-01.

https://www.openssh.org/releasenotes.html

>ssh -J user1@bastion1,user2@bastion2 targetuser@targethost

Are you using SSH key auth or password authenticating three times when you do this?

For me, this is always used via ProxyJump rules in my ~/.ssh/config

It is also nice that it works recursively, so I can logically structure my rules so that the one for my regular targets say to use bastion1, then the rule for bastion1 says to go via bastion 2, etc.

I find this easier to reason about and maintain rather than juggling a bunch of these multi-step rules.

And with match/exec rules you can always connect to MyHost and make it conditional whether to use a jumphost or not, so it's like an on demand vpn.. only with ssh.

    Match host="MyHost" exec "! grep Home ~/.wifi-loc-control/.current"
    ProxyJump home-jumphost.mydomain.tld
What I've found beautiful about -J is the host you jump through requires no privileges on the final host. Only my laptop has the SSH key to access my home server, not my cheap VPS.

And this allows me to have zero open ports on my home internet. I do a reverse tunnel to my VPS from my home server (in a FreeBSD jail), and that port is what my laptop client jumps through.

There's a asymmetry here that "-R" works both for reverse static and dynamic (using SOCKS protocol) forwarding, but "-D" is required for dynamic forwarding which "-L" cannot do.

Why is that?

It's historical. Some older flags could be easily extended for dynamic port support and others could not.
As a sysadmin, one of your biggest ROI is learning the ins and outs of SSH.
Learning how SSH port forwarding is great as a pseudo-vpn for everything from GUI-client database access to (in physical infra) access to web-admin tools for appliances.

The socks proxy support can also deal with bad web filtering and privacy issues on public wifi networks (though nowadays if you're ssh'ing to a cloud IP, you'll get lots of "bot" restrictions).

Yeah, I get use out of the SOCKS proxy mode in combination with a "split VPN" at work.

I need VPN to get into some internal resources via SSH, but there are lots of external/public/AWS resources I also need to access, and the full VPN adds too much overhead and fragility for those.

Using the available split VPN, I can point a browser instance at a localhost SOCKS proxy port to relay over SSH + VPN for other web resources I need to access internally.

Unfortunately, Firefox proxy config rules are sort of backwards for my needs. I want to say "only use proxy for these 3 domains" whereas it wants to use the proxy by default and only allow me to bypass specific domains.

Firefox can be used with socks proxies nicely.

- Use Multi-Account Containers plugin, where you define which domain should be opened in which container.

- Use Container Proxy to configure which socks proxy to use for different containers.

Pair that with ssh tunnel that you use as socks proxy and you have nive tunneling solution for browser.

Or use plugins like foxyproxy that implement transparent rule-based routing without containers.
> The socks proxy support ...

I just love SOCKS proxy in SSH tunnels: at some point I had a dedicated server (on a fixed IP) with countless machines (usually headless Pis dropped at a family member's place and/or SME office) automatically setting up, 24/7, reverse tunnels to that dedicated server.

Then I could, from anywhere, both access their LANs (to fix stuff) and have a browser, running locally, pretending to be in this or that country.

Basically because I had all those reverse tunnels always there, I could always decide how to use them (just SSH in or SOCKS in etc.).

Or you could just install something like Tailscale and never have to think about it again.
or tell an llm to do it for you
I'll mention it here, because I learned about it here.

"~C" will drop you into the SSH command line, allowing you to, among other things, effect port forwarding

  -L8080:localhost:443
Learning that "~C" exists, and what you can do with it, has supercharged my use of SSH tunnels, which were already awesome on their own.

But for some reason this has been disabled by default in more recent ssh configurations... to ensure its available

  -o EnableEscapeCommandline=yes
or, in your ~/.ssh/config

  EnableEscapeCommandline yes
(edit: formatting)
Eh, once I started using master sockets, I never went back. Problem with ~C is it's hard to keep track of what you have open.
Very refreshing to see a utilitarian series such as this. Disappointing that the latest ai drama gets 20x more discussion and visibility on this site
If you have many different remote devices behind NATs or firewalls, a cool trick to access them all via EC2 server (or such) is to setup Remote Forwarding via UNIX socket on the server side, to devices' port 22. Preferably, UNIX socket filenames should start with a common prefix, so an SSH config can be written that will use ssh+socat in a ProxyCommand to establish the connection.

It's amazing how lightweight this method actually is. I have managed to connect hundreds of devices using a single EC2 nano instance.

Should add how to bypass MFA using phishing and SSH Multiplexing to the article.
Need to mention sshuttle [0] here, as it magically solves a bunch of these problems without constant reconfiguration

[0] https://github.com/sshuttle/sshuttle

Yup, use it regularly in order to jump through networks where a VPN/Wireguard setup is not possible. It can also forward DNS requests and handle local NS operations reasonably well, such that it can be used as a low-key split-VPN client (i.e. only forwards traffic for a specific domain or IP range without redirecting any other traffic). Note that for integration with `systemd-resolved`, one needs to jump through a few hoops, but I feel its works very nicely: https://github.com/sshuttle/sshuttle/issues/688#issuecomment....
Another option that I never see mentioned anywhere is -w which allows you to create either layer2 or layer3 tunnels via a tun(4) interface.
My favorite use of this is peer-to-peer transfer of Docker images. The Docker CLI only allows you to use registries authenticated with HTTPS but there's an exception where it allows HTTP transfers over localhost.

So, if you use SSH tunneling to forward a port from localhost to a remote, then Docker unwittingly pushes to a remote. This is super useful "off the grid" with robotics/embedded applications where you don't want to bother with a registry and a good Internet connection.

Example, docker pussh: https://github.com/psviderski/unregistry

Thanks! I will keep this for reference. I use ssh alot but thie reminds me that I can learn new ways of tunneling :-)
Weird.. I have almost this exact same "cheat sheet" printed out and stuck up on my whiteboard.. except it's slightly different. Only 4 panels instead of 6 and the panels don't have titles.

I know this is a solid "cool story bro" moment, but whatever hah

BTW I use this and a systemd unit file and SSH tunnel my Jellyfin to a public VPS to its local host.

I then use nginx to proxy it.

Because its a unit file, sshd reconnects if my ISP's IP changes. Does so within 30s. Also hides my ISP IP in case I have to turn it off.

And no data is effectively on the VPS. Its just a mostly empty machine.

Always annoys me -p(port) for ssh clashes with -p(reserve permissions) for sftp and scp. Unavoidable given history.

Also annoys me we "invented" ssh:// url format after the tools were baked so it's a somewhat odd bonding into the model.

The best part of this is you can daisy chain tunnels and have a network connection between any machines you can ssh into.

If you have a private server with a public IP, you don't need localtunnel etc.. you can just use ssh tunnels to expose your home network services over public IP.

I think that the start of the article is at least a bit exaggerated:

    SSH is yet another example of an ancient technology that is still in wide use today.
Ancient technology? If it was telnet or FTP... But SSH is much younger than, let's say, IPv4, which is _maybe_ ancient technology still in wide use today.