HN Explorer

Ask HN: moving beyond screen to persist SSH sessions?

49 points by sounddust ↗ HN
I work from my laptop, and my laptop is constantly moving between my home, cafés, libraries, schools, etc. I have SSH windows open and I want them to simply stay open and work all the time. I want the session to automatically persist changes in wireless networks, temporary lack of network connectivity, putting my laptop in suspend, etc. I also want my SSH-forwarded ports to keep being forwarded as long as I'm connected to the internet. It's such a huge pain to constantly re-login to all my SSH windows on every minor blip of connectivity.

Is there any way to accomplish this in any OS? Every time I have posted this problem elsewhere, everyone shouts "screen!" but I think the HN audience understands that while screen is cool, it doesn't even come close to resolving this issue; so I'm hoping for a better answer here.

To give an example of my problem:

Let's I have 5 SSH windows open doing various things (perhaps I am running top, emacs, a window that i'm using to type svn commands, etc). I also have port-forwarding set up so that I can use Firefox securely through a SOCKS proxy, and access websites that are only available from behind the SSH machine.

If my wireless network switches or even drops for 1 second, I now have 5 dead SSH sessions. I have to somehow reinitialize them, and when I do it's a mess to get them back in the correct state. I can solve the latter with screen. How can I solve the former automatically, without wasting time re-establishing those connections?

35 comments

[ 5.4 ms ] story [ 88.2 ms ] thread
gaius
A VNC or RDP session running on a stable host somewhere, running full screen on your laptop, so you only have one thing to reconnect to.

What you want would be cool but it can't be done with TCP/IP.

sounddust
VNC would be painfully slow, though.

it can't be done with TCP/IP. It surely can, but would have required someone to figure it out. Surely a combination of an SSH client which reconnects (and therefore temporarily remembers your password/key) with screen might do it. But I'm hoping someone has been in the same situation and has figured out a somewhat elegant solution.

gaius
You'd get bogged down in retransmit backoff before you knew it!
safetytrick
vnc all the time... if you hate yourself
andrewtj
Look in to NoMachine NX (http://nomachine.com/) rather than VNC. For some tasks I use a NoMachine NX desktop on a Linode Ubuntu server from the other side of the planet - it's quite useable. Over 3G web browsing is often faster via the remote desktop than browsing sites directly.
Harkins
You have to run your screen session from a stable host, not a laptop that gets disconnected all the time. Do you have a web host with ssh access? A buddy with a VPS somewhere? A spare $5 a month to spend on a shell account host to never have to deal with this again?

You could try using ssh-agent to reduce the pain of your connectivity issues, but screen really is the answer.

sounddust
I have plenty of access to shell accounts. But using a shell account to screen to another host just adds another layer of complexity, because when I lose my wi-fi connectivity for 1 second, then my SSH session to the shell account is lost instead of access to the real host. It still requires a waste of time logging into the shell account.
jerf
I think you may be constraining the solution down into impossibility. When you lose the network, you will lose the SSH sessions. Period. TCP will guarantee that. (Although not at "one second".) That's part of TCP's point.

The best you can hope for is either to set some stuff up as described so SSH automatically relogins to a screen session, or find a better network connection. Fundamentally, you can't patch around a bad network connection.

sounddust
Hmm, I think I just mis-stated my problem as being more difficult than it really is. I posted an example to the question for clarification.
apage43
To fix constantly re-logging in to ssh: - set up key authentication and run ssh user@host with your favorite parameters in a loop (shell script) so it reconnects when disconnected. - in fact, have it run screen when connected, so you get right back to where you were effortlessy.

As for port forwarding, you will still be disconnected from everything on connectivity blips when ssh has to reconnect if you use ssh port forwarding. Try a VPN setup instead. You might have to fiddle with the host to keep it from sending RST's/FIN's as soon as you drop (though it shouldn't unless a packet for you comes in during the blip, I think). When you come back, if you come back soon enough, the connections should be restored without having been disconnected. (AS LONG as you have the same IP address on the VPN, so use a static IP setup)

Also, if you run your SSH -through- the VPN connection and recover quickly enough SSH shouldn't disconnect at all either.

More Edit: For long connectivity lapses (changing locations) all your TCP connections -will- drop. This is not something you can work around.

sounddust
Thanks for your response. Just to clarify, I don't care if my forwarded TCP connections drop when my internet connection drops, just that they are re-established automatically. I also don't care about the actual TCP connection dropping, I just want all my SSH terminals to automatically pick up where I left off. Of course screen factors into this, but it has to be more than just screen.

Regarding the VPN, could this connection be automatically established upon joining a wireless network? (i think you see where i'm going with this question)

mightybyte
I don't think it's any more complex than running screen on the server, using SSH keys to allow you to connect to the server without a password, and possibly orchestrated with a small script that checks network connectivity and re-establishes your session after it's broken. Your 5 shells are running in one screen session on the server, so you don't have to restart all of them. It's just a simple "screen -D -R". If there is no existing screen session to attach to, you can have screen automatically create your sessions via the config file.

This is pretty much what I do already, except that I don't switch connections as frequently as you. But when I go to a new place, it's just connect with ssh and reattach with screen.

apage43
If you don't care about connection dropping, indeed just set up key authentication and run ssh in a loop as described above, instead of a complicated VPN, which will only really partially mitigate dropping of -forwarded- connections.

Secondly, you should be able to set scripts to trigger when a network interface comes up in most linux distros, but how to do it varies, for example /etc/network/ifup.d/ in ubuntu.

nuclear_eclipse
I do something of this sort. I have a shell alias called 'irc' that will connect to my remote server using the following local command, 'kepler' being an SSH connection alias defined in .ssh/config:

    ssh -t kepler bin/irc -D 1083 -L 3128:localhost:3128
This provides me with a single three-letter command to log in to the remote machine, set up appropriate port-forwarding, and automatically run a script, which itself contains:

    #!/bin/sh

    if [ -f /tmp/jreese.agent.pid -a -f /tmp/jreese.agent.sock ]; then
            echo "Socket and PID files found; loading..."

            export SSH_AGENT_PID=`cat /tmp/jreese.agent.pid`
            export SSH_AUTH_SOCK=`cat /tmp/jreese.agent.sock`
    else
            echo "Socket and PID files not found; starting SSH agent..."

            UMASK=`umask`
            umask 077

            eval `ssh-agent`;
            ssh-add $HOME/.ssh/id_rsa

            echo "Agent PID: $SSH_AGENT_PID"
            echo "Agent Socket: $SSH_AUTH_SOCK"

            echo $SSH_AGENT_PID > /tmp/jreese.agent.pid
            echo $SSH_AUTH_SOCK > /tmp/jreese.agent.sock

            umask $UMASK
    fi

    screen -Rd irssi -c ~/.screenrc.irssi -p 1
EDIT: do note that this script is not completely secure, because it relies on my knowledge that I am the only person with login access to my server. Otherwise, you should be adding some extra checks to the script to ensure that you are using files actually created by your own uid. I don't know the full potential of problem if someone created those files from a different UID knowing that you'd be using them. Anyways...

This script sets up a persistent, "secure" SSH key agent on the remote machine that can be reused for as many disconnects and reconnects as I feel like making until the server (if ever) needs a reboot. Then it starts or reconnects to a named screen session that I can continue using as if nothing ever happened.

Combine this method with a few different aliases as needed, and a local session key agent to only input your key's passphrase once, and then after a disconnect, press up-enter in each terminal as necessary, and boom, you're back in business.

--

On a related note, a drop of network connectivity should not be killing your SSH connection, especially for as short a time span as one second. SSH is designed to stay connected through at least 30 seconds of network loss; after the connection is regained, it should automatically reconnect to the SSH server and continue the session where it left off, even maintaining a buffer of unsent input during the connection loss.

I've personally suspended and then unsuspended my laptop within a one-or-two minute period without losing my SSH session, althought that's likely stretching things.

What's certain is that a 1-2 second lapse in your connection should most certainly not be dropping your SSH connection; you should definitely be investigating that to figure out what the problem is.

briansmith
Chcek your SSH configuration. By default SSH using some kind of ping system where, if the client loses contact with the server, the server closes the session. However, you can turn this off, or you can adjust the timing parameters to handle unreliable connections. I did that when I had a horrible connection (from the other side of the world) and it always worked fine.
Dobbs
Not quite what you are asking for but it would solve the problem:

1. Get an Android.

2. Hack it and install the debian subsystem.

3. see: http://brad.livejournal.com/2400054.html

So now instead of running an ssh to a remote server you are running an ssh to your android which then hosts all of the outward ssh sessions. Because of the androids 2g/3g you can't lose your connection (well not easily).

icey
Man, it's a tough way to solve his problem, but it would be hard to find a solution that is more hacker than this one.
eli
I truly feel sorry for you if your cell phone has a more stable internet connection than a desktop.
Dobbs
Cellphones: always connected to cell towers. Conneciton isn't great but its almost always there.

Laptop: network changes as you move from one wifi hotspot to another.

I'm advocating moving his ssh connections to route through the android rather than directly from the laptop to the server.

aolnerd
When you start a local shell, you can generate some session identifier (export DURABLE_SESSION=$RANDOM) and put it in your environment. Then have your ssh client send it to the server with "-o SendEnv DURABLE_SESSION".

One question is when does sshd know when to ditch interrupted sessions.

nuclear_eclipse
You can always manually interrupt an SSH (or Telnet) session using the following key sequence:

    <Enter><Tilde><Period>
This sequence is caught by the SSH client to force a session disconnect.
aolnerd
You are of course correct, but that's not related to what I'm getting at.

My question raised the point that the server would not be able to tell the difference, after an interruption, between session state that the client will hope to resume and session state that should be discarded. That state would accumulate until some routine cleared it out.

igorhvr
To solve your problem just use ssh -o TCPKeepAlive=no restOfYourCommandGoesHere

From the ssh_config manpage:

"TCPKeepAlive

Specifies whether the system should send TCP keepalive messages to the other side. If they are sent, death of the connection or crash of one of the machines will be properly noticed. However, this means that connections will die if the route is down temporarily, and some people find it annoying.

The default is ``yes'' (to send TCP keepalive messages), and the client will notice if the network goes down or the remote host dies. This is important in scripts, and many users want it too.

To disable TCP keepalive messages, the value should be set to ``no''."

a-priori
While that may improve the situation, it won't help if your IP address changes during the outage.
thorax
On Windows, I use Bitvise Tunnelier and the SSH tunnels/connections always keep retrying if there's a disconnect. For the terminals, I just use screen so I can get right back to what I was doing, but it is annoying to lose all those consoles if I had a bunch open.
randomtask
That depends. Can you mess around with the server? If so you could install HIP [1][2][3] on both client and server as a way of providing IP mobility. Basically the way it works is it inserts a layer between the routing and transport layers (called the shim layer), which handles mobility. The transport layer gets a locally significant identifier that uniquely identifies the connection on that host. This remains persistent even after the IP address changes. The shim layer then notices when you've got a new IP address and alerts the other host or notices when the other host has stopped responding for a while and waits for it to make contact again. You'll probably also need access to a DNS server that you can update on the fly, since HIP resolves HITs (HIP identifiers) to IP addresses via DNS. This isn't by any means a production solution, but if you're willing to try it out I'd love to hear how it works out for you :) Also the third reference explains how it works far better than I did.

[1] http://www.openhip.org/

[2] http://en.wikipedia.org/wiki/Host_Identity_Protocol

[3] http://infrahip.hiit.fi/

icey
As a very meta aside, this is the sort of Ask HN posting that really deserves to be on the front page. If you're reading this whole thread and you think it's interesting; please be sure to upvote the story so we can all set a good example.
sounddust
Thanks; I was actually concerned that my question might be inappropriate because it's a sort of "help me!"/tech support type question. I tried unsuccessfully to find a FAQ which explains what types of questions are appropriate, but couldn't find anything so I just posted anyway. I figured if it wasn't ok, it wouldn't get voted on; so no harm done.
sunkencity
I think it might be the change of ip adress that is killing your connectivity. I can close the lid of my macbook running os x or linux, let it fall to sleep, go down to the porch outside and let the machine wake up from sleep and still have the ssh connection active.

A thing you can do to get more stable ssh connections on your side is to run a nat service on the machine. For example, running connections on a OS X machine that has parallels installed (virtualization software) and that runs NAT is far more stable than a machine without.

schammy
I have this exact same problem. I don't forward traffic through SSH but I run a service that has almost 30 servers. Sometimes I have connections open to 10+ of them, sometimes (shudder) all of them. When my connection dies I want to throw my computer out the window.

I haven't read the comments here yet but hopefully someone will have at least a partial solution to save my sanity :)