224 comments

[ 0.25 ms ] story [ 3992 ms ] thread
Workaround (yes, it's client-side):

  # echo -e "Host *\n\tUseRoaming no\n" >> /etc/ssh/ssh_config
Disclaimer: won't work on all operating systems, shells, etc. YMMV. Consult a doctor before following any advice you get from the Internet. Void where prohibited. Restrictions may apply.

Edited per comments below

Would this work too?

  $ echo "UseRoaming no" >> ~/.ssh/config
Yes, but it would not apply to other users.
Yes, but it'll only affect that user.
And only the last `Host` entry in that config, if present.
Presumably you would have to connect to a malicious host to be effected? Or perhaps a MITM on your connection to a legit host can exploit you somehow.
> Presumably you would have to connect to a malicious host to be effected?

Malicious or compromised.

> roaming code in the ssh client could be tricked by a hostile sshd server, potentially leaking key material.

From the updated OP:

> The authentication of the server host key prevents exploitation by a man-in-the-middle, so this information leak is restricted to connections to malicious or compromised servers.

Malicious, compromised, or new servers. Because really, how many people check the host key for a newly spun up EC2 instance?
Don't ever fix your `ssh_config` by appending stuff to the end of the file. The configuration syntax allows for block constructs without an explicit end marker (like `Match` and `Host`). Appending will cause all kinds of sadness.
It's from the linked page.
Indeed. The page gives bad advise. If your config has `Host` blocks, like often in people's personal configs, or if you have a `Match` block, the new directive only applies to the last of those blocks in the config file.
Sufficient to place it at the top of ~/.ssh/config? Or does it also need a block header of its own, like:

    Host *
    UseRoaming no
I confirmed that it does not need to be placed into a Host block, as long as it is above all of your blocks.

For instance:

   UseRoaming no

   Host *
      Blah yes
Test with: ssh -v remote.ssh.host.com uptime 2>&1 | grep -i roaming

If it returns nothing, the config fix is active. If it isn't active, you'll see 'debug1: Roaming not allowed by server'

Which is why they're appending a newline and Host *?
Note that the example was changed as a result of my initial comment.
I'm emailing the one-liner to folks and found it helpful to just include the root subshell:

    sudo bash -c 'echo -e "Host *\n\tUseRoaming no\n" >> /etc/ssh/ssh_config'
The most terrible thing here is that OpenBSD usually does some full disclosure. This is sure going to be a nasty bug.
Why is this terrible? In any case, the software is open source so any security fix commit can also be considered as full disclosure.
But they didn't disclose the bug yet. The implication is that this bug is /so bad/ that it justified breaking the normal "just disclose it" approach.
(comment deleted)
As far as I know, this will affect any OSX, am I right ?
(comment deleted)
This does not affect the sshd.
According to Theo, yes.
Is there any way to update openssh on my side, without waiting to apple ? Is this enough ?https://mochtu.de/2015/01/07/updating-openssh-on-mac-os-x-10...
You can patch it with:

  # echo 'UseRoaming no' >> /etc/ssh/ssh_config
or

  $ echo "UseRoaming no" >> ~/.ssh/config
The second one. There is no /etc/ssh on a stock Mac. (Although there is a /private/etc/ssh_config on mine, without the intervening ssh dir)
Hue? There is one on mine. Maybe mine's different because I upgraded from 10.9 to 10.10 to 10.11 whereas you might have started with a later version (or an earlier one)?
You have an /etc/ssh, or just an /etc? I have /etc, which is a symlink to /private/etc, but as I said it has no ssh dir.

FWIW I am on 10.10.5 not 10.11. Maybe there's a change in 10.11. (The article the grandparent links is focused on 10.10 so I assumed that was in the scope of this discussion.)

PS are you on homebrew? maybe homebrew adds it.

Homebrew should not add anything outside of /usr/local iirc.
As mentioned in another thread, don't do this blindly. If there are `Host` or`Match` blocks in the config, this new line will only apply to the last of those blocks.
Thanks DrRobinson ! I added it to my .ssh/config
Yes, that should be enough, be wary of $PATH issues tho.
Even easier, just place this text:

    UseRoaming no
on its own line in your ~/.ssh/config file. If you don't have such a file, create it and put this line into it.

Doing that will only protect you in that OS X user account, but I bet you only ever use one account on your Mac to SSH anyway.

I updated /private/etc/ssh_config on OS X (system-wide vs. user-specific).
I believe that is the config for the daemon, not the client. I think you want ssh_config.
Yeah, typo'd it in my comment above. Thanks, fixed.
It will affect any OSX that is new enough to have that code in it. Same is true with other OS. For instance, on an older, patched FreeBSD 7.2 system, we see this result:

  /root/.ssh/config: line 1: Bad configuration option: UseRoaming
  /root/.ssh/config: terminating, 1 bad configuration option
... which means that sshd predates the roaming code. I haven't tested, but I'll bet my snow leopard workstation also predates that code.

So ... if that line:

  UseRoaming no
produces no errors when you ssh as that user, then you had the problem and you fixed it. If it produces the error above, you never had the problem in the first place (although with an older sshd like that, you should make sure you're not exposed to other, older vulnerabilities).
Ok, an update - snow leopard is indeed too old for this to be relevant. SL with the last and latest software updates still doesn't have the roaming code in sshd and therefore you don't need to fix anything (at least wrt roaming...)
Someone tweeted this, the roaming code doesn't seem to be very well managed: https://twitter.com/marver/status/687644904575627264

Edit 1: Here's some relevant commits too https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2

Edit 2: This mailing list post seems to discuss the vulnerable feature http://www.gossamer-threads.com/lists/openssh/dev/49018?do=p...

Edit 3: Got a better description of actual impact of the bugs:

>Experimental roaming code in the ssh client could be tricked by a hostile sshd

>server, potentially leaking key material. CVE-2016-077 and CVE-0216-078.

>Prevent this problem immediately by adding the line "UseRoaming no" to

>/etc/ssh/ssh_config.

> >server, potentially leaking key material. CVE-2016-077 and CVE-0216-078.

Presumably you meant CVE-2016-0777 and CVE-2016-0778

I'm just quoting OFFICIAL SOURCES here.
So the real question is, can a MITM intercept connections to boxen you frequent to exploit this? Or is it limited to connecting to hostile honeypots?
Tested quickly, doesn't look like it. Host key checking happens before the roaming stuff.

That doesn't mean it's limited to hostile honeypots, if it turns out that this is easily exploitable it'll be a complete shitstorm with sysadmins getting owned left and right with rootkits implementing automatic exploitation.

(comment deleted)
Does this affect mosh?
No, the roaming in Mosh is unrelated to OpenSSH's roaming feature.
Irrelevant; it still affects mosh as mosh does an ssh connection first to do auth and key exchange. That ssh session is potentially vulnerable when connecting to a malicious or compromised host.
Doesn't mosh initially establish a connection via SSH, though?
But Mosh uses OpenSSH for authentication. And if roaming is enabled by default on the client this could still be an issue, no? I'd say it's still a good idea to add "UseRoaming no" to ssh_config.
But authentication in Mosh still relies on OpenSSH, so auth could be intercepted by an attacker after which Mosh is completely open to them:

https://mosh.mit.edu/

> However, in typical usage, Mosh relies on SSH to exchange keys at the beginning of a session, so Mosh will inherit the weaknesses of SSH—at least insofar as they affect the brief SSH session that is used to set up a long-running Mosh session.

Right, but mosh uses ssh for initial key setup. I suspect it is possible to exploit that. (As an occasional Windows user who's using the Mosh Chrome extension, I have no idea how much risk I'm at; can't edit ssh_config, it's not even exposed)
Yes; mosh depends on ssh to set up the connection (auth and initial key exchange).
Does it strike anyone else as bizarre / poor form for an experimental feature to be enabled by default in OpenSSH, which is normally very conservative with option defaults?
What's more, it was not documented in the ssh_config(5) man page. If it had been, I probably would have disabled it long ago when hardening my SSH config.

Time to head to the source to look for other undocumented options...

Update: my findings are here (scroll to bottom for the upshot): https://gist.github.com/AGWA/e92d4f5343be1f7a941d

UseRoaming is the only one to be concerned about. There are many other undocumented options, but they're all aliases for a documented option or are deprecated/unsupported.

Please report back what you find!
Actually, I quickly generated a list of undocumented config options: http://sprunge.us/QDSE

Edit: Fixed version http://sprunge.us/LVYB

I believe "kerberosauthentication" and "rhostsauthentication" are documented as well. I'd check the "official" OpenBSD man page instead of whatever you have on your system. Regardless, thanks.
Neither of those are mentioned on the latest manpage from git.
It just sounds like a big code-base fuckup. They more or less admit it: "Server side was disabled/gutted for years already, but this aspect was surprisingly forgotten."

Sounds like this was put in at one time, forgotten about, and the code lingered for a long time until someone pointed it out. SSH as a protocol is pretty crazy. Everyone loves it, but its a lot of things in one, which ironically goes against the unix philosophy. Its a remote terminal, a file transfer server, a network tunnel server, a socks server, etc. There's a lot of stuff in there and I imagine difficult to work with sometimes.

No word if this is enabled in Putty, but I imagine it is if its using openssh libraries.

Pretty sure this is OpenSSH only. PuTTY and SecureCRT, at least, are reported not to be affected.
We use WinSCP also, a file copying utility; it would be useful to know whether that's affected if anybody has that info.
Given that this exploit happens during capability negotiation (or whatever SSH calls that part of the protocol), it also cannot be mitigated via pledge[2], which is where OpenBSD has been focusing a lot of attention. This is an unusual stumble for the OpenBSD team. Client-side privsep support, if such a thing existed, might mitigate attacks like this. As it stands protecting against exploits of this type wasn't even on the hardening roadmap.

http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/...

PuTTY is completely different code.
SCP/SFTP is actually a separate binary.

PuTTY is definitely not affected since it has its own SSH implementation.

Server-side you can see the "full config" sshd uses:

  $ /usr/sbin/sshd -T -f /etc/ssh/sshd_config
Modify, if necessary, to point to the location of your `sshd` and configuration file.
I came here to ask exactly this, and also why this feature isn't documented. Sounds like a nice little exploit.
What was this experimental / undocumented roaming feature even suppose to do?
Complete guess: handle changes of ip address on the client side.
Allow a mobile user to reconnect to a disconnected session without (user) re-authentication.
ETA: I was way off... you can ignore this... :-)

cf. section 3.3.5 [0], which describes "Roaming (Suspend/Resume)". This is documentation for an application by a company called AppGate (later acquired by Cryptzone) that wrote {some|most} of the code in OpenSSH's "roaming_client.c".

This gives a hint of what the ramifications may be: basically, a MITM, who observed the initial session negotiation, can disconnect the client and hijack an active session.

> Roaming is a feature which allows clients to suspend the connection to the AppGate server and later to resume it again. The user does not need to re-authenticate when reconnecting. Indeed, the entire process can be completely automated and nearly invisible to the user. All established connections will remain alive while roaming. This feature is intended for mobile users who move around between networks.

> Technically, roaming is accomplished by closing the TCP tunnel when the connection is suspended. When resuming, a new TCP connection is made to the server and the SSH data stream is continued through this new connection. The user does not need to authenticate again, instead the client authenticates to the server, without user interaction, with the help of a random password which was made up when the user authenticated at the start of the session. In addition to knowing this password, the client must also know the encryption keys and encryption state to be able to reconnect. It is therefore impossible for a third party to break in and take over a suspended session.

I particularly like that last sentence.

[0]: http://download.cryptzone.com/files/download/AppGate-10.2.3/...

That's not correct though, MITM cannot exploit this. A malicious SSH server can exploit this during a connection though, but only after host key verification.
Oh Cryptzone, funny they offered me a job a few years back. Interesting that this was left in OpenSSH like this, even more as a default.
theo must be mad.

Shaming people for leaving useless non essential feature in their code that results in security breach.

And now the jewel of his crown has been compromised.

The funniest part is now that his jewel has been tarnished, maybe people will understand what he was saying.

And maybe too, people that believed privacy can be achieved on the internet will finally look at the problem of believing the 2 general paradox can be solved without at least 2 different constant link on different plans. And the problem is belief is a poor substitute for thinking - critical thinking.

And maybe people will discover the sad truth of the internet.

Security requires a perfect world, where human beings neither makes mistakes nor are corruptible.

Errare humanum est, perseverare diabolicum

Oh! Some says that is what 2 factor authentication is.

I will answer, my intuition is telling me that 2 factor is good for a fixed amount of time/information and that using it correctly would annoy people to the utmost points.

Then people would say well let's accept that fraud exists. Business first. (costs/benefits)

Then I say giving 3% of all e-commerce to the bad guys is like admitting organized crime have a strong budget for even more crime ... and that we are fucked.

Unless you don't understand that ISIS is basically a startup. A startup that overthrow a state to make even more money and industrialize crime.

What Theo (and other OpenBSD developers) have been saying all these years is that it's impossible to /not/ make mistakes, which is why sane design and exploit mitigations are important.

Mad ? Maybe, I wouldn't know. On the other hand, I bet he's really glad all that effort to have ASLR by default was made, because it makes it more difficult for an attacker to exploit vulnerabilites such as this one.

ASLR makes exploits more difficult as long as you have true randomness. It is just a mitigation ... for another problem.

In this case the elephant in the room is stack injection and dynamic libraries. If processes where confined to a well known address space that was self contained ASLR would be useless.. But dependency management would be hellish.

Back to the case. OpenSSH has been openly criticize by 9plan teams for exactly the same reason openSSL has been criticized by openSSH team : too much complexity (not to say plan9 came to anything usable and were right (2 wrongs do not make a right)).

http://harmful.cat-v.org/software/ssh

I do have a feeling as a physicist that the S in CS stands for Shortness of thinking. What is obscure is not profound.

And, at the opposite of engineers I don't believe that security is achievable at all on computers. It is like believing there exists a way to avoid triangulation with one strong radio source.

Computers leak to much information (especially in the physical world), and C is a map that is taken too much for the territory. Every bugs are exploiting a wrong mapping of concepts to implementation.

Modern security is like string theory, admired by everyone because it requires great technical knowledge to master, understood by none because it is way to complex for our "human" brains. We live in an era of belief in solutions.

A bit of critical thinking and of distrusts of experts and stuff that you cannot understand without devoting your life to a subject is at my opinion a must.

I distrust mathematicians for their capacity of dealing with the real world and its uncertainties, and cryptography (as much as functional programming, big data, algorithmic, IA, machine learning) is math driven in a pure Aristotelian world. Where perfection and harmony is the pillar of thinking.

I come from micro-electronics, I only see wires (that are antennas), oscillators, multiplexers, gates and basically a dumb automat I can automatize in respect to time of propagation of signal, and I know that modern computers under the hood are in the physical realm of relativity with approximate answers.

I theorize, build, measure, and retheorize ad nauseum until the product is measured to work the way I expect it to in the domain of validity with a good enough confidence and margins of errors are always on my mind to be controled.

Security requires a zero margin of ambiguity. Physical world is bound to heisenberg equation and coupling. Purity does not exist and this cannot be mitigated. The postulate of cryptography are wrong from the core. Real world always win at the end.

Aristotle way of thinking must die. Math is not science.

What in the hell are you on about?
The base of science : critical thinking as opposed to expertise and "commonly accepted wisdom".

Trust, but check.

And if cannot check, I cannot trust.

Their map (model) is very nice, very detailed and self consistent. But is is not the territory (implementation) and the more complexity we stack the greater we prove the map diverge from the territory. And also the less it can be audited.

Don't expect normal people to trust what they cannot check. It is faith security experts are expecting from users, not trust.

I do my part of the contract as stated by common accepted risk management "best practice" regarding computer security.

I do not trust blindly.

You seem to be experiencing a thought disorder called derailment [1]. This can be indicative of serious mental illness, have you ever talked to a doctor about that?

Do you ever feel like ideas or sentences that you are talking about need to be talked about urgently or are you ever annoyed by others telling you to calm down or talk slower?

[1]: https://en.wikipedia.org/wiki/Derailment_(thought_disorder)

(comment deleted)
If it's not obvious, don't just add that config option, you have to also restart it.

Actually wait, is this only affecting the ssh client and not the server/daemon ?

Yes, it's client-side only. Server-side, it's disabled (cf. readconf.c).
Since this is a client side issue, can this be used to exploit those automated scanners who try to break into your SSH machine?
(comment deleted)
No. The scanners are looking for password-accessible accounts, not keyed accounts. The scanners won't have useful keys, nor listening ssh daemons.
Authenticated scanners that use key auth like Qualys' security appliances could have private keys that are valid across the organization, and if using an affected client version, could leak this information to a malicious system on your network.
Check me on this, so this is a client-side problem only, so ssh-ing into only know servers shouldn't be an issue and clients cannot cause problems for servers?
If an enemy takes control of just one of the hosts you ssh into, he will get your private key and can use it to ssh into any other box where you use RSAAuthentication.
I already added the mitigation, but I wondering if my servers can be patched on the weekend.
There is nothing to patch on the server side. You need to ensure the ssh client is updated on the machines you are sshing from.
ok thanks, that's what I figured, but it I was getting a bit worried that I was missing something.
Use the workaround immediately and patch ASAP, both on your workstation and across your infrastructure where SSH is used, which can be in surprising places.

SSH is designed so that, even if you connect to an evil host, the host only learns your public key, not your private key. This leaks your private key to an evil host. You might think "I only SSH into boxes that I own, so I'm good, right?", but if you in the future lose a box to the enemy, they can then use that box to grab your private key, and then use that key to get into every other box you can access with that keypair.

This turns your sysadmin's photo blog hosted on Digital Ocean into a potential vector into your entire infrastructure, since there is a high likelihood they use the same private key for both.

If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

You should expect adversaries to add an exploit for this to their rootkits. That would be bad enough since attackers targeting your infrastructure may exist, but this is also plausibly exploitable in a spray-and-pray fashion. Root any server on the Internet, using e.g. a WordPress vulnerability. Install key-stealing malicious SSH server. Forward all keys to your C&C infrastructure. Your infra does a reverse lookup from public key to Github account (trivial -- try "ssh whoami.filippo.io" if you don't believe me) and immediately optimistically tries to use the stolen key to log into the web tier of any site listed in their email/bio/Twitter profile/etc, pinging the attacker when it succeeds.

This may be a start:

  Host *
  IdentityFile ~/.ssh/%h
Comment: One key per host, named after the host you connect to
(comment deleted)
You also need to set

    IdentitiesOnly yes
if I remember the config setting correct. Note however that this only limits the offered keys during the authentication phase. If you use AgentForwarding, this still has the entire keyring available afterwards.
which is not very helpful if you for example want to run git via ssh or similar commands on the target server.

There's places where there are two alternatives: agent forwarding or copy the key. Using ssh agent forwarding allows me to use a smart card for the key. It still has it's problems but it's much better than having the key on the remote machine - for example the use of a smartcard mitigates this vulnerability since the key never enters the process memory.

Not sure which exact scenario do you have in mind with ssh+git. ProxyCommand method works just fine for me with my private jumphost in the middle and github on the far end.

Note that for ProxyCommand to work, you don't need a full shell on a jumphost, just "AllowTcpForwarding yes" is enough. On the other hand, with AgentForwarding method you do need a full shell on a jumphost.

I have a vm that I use as development environment, my laptop is just a dumb terminal. I need to check out code there. It's not a jump host.
If you have clean network path from your terminal to the dev vm, why need for either ProxyCommand or ForwardAgent - just ssh to the vm directly, no?

Of course local, not forwarded ssh-agent on the terminal would be super-handy to avoid typing pass-phrase time and again; but that's different and independent from ForwardAgent.

I do ssh directly to the VM. It sits behind a VPN connection at aws. I need to make ssh connections from there to github. The key resides on a smartcard in my laptops usb slot. And that's where ssh-agent/ForwardAgent comes into play. I forward my local key to the remote VM.
Fair enough. Makes perfect sense, thanks!
Does anyone know how this interacts with ProxyCommand? If I fix this in ~/.ssh/config on my laptop, and I ssh to hostB through hostA using

  Host hostB
    ...
    ProxyCommand ssh hostA -W %h:%p
in the config on my laptop, do I also need to fix it on hostA?
CVE-2016-0777 doesn't interact with ProxyCommand in any special way. That said, after connecting to hostB with your example config above you will have 2 ssh sessions:

1) from your client to hostA

2) from your client to hostB

So to answer your question - no, vulnerable client on hostA is not a problem (or at least not in this particular use-case).

> Vulnerable client on hostA is not a problem.

Ok, that's what I was wondering. Thanks.

It is not my understanding that SSH will attempt to use only the file identified by IdentityFile, and yes, that is surprising now isn't it. If the server does not cooperate with you, your proposed configuration will give the server all of the keys.

Edit to add: While I was testing this understanding (which is correct) mioelnir's comment added the setting you need to get the behavior which I thought was automatic.

I have this:

    # IdentityFile magic, should be placed at very end of file
    Host *
    IdentityFile ~/.ssh/keys/id_ecdsa_%r@%h
    IdentityFile ~/.ssh/keys/id_rsa_%r@%h
    IdentityFile ~/.ssh/keys/id_ecdsa_ANY@%h
    IdentityFile ~/.ssh/keys/id_rsa_ANY@%h
    IdentityFile ~/.ssh/keys/id_ecdsa_%r@ANY
    IdentityFile ~/.ssh/keys/id_rsa_%r@ANY
    IdentityFile ~/.ssh/keys/id_ecdsa_ANY@ANY
    IdentityFile ~/.ssh/keys/id_rsa_ANY@ANY
What if my infrastructure is over 50 hosts? How am I going to distribute key files for 50 people with 50 keys every 14 days?
You could use something like ansible for the management of keys. Or you could centralize your authentication, which would probably be a lot more sane.
Even with central management, I don't know that I want to have 2500 keys floating around even if I have a management stack in place. That seems like an attack vector all in its own. Changing a key every other day on a 90 day rotation with 50 boxes. And fifty boxes isn't even that much. That's like 2 racks.

Even with config management this won't scale past about 2 or 3 people and 10-20 boxes.

Central auth is an option I guess but I think the better way to go would be a 2 factor with the key and hotp.

It's not that bad, it's part of the user data and should be provisioned the same way.

OpenSSH can also be used in a PKI fashion, where you use certificates instead of known_hosts and authorized_keys records. It's quite all right, but it comes with the same problems a full PKI does as you need to keep track of when the certificates expire. You also need a way to distribute CRLs so you still need configuration management.

The suggestion was not PKI though. I'd be happy with that. I already have a PKI in place. The suggestion was for a 1 key per box.
You automate it. At 50 hosts some automation like puppet or ansible, etc is worth the trouble, especially if it helps you make sure that all hosts have the correct keys on them - that's just basic security - make sure there are no keys that shouldn't be there.

What do you do now if someone leaves? Remove that person's key from all 50 hosts one at a time?

Or, at the very least, you use tmux with sync panes or csshx - and log into all 50 hosts at once, and then you can issue one rm / scp command if you are still doing things manually.

Well I have an automated credentials and authentication infrastructure in place. The point is that if I have 50 hosts (and my infrastructure has considerably more) and 3 employees with 50 keys each, there will be 2 key changes a day on a 90 day rotation.

Are my guys going to spend 5 minutes every morning making and pushing keys?

What if my hosts auto provision themselves and there are 5 new hosts every morning?

Am I going to make keys as part of infrastructure deployments and push them back to the workstations and update other peoples ssh configs?

I'm just saying if you've worked at scale, you'll realize that a key per box won't scale. I mean, anything can scale if you put enough effort into it. But the chances of disaster in lack of access or security breach, from over complication is way to high here.

Automating bad processes just makes it easier for them to fuck you.

I really hope that sysadmins/people are not using the same private key for work and pleasure. And for that matter, the same key at work for DEV/TEST/PROD. But you're probably right that a lot of them are.
As a sysadmin I am thankful I realized this was a mistake a long time ago, but you are right that I see it happen way too often for comfort.
Depends how you look at it. I use my key pair as my digital identification, whether at work or at home. I only have the one set. That logic held up until now I suppose, with the leaking of private keys.
For the at-work keys, I differentiate between work/workstation and work/laptop. Full disk encryption and the bcrypt based ssh private key format hopefully buy me enough time to revoke the laptop keys and have a KRL update pushed to all servers if it gets stolen.
It does depend. This may be leftover behavior from my contract days, but I've always had at least one key per entity with which I interact[1].

The reasoning is that it is far easier for me (or the entity, but that's a bit different) to delete one keyfile to sever access than it is for me to rekey everything else. I don't want access to things I'm not actively engaged with - compromises happen, even to engineers' laptops, and that conversation with former employers is too much like calling up your exes to tell them about a VD test result for my comfort.

As far as it being digital identification, lots of companies have IDs separate from your DL/passport. This is usually because the company ID provides access to something your other ID doesn't. Same principle.

[1] There are lots reasons to have lots of different keys, and only having one per entity is pretty rare for me.

If you want to stop this, require a specific, non-standard, RSA key size for each access group. They'll have to generate new keys. For ECC, maybe ssh certificates will work?
> If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

A better mandate would be to use ssh-agent. Using separate key pairs without an agent doesn't prevent key theft due to memory disclosure vulnerabilities like this one; it merely compartmentalizes the damage. But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities. Stealing a private key would require filesystem access or remote code execution, at which point all of your private keys are at risk, so you might as well just use a single private key per device.

For the best security, store your private key on a smartcard, so even an attacker with filesystem access or remote code execution can't steal it.

> But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities.

This is what I would assume, but has it been confirmed that this vulnerability doesn't affect identities provided by an SSH agent? If any of this "roaming" support was provided by the agent, it could be the case that the leakage can be triggered via SSH agent protocol requests. I doubt this is the case but I don't see anything confirming or denying it.

It would be nice to hear some official word on whether agent-based authentication is vulnerable, because if it isn't, the seriousness to me is greatly reduced, as I never load identities in the client directly.

The main issue identified by Qualys is a more generic memory disclosure bug:

https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-077...

This means they are basically able to dump the memory of just the running `ssh` process -- eerily similar to Heartbleed.

This means private keys stored by the `ssh-agent` process, outside of the `ssh` process connecting to an Evil Server(TM) are not affected.

This is because the protocol used between SSH Agent and SSH client does not transfer the entire private key, rather the SSH client asks the Agent to do a signing operation on it's behalf.

Good to know. Note regarding your final sentence, however: this design does not necessarily prevent the SSH agent from having a bug which can cause key disclosure via the agent protocol. It returns data buffers, so if it could be tricked into leaking key material into those data buffers, it may end up getting sent back to the server just the same (e.g. as a signature value). This would, however, be a bug in the agent and not the client.
The Qualsys analysis states that keys provided by an agent are not disclosed, and that encrypted keys are disclosed in their encrypted form.

https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-077...

Quoting from above: Finally, for these three reasons, passphrase-encrypted SSH keys are leaked in their encrypted form, but an attacker may attempt to crack the passphrase offline. On the other hand, SSH keys that are available only through an authentication agent are never leaked, in any form.

So if you use an agent, and follow the good advice to encrypt your private keys you should be safe(er).

See for older client versions:

http://martin.kleppmann.com/2013/05/24/improving-security-of...

or better for newer clients:

http://www.tedunangst.com/flak/post/new-openssh-key-format-a...

If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

As patio11 notes, this would not at all be excessive as a response to the threat model, but even with an aggressive key rotation mandate, this still doesn't make your production infrastructure as secure as your corporate Confluence wiki with SSO. Your employee's private key is still a static credential, and any rotation policy (14 days? 30 days? 90 days?) will leave a significant window for an attacker to use a stolen credential. Additionally, using a single key per employee for all infrastructure access magnifies the attack surface of a stolen credential to everything you operate.

Full disclosure, I'm a co-founder at ScaleFT, a startup focused on solving these sorts of problems. We're releasing a patch to fully mitigate this for our users this morning.

Ironically, ssh'ing into a random server is the exact thing you shouldn't do if you aren't patched.

Make sure you patch BEFORE doing something like ssh'ing into whoami.filippo.io "just because someone on the internet told me to"

This site is (from what I can tell) not malicicous, but copy pasting scripts and blindly following advice from the internet is a prime example of how an exploit would occur.

We went to other way and kissed SSH goodbye a long time ago in favor of immutable infrastructure and automation: https://boxfuse.com/blog/no-ssh
you've never had to debug anything in production, have you?
It's not 2006 anymore. For the vast majority of the cases there is no need for SSH when it comes to logging, profiling and debugging.
So how does the exploit actually work? How can a malicious sshd actually use this to acquire the private key?
Yet another direct-link post to a cryptic CVE for those of us who aren't in the know on security issues. Upvoting is a great shibboleth to show that some posters "get it", while the rest of us are left in the dark on severity, left to wade through the comments section to try to get a sense of what the real impact is.

From what I can tell, if I accidentally SSH to the wrong server (or a compromised one), my private key can be obtained. I have no clue if that's actually the correct interpretation.

Oh, come on. The issue is very clear. Your interpretation is correct.
How does a client bug end up leaking private key material to the server if the private key is in an entirely different process (the agent)? Or is this only an issue if you are loading your identity directly in the client?
I would love to hear from someone who knows the answer to this. Does this affect ssh-agent? I am rotating my keys regardless, but I am curious.
According to: https://news.ycombinator.com/item?id=10902833 it does not:

> Finally, for these three reasons, passphrase-encrypted SSH keys are leaked in their encrypted form, but an attacker may attempt to crack the passphrase offline. On the other hand, SSH keys that are available only through an authentication agent are never leaked, in any form.

Testing the ssh client config workaround:

   ssh -v user@localhost 2>&1 >/dev/null | grep -i 'roaming'
returns "debug1: Roaming not allowed by server" when vulnerable, and nothing when not. YMMV, only tested on a few machines, etc.
does this require a ssh server running on localhost?
Yes, that command is connecting to an ssh server on localhost, but you could connect to any ssh server that you trust...

  ssh -v -T git@github.com 2>&1 | grep -i 'roaming'
Note: this actually means that everyone should regenerate all their key-pairs after updating.
This is a surprisingly bad fail. I've never seen or heard of this option before. In fact, my vimlint for ssh_config doesn't even show it as a valid option!
For shells where echo is a builtin that does not know about -e:

printf 'Host *\nUseRoaming no\n' >> /etc/ssh/ssh_config