88 comments

[ 2.5 ms ] story [ 161 ms ] thread
Usernames are not a secret, passwords are a secret.
I don't know, judging by my SSH logs it seems a lot of the automated malicious login attempts are looking for common software used to deploy code. Knowing that a target machine has a username used by some popular software might be valuable information in an attack.

But on the subject of passwords, best practice for SSH for a long time has been to disable password based login entirely and rely on keys.

Exactly - it also works for non-SSH accounts, thus allowing software enumeration by testing for default/common/known default service users.

For instance, an OpenBSD box running Tor may have a user "_tor", a Debian-based box (e.g. Ubuntu) may have a user "debian-tor", and so on (depending on how Tor was installed, in my case via pkg_add & apt-get; usernames might vary for different OS/repo versions). I have tested this using the PoC against some of my own systems (the ones that have PasswordAuthentication still enabled) and it works for those.

...and apparently the same issue exists in Dropbear up until current version (2018.76 / Feb 2018), which has an entirely different code base. A comment on /r/blackhat [0] led a colleague and me to look at Dropbear's sources, and it happens to have logic that is sufficiently similar [1] for the same PoC to work; tests against v2018.76 and a couple of earlier versions (e.g. v2013.58) are successful.

Shodan shows some 66k services identifying as SSH-2.0-dropbear [2], as opposed to some 15k identifying as SSH-2.0-OpenSSH [3].

Issue has been reported to the vendor today.

[0] https://www.reddit.com/r/blackhat/comments/97ywnm/openssh_us...

[1] https://github.com/mkj/dropbear/blob/master/svr-auth.c#L175-...

[2] https://www.shodan.io/search?query=SSH-2.0-dropbear

[3] https://www.shodan.io/search?query=SSH-2.0-OpenSSH

They are useful though. If you check a bunch of usernames from a person leak and find matches, you got a password to try.
A correct sshd_config includes:

    PasswordAuthentication no
+1. But just to be sure: that does not prevent testing for usernames and hence enumerating software by testing for known/common service account usernames (e.g. "_tor" on OpenBSD and "debian-tor" on Debian-based OSs). (No claim was made to the contrary; just mentioning this to prevent anyone from thinking otherwise.)
To disable logging in with password you also need:

  ChallengeResponseAuthentication no
I got a chill up my spine when I read this but fortunately it looks like this is the default on Ubuntu 16.04 and 18.04 (at least).
Apparently Debian disabled it in 2005:

  openssh (1:4.1p1-1) experimental; urgency=low
  
    […]
    * Disable ChallengeResponseAuthentication in new installations, returning
      to PasswordAuthentication by default, since it now supports PAM and
      apparently works better with a non-threaded sshd (closes: #247521).
    […]
  
   -- Colin Watson <cjwatson@debian.org>  Tue, 31 May 2005 01:33:33 +0100
https://bugs.debian.org/247521
You can also use https://tinyssh.org instead of the sshd from OpenSSH. It only implements key based authentication and newer encryption algorithms. It's available on Ubuntu:

    sudo apt install tinyssh
    wget https://github.com/yourusername.keys -O ~/.ssh/authorized_keys 
    sudo systemctl enable tinysshd.socket
but it's not widely used, which makes using it risky. To my knowledge there hasn't been a formal audit. Also it doesn't implement the scp protocol, but you can use rscync over ssh

    alias scp="rsync -e ssh --progress"
> Also it doesn't implement the scp protocol

From the FAQ[1]:

> TinySSH doesn’t have problem with scp protocol, only doesn’t have scp program.

I assume this means an OpenSSH `scp` command will work just fine when pointed at a tinyssh daemon, without the rsync alias.

[1]: https://tinyssh.org/faq.html

TIL TinySSH exists.

Immediate pro: Public domain / CC0

Immediate potential-con: Doesn't implement compression

Are you willing to send me a list of all your usernames on all your systems?
Sure. It's root and user, on everything :)
Does that mean that `wc -l /etc/passwd` is just two?
No. There will be plenty of system user. But they do not have SSH access permitted.
It doesn't matter if ssh access is permitted.
So how is an attacker going to access the machine?
If a certain account is present then the attacker might know a certain service is running and then target that.

Might also reveal the purpose of the machine (oh, this is probably the build server) or who is managing it.

Nothing crucial should be directly accessible from the Internet.
Indirectly then, or, the publicly facing servers are crucial enough.

It is information leakage, not the end of the world but it can potentially be helpful for an attacker.

there is an "ubuntu" user on most of my systems. I hope you find this information useful.
I know you're being facetious but, yes, I would find that information useful. Now I've just IDed the distribution I'm targeting.
A disappointing number of servers happily give away this information in their http response headers. For those that don't, `nmap -O` will sometimes do the trick.
nmap's OS detection won't normally tell you the distribution, only the OS (and maybe kernel version).
This is usually in the OpenSSH version string anyway, e.g. "OpenSSH 6.7p1 Debian 5+deb8u4 (protocol 2.0)". Guess what distro that server is running?
I believe only Debian does that, out of vanity (I presume).
https://salsa.debian.org/ssh-team/openssh/raw/master/debian/...

Subject: Include the Debian version in our identification

This makes it easier to audit networks for versions patched against security vulnerabilities. It has little detrimental effect, as attackers will generally just try attacks rather than bothering to scan for vulnerable-looking version strings. (However, see debian-banner.patch.)

List of all users allowed to use SSH on my systems:

- root

- user

- ansible_conf_daemon_user

End of List. It's all containers so IDGAF and I use pubkey authentication on each user.

For non trivial environments with audit requirements this is not an option.

Running all software under the same uid is also usually best avoided, you'll always have data at rest somewhere (which needs to be backed up, access controlled, and so on).

Of course there is other users but they aren't configured for SSH via AllowUser.

Additionally most of those are containers where running it all as the same UID isn't a problem.

User ids without login privileges can be just as interesting to enumerate.

Containerization isn't really relevant to protecting data, as you'd normally find data volumes and backups attached.

Well, it would but the exposed containers don't have any non-default users.

Plus, this is LXC, not Docker. There is no data volumes or backups attached. Backups are shipped to an append only server via SSH tunnel with low privileges, which also doesn't have nonstandard users setup.

Does anyone still use password authentication on servers that actually matter? I mean, I'm just a hobbyist, and I switched to keys several years ago. Basically, I just use root and user, because anything else unnecessarily adds information.
Yes. Large corporate networks often still have (some) production systems that allow password-based authentication. I don't know how widespread it still is, but I still encounter it frequently at clients (which may be a skewed sample).

Just to be sure: PasswordAuthentication does not need to be enabled for the PoC to work, and username testing can also be used for software enumeration by testing for common/default non-SSH users, .g. "_tor", "debian-tor", etc. (I apologize for repeating here what I also stated in other comments in this thread, but this aspect should not be overlooked.)

Can somebody please explain to me why exactly using good passwords and allowing root login is bad security? Every SSH security tutorial mentions those but they never mention the reasons.

If somebody hacked into my password manager and managed to steal the root user's password then they could do the same with my private key. Where is the difference?

Also, if somebody hacked into a non-root administrative account, they can just use sudo to elevate. What makes disabling root login more secure?

1. A "good" key safe for the forseeable future would have around 256 effective security bits. To get a key that strong, your password would have to be (speaking VERY generously here) 40 characters long and truly random.

2. When using real password authentication, connecting to a compromised server results in a disclosure of your password. When using challenge-response authentication, it does not. Note that correctly-configured OpenSSH uses challenge-response even for password-based methods. Incorrectly configured SSH will send passwords over the wire.

> Incorrectly configured SSH will send passwords over the wire.

What do you mean?

Real password based vs challenge response.

With real password when you type your password into the ssh password prompt it then sends that password to the server.

Challenge response is a method in which you usually encrypt or hash random server provided data. You send the hash. The server knows your password and can hash the data with it too. The hashes are compared not the password. In this method the password never leaves your machine. Replay attacks are prevented due to the server sending different data for you to hash on different login attempts.

Note also that requiring a password for sudo might have some issues, as the password is sent over the ssh connection as you type. I'm not sure what the state of the art is wrt counting and guessing characters - or if mossh makes the problem worse - or helps. For the general idea, see:

http://www.securiteam.com/securitynews/5KP0O0A3PU.html

Does anyone know of recent work on passively attacking interactive ssh in order to guess what's being typed? I could see machine learning helping here, given the ability to observe enough sessions to form a training set...

Of course, with Pam for sudo authentication you could use otp, u2f with or in place of a password.
Mistakes: if it’s a server used by a small number of security conscious admins, it’s moot. If it’s used by a lot of people, however, the odds go up dramatically that someone will have a weak password, one which is on a common list, or reused on a site which you’re just about to learn was breached.

Keys can be leaked but since it involves copying files and users aren’t penalized for using strong ones it’s much less common.

Just to add to @borealid's comment: a password that encodes 96+bits of entropy is effectively a shared symmetric key (OK, technically a secret).

Point being - it needs to be managed like a key: it's inconvenient to type, hard to remember, should be stored encrypted at rest, preferably in an hsm.

Gnupg agent/ssh agent and the various key agents for desktops all do this, and generally support hsm as well.

There's also a small difference in the complexity of enumerating the keyspace of rsa/ed2259 vs trying random passwords. With key stretching I'm not sure if there's implicit differences in rate of attempting login.

As for no direct root login via ssh - I think it depends a bit on the server. On a shared system login via regular user +sudo leaves an easier audit trail (who ran "sudo apt-get dist-upgrade" on Friday evening and took the weekend off without checking that production systems didn't crash... Was it "root" or "bob "?).

Then there's the nudge towards only doing what needs root as root (eg: curl/wget stuff as regular user, verify signatures, content - then run as root).

Is it more secure to locally dump and asymmetrically encrypt backups on disk as root - locally, and have a dedicated backup-user that gets the archives? Probably.

In general ssh keys (and certs more so) makes it feasible to split up access and authorization, revoke access etc without relying on something complex like ldap/kerberos for user management.

> Also, if somebody hacked into a non-root administrative account, they can just use sudo to elevate.

If they only have your ssh key, they'd typically need your password to elevate. Or you might require a shared root password - or set a password on wheel and require newgrp wheel before sudo. Not sure if the latter with disabled root account has much benefit, though.

One other nice thing about keys is that you can restrict them to a specific command. If you have users who are running well defined tasks that can be a great way to avoid handing out full accounts.
There are situations where using a password can be a little more convenient, for instance if you like to connect to the machine from random computers and you don't want to/can't carry your private key with you all the time. Obviously it's not the best security-wise, using a hardware token containing the key or even OTPs are probably a better solution.
I had a server that used password authentication for the root user (on the standard port). After a few months of uptime it was hacked.

I suppose even minimal extra security would've helped (no root user, only allow x login attempts in y time, changing the port for ssh), but I was still surprised someone got in so soon by simply guessing a 20+ string of gibberish (I still wonder if maybe something else happened).

All that said, I've had servers running for years without any successful hack, and in all those cases using ssh keys instead of password logins probably made all the difference.

The current version of OpenSSH doesn’t allow root login by default.
It was something else. Nothing or no one brute-forced a 20+ random string pass. More likely some service (Wordpress, phpbb etc) got owned. I’ve had multiple servers running for years with < 15 char non random root pass (stupid as it is to do so) with no problems.
I'd be inclined to think the same, if not for the fact that this server wasn't running anything particularly 'ownable' that I can think of.

The only alternative that I can think of is that one of the two other users' machines was compromised, but AFAIK I didn't give them root access. Plus, the logs show an extreme number of login attempts (tens of thousands of entries in auth.log) with an eventually successful root login.

Any other ideas of what might've been an alternative cause?

From the logs it sounds like maybe the password was in some lookup file or something. i.e. somehow that password was leaked and added to some list somehow.

Maybe you’re right though.

it's maddening that I can't tell for sure, but I'll agree that I find it still somewhat hard to believe that a 20-character random root password was brute-forced.

EDIT: to be clear, I am pretty sure was the only person who used the root password to log in, and the single other user used a similarly long and random password with a different username. if I'd been compromised my problems would be quite a bit bigger. All I know is that right before the 'fatal' root login, there are 40.000+ lines of login attempts from ip's that originated from shady places (at least based on entering them in various GeoIP sites).

Some servers I manage accept either password with two-factor code, or keys. Helpful when logging in from a new machine without having to carry authorized keys around somehow.
I use a YubiKey with a GPG key as my SSH key, means I just need to plug it in to authenticate to any of my remote hosts.
yeah but what if some user names are known to have a weak secret?
(comment deleted)
Incidents happens when several things which are innocent on their own suddenly interact in unforeseen ways.
As others have noted - this allows user enumeration. Which can be very useful:

Does a system appear to run one of the many horribly insecure "enterprise" backup solutions? Or some other horrible, ancient system?

Does $person from orgchart appear to have a $user? (target for spear phising, compromise of laptop).

Information hinting at distribution (eg: Ubuntu).

Is there a jboss user? Etc.

How much of a difference does this really make? Most attacks which I’ve seen simply launched the attack immediately so if you have enterprise software using the vendor’s configuration you’re screwed anyway, and both options are quickly blocked by fail2ban after splatting off the public key-only setup of any security conscious server.
Maybe knowing software a is likely running and managed by user b helps constructing a targeted attack.
Perhaps but it’s a limited change from what’s already possible. Spending time on basic best practice stuff securing those accounts will pay a lot more dividends in more scenarios than adding a minor impediment to finding them.
Fwiw I mostly agree - reading /etc/passwd shouldn't be able to help compromise your system. But it's also an error for sshd or any daemon to allow the enumeration of your list of users. I mean when we get to read /etc/passwd via httpd we call it an exploit... (even if we don't get shadow too).
Absolute nonsense. Usernames can be sensitive information in many situations.
If you're a startup and this matters to you, you're doing it very wrong.
Sorry, can you explain what you mean by this? I didn’t follow.
I think he’s saying:

a) Why are you using bleeding edge software in production, and especially so unreleased versions of OpenSSH?

b) Why are your SSH servers exposed to public traffic?

c) User enumeration is useful for finding accounts with weak passwords. Why do you have personal accounts on prod servers? Why do you have _any_ accounts not using public key authentication at all?

FWIW, regarding a),

> We believe that this issue warrants a CVE; it affects all operating systems, all OpenSSH versions (we went back as far as OpenSSH 2.3.0, released in November 2000), and is easier to exploit than previous OpenSSH username enumerations...

As for b) and c), I 100% agree.

In fact: if you're using KVM-based virtualization, and you have VNC or serial access to your node (GCP gives you serial access, via web UI or an SSH-based proxy), you could completely _disable_ standard network-based login. (Writing an admin tool that connects to the serial console, gets a usable shell, and enables your sshd for when you need it, is an exercise for the inspired sysadmin. :P)

But this fun little trick is going to break a lot of stuff.

Ah, my bad. I got the impression that it was introduced in the commit on the 31st of July.
Just remember (at least for non ephemeral systems): always have two ways in. That way if one path is down (fiber cable cut, serial disconnected, cloud admin panel down for maintenance, ssh authorized_keys file has wrong permissions, ssh/ssl cert(s) expired... ) - you can still get in via the other path.
A very good idea.

Perhaps a valuable resource would have a disposable proxy in front of it; then one way in (or even no ways in!) might be viable.

b) If not SSH what else would one use?

c) Not only production servers utilize internet.

b) use ssh, but control access with a firewall or vpn.

c) fair enough, although you should probably be using keypairs and a vpn for test and dev envs too.

I see that recommendation a lot, but I'd expect most VPN products in use to be _less_ secure than a standard OpenSSH. It's a comparably less complex protocol and codebase than IPsec, for example, and pretty well audited by now. It's not as if VPN products hasn't been open to this type of attacks before (I'm looking at you, Cisco).

  ChallengeResponseAuthentication no # \
  PasswordAuthentication no          # | nukes "Password: "
  UsePAM no                          # /
  PermitRootLogin no                 # (duh)
  PubkeyAuthentication yes           # enables public keys; very important
  AllowUsers user1 user2             # maybe? combine with an obscure username...
If you're doing admin on the machine you might not want to disallow root login.
I once set up a Slackware-based recovery partition on a USB HDD, and configured it so I could SSH to the machine, as root, with no password. (Took a bit of doing, too. OpenSSH is well built.) This HDD was only ever used for recovery.

Incidentally I only ever used it at home, on a LAN that only has devices I know about on it. In a work environment I'd only ever do the same running off an isolated lab switch.

But in certain situations, "login to the box as root w/ no pw" is awesome for rapid bringup/diagnosis/repair/whatever.

What do you mean? The email was sent from a a security research group.
To hazard a guess: it should be standard practice to use public key authentication for ssh (>= 2048 bits rsa or ecdsa), so this attack shouldn't make compromising a system any easier. Better yet, use a Yubikey or equivalent.

Other commenters have mentioned this attack getting information on hosted services on the system. But, usually you can get that from a straightforward nmap or other recon tool anyway.

Without suggesting anybody should use less than 2048-bit RSA in 2018, the security margin for even 1024-bit SSH RSA user authentication is pretty good.

In SSH first every connection is encrypted (before either side knows who the other is, encryption is used already, this is the same approach taken in TLS 1.3), then the server has to authenticate to the client, and then the client authenticates to the server on behalf of a specific user.

User public key authentication is a multi-phase protocol. This would be too awkward for something like HTTP where round trip times may be appalling, but SSH is rarely used at long distances.

In phase 1 the client proposes that it can authenticate for some public key Pn, it sends this proposal to the server, and the server can either say OK, go ahead, or reject this authentication without seeing it done, on the basis that it doesn't trust Pn anyway. It can keep trying different Pn until either it runs out or the server grows tired of this nonsense and disconnects it.

Once it has an OK, in phase 2 the client signs a nonce with the private key corresponding to Pn, which the server can now verify. If the signature checks out, the authentication was successful.

This means an adversary who knows your SSH username but has nothing else would need to literally _guess_ your hypothetical 1024-bit RSA key, not just (already hard but plausibly within the grasp of a resourceful adversary) break a 1024-bit RSA public key to find the associated private key, as your client only tells authenticated servers even the _public_ key.

Aside: One thing I'm surprised I haven't seen done with this part of the SSH design is user TOFU [Trust On First Use]. Lots of home or small / non-security business SSH setups do TOFU for server authentication, sometimes with a fig leaf ("Ask Jerry in IT to tell you the SSH keys" or "The SSH server keys are pasted into this Wiki page"). But I've never seen people do TOFU for the user authentication. In a similarly lax environment why not set up new users, and then when they first connnect just accept whichever public key they offer as "their" key, set that as trusted, and from then on insist on the same key for that user ? I feel like password auth is often left enabled because people are using it for an "initial" authentication step when first setting up without yet trusting any keys - and then never remove it. With "User TOFU" they'd have no incentive to enable password authentication at all.

low security impact, best practice as others has mentioned
This is typical of Qualys security advisories. They frequently find unintended software behavior which can be arguably called vulnerabilities but which can’t directly compromise software. It’s good they find these kinds of things, but I wish more people focused on the basic fact that their security advisories frequently only impact servers which aren’t adhering to best practices. It’s not good that OpenSSH allows user enumeration, but you’ve committed a critical error in system administration if that minor bug gives an attacker any leverage at all.

If you strictly use public key authentication and don’t expose your SSH server to the public internet, this is a non-issue. Furthermore there’s no defensible reason not to do that - it’s very straightforward to firewall SSH access behind a private network, and VPNs exist precisely for this reason. Access to the network and access to machines within the network should be strictly decomposed. If you allow SSH access from the public internet and have a VPN, you have to worry about two (privileged) points of failure, not one. Put everything behind a public-key authenticated VPN and SSH vulnerabilities are obviated, with the exception of insider threats in the organization.

Stop letting attackers route to you. Put your ssh server behind an onion. Do the same if you have POP/IMAP and other servers that have no reason for being accessible through transparent routing.