If anyone else has this issue please install fail2ban. It basically does a 'tail -f' on your logs and blocks addresses based on your rules. For example, if there are 6 failed login attempts in a 24/hour window, block them.
Be careful, though, I had it installed and blocked myself for a few minutes when mistyping my password. I've switched to SSH keys completely since, but be careful not to set too low a limit.
100% agree. We run some high traffic ssh bastion hosts with fail2ban. We use fail2ban, ip block white lists, and good syslog monitoring for easy troubleshooting.
Being the enlightened hacker that I am, I use SSH keys for all of my ssh'ing, and I know a fair number of sysadmins that require ssh keys, so I kind of thought it was more or less default nowadays. I guess I was wrong, so what is the reasoning behind allowing password access to your servers? I guess if you have a fair number of non-technical users, keys might be too hard to deal with? But scripts can make the process fairly painless. I'd be interested to hear from someone who does manage more than a couple machines.
You have to trust the PC at the internet cafe in Elbonia. That's the machine that runs the ssh client, and that's the machine you physically type your one-time-password into... or disclose your private key to.
Before fail2ban was available my friend and I had our own little script which to this day still works perfectly. If we get 3 failed login attempts with the wrong username we ban the IP address, we allow up to 10 failed login attempts for a username that exists.
I've never locked myself out, nor have any of my users. At that point it doesn't matter how fast they are coming either, for the uptime of the script it keeps track of the IP addresses.
One thing we did add was a negative counter, so that if you logged in successfully you basically got a -1 added for your IP address, so if you were to log in successfully 100 times, now you have 110 times to guess your password before being locked out, which honestly hasn't been an issue yet.
I do find it funny watching the log from my script and seeing how many hosts we have blocked. Last time when the uptime on my box was almost 2 years, I had blocked almost 5000 IP addresses. Do note that I block them completely from SSH, and severely rate limit them to any and all other ports.
> Doesn't that provide information about which usernames are valid, though? Or does that turn out not to matter?
Yes it does. A targeted attack would be able to get information out of that. A giant botnet spamming authentication attempts? They apparently don't even notice if their connections are being dropped, never mind the difference between valid and invalid usernames.
Sure, it provides information, but if you find a username you have 7 more tries left before you get kicked out... so the chance of you actually getting a password that works is tiny.
Some users have their home IP's never change, BUT they will forget their password after a month of not having used it. So now they will get their negative count, AND 10 tries to get it right.
It has saved the support staff some grief in the past. I've seen a couple of them get to 15 tries and then log in successfully.
I've had password auth disabled for many years, and it's quite liberating. Only a handful of times in all those years have I wanted to log in and not had a key handy. None of those times it was critical. It's interesting to listen to people give excuses about why disabling passwords is nice but won't work for them. None of them hold water, of course. Carrying physical keys for my house, car and office is a real pain. But I'm used to it and it's well worth what I get in return. So it is with ssh keys, but more so.
Standard response to SSH "bruteforcing" articles: use SPA or port knocking to hide sshd. Use key-based auth and disable passwords/root login. Use the AllowUsers setting to restrict which accounts can login.
One thing I've never understood is the point behind denying root login if you're enforcing key-based auth
The administrator then has to do one of
a) remember/manage a root password for su
b) remember/manage a user password for sudo
c) use passwordless sudo
What is the advantage?
Or is it just a hang-over from the telnet days where capturing the connection header was simple and thus preventing root login stopped the capturer from getting the root password at the start of the session?
The admin should be logging in as a standard user with sudo access and using sudo for any administrative tasks anyway. This minimizes how much code is running priviledged, it also allows his actions to be audited because they're all logged.
Additionally, if he's using key-based authentication, that key should still be encrypted so it can't be stolen and used by someone with unauthorised access to his filesystem. So he still has to remember the password to unlock his key when he fires up ssh-agent.
Lots of mentions of fail2ban here - would like to add a quick plug for OSSEC (http://ossec.net).
You can configure it to block brute-forcers, scanners, and other rapscallions, but it also provides a bit more power - rootkit checking, configurable file-change checking (i.e., don't alert me to md5 changes in my custom log files, but if /bin/su changes, holler!), and a bunch of out-of-the-box alerts for common services (apache, mysql, sshd, postgresql, etc etc - see http://www.ossec.net/doc/rules/index.html for the full list).
Also, you can configure alerts/blocks for your own events - if you have a custom app, this can be really, really useful. For example, if you see a certain Redis error more than X times in X seconds, sound the guards!
For personal servers with just a handful of nonobvious usernames, the simplest solution seems to be updating sshd's PermitRootLogin and AllowUsers settings. I just updated all my servers. Non-western names FTW.
I've been running SSH on a non-standard port on multiple machines for close to 10 years and haven't got a single attempt to log in that wasn't legit. Got quite a few of those over ESMTP though.
31 comments
[ 3.6 ms ] story [ 90.3 ms ] threadEven still, using a smart card, doesn't the encryption happen on it and not on the PC?
Even still, using a smart card, doesn't the encryption happen on it and not on the PC?
I've never locked myself out, nor have any of my users. At that point it doesn't matter how fast they are coming either, for the uptime of the script it keeps track of the IP addresses.
One thing we did add was a negative counter, so that if you logged in successfully you basically got a -1 added for your IP address, so if you were to log in successfully 100 times, now you have 110 times to guess your password before being locked out, which honestly hasn't been an issue yet.
I do find it funny watching the log from my script and seeing how many hosts we have blocked. Last time when the uptime on my box was almost 2 years, I had blocked almost 5000 IP addresses. Do note that I block them completely from SSH, and severely rate limit them to any and all other ports.
> Doesn't that provide information about which usernames are valid, though? Or does that turn out not to matter?
Yes it does. A targeted attack would be able to get information out of that. A giant botnet spamming authentication attempts? They apparently don't even notice if their connections are being dropped, never mind the difference between valid and invalid usernames.
It has saved the support staff some grief in the past. I've seen a couple of them get to 15 tries and then log in successfully.
I've migrated to fail2ban which supports iptables and can monitor services other than ssh.
[0] http://www.deer-run.com/~hal/sysadmin/pam_cracklib.html
[1] http://lani78.wordpress.com/2008/08/08/generate-a-ssh-key-an...
The administrator then has to do one of a) remember/manage a root password for su b) remember/manage a user password for sudo c) use passwordless sudo
What is the advantage?
Or is it just a hang-over from the telnet days where capturing the connection header was simple and thus preventing root login stopped the capturer from getting the root password at the start of the session?
Additionally, if he's using key-based authentication, that key should still be encrypted so it can't be stolen and used by someone with unauthorised access to his filesystem. So he still has to remember the password to unlock his key when he fires up ssh-agent.
You can configure it to block brute-forcers, scanners, and other rapscallions, but it also provides a bit more power - rootkit checking, configurable file-change checking (i.e., don't alert me to md5 changes in my custom log files, but if /bin/su changes, holler!), and a bunch of out-of-the-box alerts for common services (apache, mysql, sshd, postgresql, etc etc - see http://www.ossec.net/doc/rules/index.html for the full list).
Also, you can configure alerts/blocks for your own events - if you have a custom app, this can be really, really useful. For example, if you see a certain Redis error more than X times in X seconds, sound the guards!