It's interesting to log failed SSH attempts, I normally get someone trying to brute force access within an hour or two of brining a server up. Changing to a non standard port doesn't deter them for long.
Have you tried fail2ban? It really helps to keep the hacking attempts down on your ssh-server(s), especially if you increase the bantime from the default 600 seconds to a couple of million...
Important to note: You can whitelist IPs if you are worried about getting locked out from work/home. They may change periodically, but that's pretty easy to update as well.
I rolled my own log-monitoring solution. I have it so that there have to be N unsuccessful attempts in M seconds. There are multiple such thresholds (different combinations of N and M) with different banishment periods. The system will react to an acute flood, like when > 30 attempts are made in just a few seconds, say (which you would never do if you're clumsily mis-typing your password). The software also discriminates whether the same account is being tried multiple times, versus different accounts. If the same client IP address is trying different user ID's, it will be banned more easily than a client trying the same account. Someone trying three or more accounts is almost certainly an intruder, since a legit user on my system knows at most two accounts: their personal one and possibly root. Trying root is punished more swiftly than non-root, also. All these inputs contribute to a score, and the score determines the position on the banishment scale which translates to a period.
Oh, and a successful login will clear the record: the software scrubs all records of that IP from its cache, so then if you make new logins from the same IP with mistyped passwords, you're starting with a clean slate.
Note that Fail2ban doesn't support IPv6 at all [1]. I'm currently using sshguard [2] instead.
Edit: Thinking about it, I guess that blocking single IPv6 addresses, as sshguard does, doesn't really help at all, as it's easy to obtain huge subnets. (This might even open the door to an iptables DDoS?) Blocking whole subnets, on the other hand, doesn't sound clever either. (Who knows how many users are sharing the same subnet?)
So, should I just close port 22 on IPv6 altogether? Not a very sustainable solution. This needs more research!
I'm very curious about IPv6 banning best practices.
I imagine banning /64 networks is completely harmless, but I'm not really confident on that. I'd be wary of banning anything bigger, but getting a bigger address range is very cheap, so that may be required.
I've recently written a banning layer over my email server, in case somebody tries to guess the computer's passwords. It's a certainty that an attack exploiting IPv6 will go through, so I'm interested on the subject. By the other side, no attack it received up to now cared enough to read the EHLO reply and see that the server does not support login authentication, so I'm not that concerned, I guess I'll fully migrate to public key before any brute-force attempt is successful.
I think /64 would be the correct subnet size to ban in most cases.
My understanding is that /64 has become the de-facto standard subnet size for IPv6 sites. For example, I think that most home and business internet connections would be likely to receive a /64 allocation from their provider.
Moreover, many times the hosts will be generating themselves a new random address (within the /64) from time to time. So blocking a single IPv6 address probably won't block the user for long.
Conceptually, whenever a site (or hosted server, etc) would have been allocated a single address (/32) under IPv4, I think the analogous allocation for IPv6 is a /64, i.e.: 1.8 x 10^19 addresses.
If you look at RFC5375, you can see why /64 would be common:
- "An allocation of a prefix shorter then 64 bits to a node or interface is considered bad practice..."
- "Using subnet prefixes shorter than /64 would rarely be useful..."
- "Using subnet prefixes longer than /64 is not recommended for general use, and using them for links containing end hosts would be an especially bad idea..."
- "Using a subnet prefix length other than a /64 will break many features of IPv6, including Neighbor Discovery, Secure Neighbor Discovery, privacy extensions, parts of Mobile IPv6, PIM-SM with Embedded-RP, and Site Multihoming by IPv6 Intermediation, among others."
I don't even think you need to pump the ban time up that high. Most of these driveby brute-force attacks are just a bot that will give up and move on to some other server as soon as it's banned.
It's useful as part of a layered strategy. Your passwords still need to be secure enough that they can't be guessed easily. If they're easily brute-forced, fail2ban might slow things down, but it won't stop it.
Once you have that though, fail2ban does do a nice job of making the perpetual brute force attacks significantly less obnoxious. I've seen machines effectively DDoSed by brute-force ssh attempts, either with the network overloaded or filling up disk with ssh's logs of the failed attempts. Not as bad as getting hacked, but still annoying. A serious botnet attack will be able to spread it over thousands of IP addresses, but IME, those are relatively rare. Most attacks are still coming from a handful of hosts at a time and fail2ban basically stops them dead in their tracks.
The other useful thing to do with fail2ban is to use it along with a whitelisted range of networks. Even if you aren't on a private network, not every server needs ssh open to the entire world. You can often get away with whitelisting the local network and maybe a few ISPs' ranges where admins are likely to access it from. I run fail2ban even on machines on private networks. If one of the other machines gets hacked, one of the first things they'll usually do is start trying to brute-force other machines on the network. Since it will probably only be a couple compromised machines, fail2ban is useful here.
I find that it does also give me useful data as well. Eg, I have simple graphite metrics based on the number of bans/minute. If it gets unusually high, I can get an alert and perhaps notice that other hosts have been compromised, or do a quick firewall rule or two to drop packets from an entire network for a while.
It's just caution. Using port knocking by itself would be a recipe for security disaster. As an addition to an already secure system, it's okay if you just want to reduce junk in your logs, perhaps. Many would ask, in that case, why use port knocking at all?
The is the usual thing I hear. It is so incredible naive, though.
Any service, no matter how locked down and properly configured, still presents an attack surface. SSHD is a program like any other, and it can have bugs.
When you use port knocking with a properly configured SSHD, you are much more secure because that attack surface has been removed.
Recent Google Cloud Engine images have Sshguard enabled by default.
Now I do wonder: does this 'exploit' still cause a log line immediately after each attempt? And also, do iptables firewall rules also apply to open connections, or just to new ones? In the latter case, Sshguard and fail2ban are circumvented for a rather high number of attempts.
One of the common gotchas though is that if you add a rule to allow "Established" connections, and then try to add a new rule after that which says "block IP X", that will not sever existing connections.
To block existing connections you can remove that entry from the connection tracking table, or you can insert your "block IP X" rule above your "allow Established connections rule".
iptables rules can (trivially) apply to all traffic, or they can be sensitive to connection state.
In the case of cracking, you'd just want to apply a blanket blan: completely black-hole any packet whatsoever coming from that IP to your box. That will freeze all existing connections.
To ban new connections, you'd have a rule that applies to TCP connections in the NEW state, leaving the ESTABLISHED ones alone.
As I understand it this will log all passwords anyone enters -- even correct ones. This seems like a terrible idea, or at the very least should be used only after careful consideration. Even logging incorrect passwords is a bad idea, due to slight typos by legitimate users.
You are correct. The patch is meant for IT security research and honeypots. The purpose is to see what passwords the SSH brute-force bots are using. It should not be used on production systems where legit users are logging in via SSH. That would be unethical and possibly criminal (depending on location).
Also, with SSH, I use keys, not passwords. I encourage others to do that as well. You could have this patch on a honeypot and by using a SSH key, your account password would not be in play.
Right. Quote from article: "In some respects, the severity of vulnerability can be viewed as mild. But that assumes OpenSSH users are using a cryptographic key for authentication. Under such an arrangement, only computers with the private key are able to access the Internet-facing server. On top of that, servers themselves should be configured to limit the number of login attempts, and that measure should also go a long way toward making exploitation impractical."
Disable password auth and ensure you have another way in if necessary. Fail2ban, changing ports and port knocking doesn't really add anything more unless you're concerned about log sizes, and just increases the hassle for you.
Mozilla's wiki has some information on strengthening the security of your OpenSSH setup.
Use the stuff from that Mozilla wiki page and OpenSSH will be secure enough already. Unless more vulnerabilities are found that apply to such a configuration, or keys are compromised, port knocking in this case isn't going to stop anyone that OpenSSH wouldn't already stop on its own. It's effectively just a pointless layer of security that only adds more hassle to connect.
I'm not saying that port knocking doesn't have uses, just that it doesn't in this case. Unless you're trying to hide the service I guess, but I don't see why that would be necessary.
It adds no security. Assuming you use keypairs the attacker either has your private key or he doesn't. If they compromise your client to the point of having your private key they are likely to have access to whatever port knocker magic you have in place.
Thanks for checking, but I'm not sure you're correct. AFAICT setting the default to "no" is not due for official release until later this month[1]. Maybe some of the distros are patching the upstream default directly in their source (seems bad idea to me), but I at least checked the CentOS version you referenced and it appears to default to "yes" in the source (and the config excerpt you cited is commented out.)
I looked into OpenSSH's commit history ([2],[3],[4],[5]) and it looks like some waffling and/or release-process side-effects resulted in the man page in 6.9 saying the default is "no", but the actual code retaining "yes" (confirmed in the portable 6.9p1 tarball). I kind of hope I'm wrong somehow; this is a bit disturbing.
Ah, you're right. I read sshd_config(5) on Arch, which uses 6.9p1 and says the incorrect default is "no". I assumed this was the case on other distros.
So to correct my previous post (I can't seem to edit?), it should be, "Debian-based distros set `without-password`, and others use the default `yes`."
Thanks for the reply. On editing, I'm not sure exactly how it works, but posts on HN become uneditable at some point.
I came across this post[1] and bug comment[2]. If I'm understanding correctly, Red Hat will not follow the OpenBSD upstream on this! So I would guess CentOS and Fedora will also keep allowing root login, with password, by default.
The reason for this isn't security-related, if you've followed the steps above and have applied them to root. The reason to not allow root login is so that each of multiple administrators must login using their username and then become root or use sudo. That way if you look to see who did what, you get a username rather than "root did that". Which, of course, you usually need root to do things that end up being checked into.
I would say auditability is still security-related, but yes, this is good practice too. You should probably also make sure root has a locked password, so that things other than SSH won't allow root login.
> There are other things you can do, but I find this to be the lowest hanging fruit for the best security.
During the years I've found that changing the port to something like 7865 decreases login attempts up to 100%. Personally it's always the first thing I do along with changing LoginGraceTime to 5s
Usually automated bulk (non-targeted) hacking tools don't like to spend time looking for services on non-standard ports. I get zero ssh hack attempts on my machines ever since I changed to a non-standard port vs 100s of attempts everyday previously on standard ssh port.
I am curious as to why DenyHosts is not mentioned in this context (beside fail2ban and sshguard. Is its use deprecated?
I ask because I administer about 300 servers with DenyHost grandfathered on the servers. I've googled a bit to see the relative merits and DH is just for ssh whereas fail2ban is more general.
I somehow didn't know about sshguard - it looks interesting (and also seems to do more than just ssh brute force blocking).
So... do I need to take DH off the herd and replace?
Shouldn't effective brute forcing be prevented by the fact that this should be completely choking the server? I'm not completely sure what algorithms are currently used for password hashes in UNIX-likes but in any case, shouldn't this choke the server when it has to generate hashes for all those attempts at once (or consequently)? If you monitor loads at all this is very detectable. It's a good avenue for DDOS in that case though.
61 comments
[ 6.1 ms ] story [ 128 ms ] threadFar too many people ban themselves with fail2ban.
* Until you have a script repeatedly attempt w/o your keys unlocked
* Until you try from a new computer on the same network and fail to setup your keys first.
Ask me how I know :)
Oh, and a successful login will clear the record: the software scrubs all records of that IP from its cache, so then if you make new logins from the same IP with mistyped passwords, you're starting with a clean slate.
I've never locked myself out.
[0] http://www.cyberciti.biz/faq/iptables-connection-limits-howt...
Edit: Thinking about it, I guess that blocking single IPv6 addresses, as sshguard does, doesn't really help at all, as it's easy to obtain huge subnets. (This might even open the door to an iptables DDoS?) Blocking whole subnets, on the other hand, doesn't sound clever either. (Who knows how many users are sharing the same subnet?)
So, should I just close port 22 on IPv6 altogether? Not a very sustainable solution. This needs more research!
[1] https://github.com/fail2ban/fail2ban/issues/39
[2] http://www.sshguard.net/
I imagine banning /64 networks is completely harmless, but I'm not really confident on that. I'd be wary of banning anything bigger, but getting a bigger address range is very cheap, so that may be required.
I've recently written a banning layer over my email server, in case somebody tries to guess the computer's passwords. It's a certainty that an attack exploiting IPv6 will go through, so I'm interested on the subject. By the other side, no attack it received up to now cared enough to read the EHLO reply and see that the server does not support login authentication, so I'm not that concerned, I guess I'll fully migrate to public key before any brute-force attempt is successful.
My understanding is that /64 has become the de-facto standard subnet size for IPv6 sites. For example, I think that most home and business internet connections would be likely to receive a /64 allocation from their provider.
Moreover, many times the hosts will be generating themselves a new random address (within the /64) from time to time. So blocking a single IPv6 address probably won't block the user for long.
Conceptually, whenever a site (or hosted server, etc) would have been allocated a single address (/32) under IPv4, I think the analogous allocation for IPv6 is a /64, i.e.: 1.8 x 10^19 addresses.
If you look at RFC5375, you can see why /64 would be common:
- "An allocation of a prefix shorter then 64 bits to a node or interface is considered bad practice..."
- "Using subnet prefixes shorter than /64 would rarely be useful..."
- "Using subnet prefixes longer than /64 is not recommended for general use, and using them for links containing end hosts would be an especially bad idea..."
- "Using a subnet prefix length other than a /64 will break many features of IPv6, including Neighbor Discovery, Secure Neighbor Discovery, privacy extensions, parts of Mobile IPv6, PIM-SM with Embedded-RP, and Site Multihoming by IPv6 Intermediation, among others."
Especially since any sane bruteforcers iterate over IP addresses instead of passwords to avoid just this.
Once you have that though, fail2ban does do a nice job of making the perpetual brute force attacks significantly less obnoxious. I've seen machines effectively DDoSed by brute-force ssh attempts, either with the network overloaded or filling up disk with ssh's logs of the failed attempts. Not as bad as getting hacked, but still annoying. A serious botnet attack will be able to spread it over thousands of IP addresses, but IME, those are relatively rare. Most attacks are still coming from a handful of hosts at a time and fail2ban basically stops them dead in their tracks.
The other useful thing to do with fail2ban is to use it along with a whitelisted range of networks. Even if you aren't on a private network, not every server needs ssh open to the entire world. You can often get away with whitelisting the local network and maybe a few ISPs' ranges where admins are likely to access it from. I run fail2ban even on machines on private networks. If one of the other machines gets hacked, one of the first things they'll usually do is start trying to brute-force other machines on the network. Since it will probably only be a couple compromised machines, fail2ban is useful here.
I find that it does also give me useful data as well. Eg, I have simple graphite metrics based on the number of bans/minute. If it gets unusually high, I can get an alert and perhaps notice that other hosts have been compromised, or do a quick firewall rule or two to drop packets from an entire network for a while.
Port knocking has been mentioned before on HN and it generally gets a luke warm to negative response. It completely baffles me, this response.
Any service, no matter how locked down and properly configured, still presents an attack surface. SSHD is a program like any other, and it can have bugs.
When you use port knocking with a properly configured SSHD, you are much more secure because that attack surface has been removed.
Who cares about a few connections per hour if the sun will die before they brute force your private key?
Now I do wonder: does this 'exploit' still cause a log line immediately after each attempt? And also, do iptables firewall rules also apply to open connections, or just to new ones? In the latter case, Sshguard and fail2ban are circumvented for a rather high number of attempts.
One of the common gotchas though is that if you add a rule to allow "Established" connections, and then try to add a new rule after that which says "block IP X", that will not sever existing connections.
To block existing connections you can remove that entry from the connection tracking table, or you can insert your "block IP X" rule above your "allow Established connections rule".
In the case of cracking, you'd just want to apply a blanket blan: completely black-hole any packet whatsoever coming from that IP to your box. That will freeze all existing connections.
To ban new connections, you'd have a rule that applies to TCP connections in the NEW state, leaving the ESTABLISHED ones alone.
Also, with SSH, I use keys, not passwords. I encourage others to do that as well. You could have this patch on a honeypot and by using a SSH key, your account password would not be in play.
It is just the kind of thing I would expect dumb IT mgmt to require so they can ban swear words from passwords.
Mozilla's wiki has some information on strengthening the security of your OpenSSH setup.
https://wiki.mozilla.org/Security/Guidelines/OpenSSH
I'm not saying that port knocking doesn't have uses, just that it doesn't in this case. Unless you're trying to hide the service I guess, but I don't see why that would be necessary.
1. Disable password authentication altogether
2. Create and use keypairs
3. Add an "AllowUsers foo" line to /etc/ssh/sshd_config so that only your user is allowed to SSH
4. Install sshguard or fail2ban and set the ban limit to a week
There are other things you can do, but I find this to be the lowest hanging fruit for the best security.
I looked into OpenSSH's commit history ([2],[3],[4],[5]) and it looks like some waffling and/or release-process side-effects resulted in the man page in 6.9 saying the default is "no", but the actual code retaining "yes" (confirmed in the portable 6.9p1 tarball). I kind of hope I'm wrong somehow; this is a bit disturbing.
[1] http://www.openssh.com/txt/release-6.9 [2] https://github.com/openssh/openssh-portable/commit/88a7c598a... [3] https://github.com/openssh/openssh-portable/commit/d921082ed... [4] https://github.com/openssh/openssh-portable/commit/47aa7a0f8... [5] https://github.com/openssh/openssh-portable/commit/7de4b03a6...
So to correct my previous post (I can't seem to edit?), it should be, "Debian-based distros set `without-password`, and others use the default `yes`."
Thanks for the correction!
I came across this post[1] and bug comment[2]. If I'm understanding correctly, Red Hat will not follow the OpenBSD upstream on this! So I would guess CentOS and Fedora will also keep allowing root login, with password, by default.
[1] https://lists.fedoraproject.org/pipermail/package-announce/2... [2] https://bugzilla.redhat.com/show_bug.cgi?id=89216#c26
During the years I've found that changing the port to something like 7865 decreases login attempts up to 100%. Personally it's always the first thing I do along with changing LoginGraceTime to 5s
You probably also want to set kex, ciphers and MAC preferences in your client config for when you connect to servers with suboptimal settings.
Personally, I have also moved away from RSA keys and prefer using Ed25519 ones instead.
I ask because I administer about 300 servers with DenyHost grandfathered on the servers. I've googled a bit to see the relative merits and DH is just for ssh whereas fail2ban is more general.
I somehow didn't know about sshguard - it looks interesting (and also seems to do more than just ssh brute force blocking).
So... do I need to take DH off the herd and replace?
So it's like 15 bcrypt rounds on average hardware for ~500ms per hash?
Linux seems to use a pathetic 5000 rounds of SHA-512: http://www.akkadia.org/drepper/SHA-crypt.txt