Tools like these are essential if you have Internet-facing SSH services.
If you do not need Internet-facing SSH, do not enable it. It can be an attack vector and there are hundreds of bots that will try to brute-force access within minutes of opening Internet-facing SSH daemon.
From quickly skimming the docs, Fail2Ban can do a few other things like execute a script for the action. I have it doing an API call to block the offending IP on OPNSense firewall.
Or just run wireguard and ssh on top of that without all the fussy config and tooling to expose ssh to the internet. You could even live dangerously and just use telnet once you have wireguard going.
Tailscale is a good solution. I have no ports open on my internet facing router at all any more. Tailscale has proved to be as reliable as the servers themselves. I like it so much I use it to access my AWS machines as well, by running it as an exit node up there.
It's not TCP based so you can't port scan it like SSH, and it will only respond on successful authentication, so a failed attempt is undistinguishable from it not being there.
It is also fairly easy to DoS SSH by having too many connections in the authentication state leaving no slots open, which SSHGuard is useful to counter.
Apart from that, SSH's intrinsic security is the same, but if you have password authentication enabled, you are only as strong as the weakest password.
> Apart from that, SSH's intrinsic security is the same, but if you have password authentication enabled, you are only as strong as the weakest password.
A number of configurations also ship with ssh root login enabled by default. For example if you setup a new Linode VPS. You need to add a user, remember to at least turn off root access and probably also password access. I get why they're doing it because it easier, but yeah ... not a huge fan of this configuration.
Never used Linode. Most of the services I am familiar with will let you upload your SSH public key that is auto provisioned for root authorized_keys by cloud-init. Using SSH certificates would be even better but I don’t know of any cloud that support them.
It has orders of magnitude less code (which has been reviewed by many people), its configuration is a lot simpler, and it doesn’t allow password authentication, so… probably yes?
What I find funny though is that every implementation not involving Donenfeld has "drama" (Free- and NetBSD). The small code still leave a lot of room for failure.
That’s why I mentioned reviewed code. The original FreeBSD code which sparked the drama wasn’t reviewed.
And honestly some of the issues that were there are not something preventable by limiting oneself to even 50 LOC. I don’t want to speculate how they came to be though. It’s really baffling.
It was reviewed, with prejudice. Netgate (company behind pfSense) asked Donenfeld "by the way, we are merging this into FreeBSD next week, would you mind having a look?". Donenfeld dropped whatever he was doing (the short notice period: not cool at all), looked at the code, was horrified by what he saw, and blew the whistle. Instead of being grateful, the team at Netgate/pfSense started being snitty. Colin Perceval (Chief Security Officer for FreeBSD) decided to just yank it, but the clear influence Netgate has over the FreeBSD project is not a good look.
It runs over UDP and doesn't respond to anything until it sees a valid key sent to it, so right there it totally removes any need for stuff like port knocking. There's no way to use passwords or low security keys so there's less brute force chance. It's basically the same as going through all those machinations to make ssh secure publically (high quality key auth only, nonstandard port, port knocking, etc) but ready to go out of the box.
This is akin to asking if a bike is more reliable than a car. They are very different use cases with different implications, so it depends on what you're doing with them and how.
Wireguard is just a very simple network bridge. Whoever has the key can send anything over the network. There isn't a robust mechanism required to keep the key, to revoke it, to audit its use, to enable it to only provide access to specific applications. It doesn't have 1/20th the features SSH has, and SSH itself lacks a bunch of security features.
Bottom line is that you should only use Wireguard if you need a simple encrypted network bridge. It does not remove the need for other more complex security products.
> This is akin to asking if a bike is more reliable than a car. They are very different use cases with different implications, so it depends on what you're doing with them and how.
Cars and bikes are both fine for point to point travel. They'll each give you a vastly different experience though. It"s prolly better to stick with what you know how to use, in my opinion. You're likely gonna have a bad day if you crash and die.
> This is akin to asking if a bike is more reliable than a car.
It's really not. It's a technical question with many technical answers, thankfully they were provided by other commenters before you posted this.
If anything, it's similar to asking if a bike is faster than a car, to which you would reply than a bike might be faster in traffic because of small size but slower over long distances because of propulsion. It is possible to compare apples and oranges over specific axes.
> You could even live dangerously and just use telnet once you have wireguard going.
Using telnet makes me feel a little dirty and a lot alive. Using it over wireguard seems like taking a 60s muscle car and adding brakes that actually work before a track day; probably not a bad idea.
Dismissing security through obscurity as pointless is an advice from the 90s.
Basically security comes with layers of obscurity. Even key based auth is a random string that no one is meant to guess.
If you have more obscurity, the less chance one reaches to the actual attack surface.
If an attacker gains a key, without fail2ban etc, he may try bunch of user names against the key file consecutively but if a login attempt is limited to 3 times every 5 minutes, see it's better than nothing.
In a sane configuration, the only thing a tool like this should be doing for you is keeping your logs clear. If you're deploying something to mitigate the risk of brute-force attacks on your server, the problem is that it's possible to brute-force your server, not that you don't have an autoblocking system enabled.
These tools are popular, but I think they're kind of silly.
Later
I read some comments below, which compared this to Fail2ban (I had written this tool off as yet another Fail2ban-style script). But it's even scarier: it's written in C. I guess this is from like 2010? Don't run stuff like this.
> If you're deploying something to mitigate the risk of brute-force attacks on your server, the problem is that it's possible to brute-force your server, not that you don't have an autoblocking system enabled.
Well, even when accepting only certificates some brute force bots are dumb enough to keep trying to connect to try passwords, and this can give a lot of unnecessary CPU load due to all the initial connection crypto handshakes.
Though I have to say most of the bots these days are smart enough to quit as soon as password auth is turned off. But 10 years ago most were really dumb. Using fail2ban was more of a DDOS prevention than a security thing.
It seems improbable that the baseline load of a couple concurrent SSH attempts is higher than the load of running an extra Python daemon that's constantly checking your log files.
sshguard is C, and the log file for sshd won't have very much output if there are only a couple concurrent SSH attempts. Don't use this, but not because of concerns about performance overhead.
When I still ran my own server it was a pretty heavy load, like 30% or so. Much more than a minor nuisance.
But it was a SPARC server with 1 core at a few hundred MHz and no crypto acceleration. As you can tell this is a while ago :) So yeah I guess these days it's much less of an issue. But the number of possible attempts will have increased too of course.
I figured I might as well go pull up some numbers from my personal systems. I've got 3 internet-facing VMs that expose SSH to the world.
For the past 30 days, I've seen a combined total of 86k SSH attempts. I've not bothered to filter out my own legitimate sessions, given that they're in the ballpark of ~3 connections a day tops.
Total load on each of these systems has stayed below 0.5, and that's including the other actual work the servers do. I don't have a great way to pull out sshd specifically, but it's safe to say that these servers haven't had their operations affected at all by the nefarious attempts.
For what it's worth, the one thing I'm doing that could potentially impact the resource consumption is hardcoding cipher/keyexchange/MAC selections. Given that I control 100% of the legitimate clients for these SSH daemons, I've hardcoded the settings as follows:
Ciphers chacha20-poly1305@openssh.com
KexAlgorithms curve25519-sha256@libssh.org
MACs hmac-sha2-512-etm@openssh.com
HostKey /etc/ssh/ssh_host_ed25519_key
Over 95% of the spam connection attempts get dropped before my server ever does any crypto, because the connecting system doesn't support that KeyExchange choice.
> I don't have a great way to pull out sshd specifically
If it's using systemd, try
systemctl status sshd
If CPU accounting is turned on, you'll see how much CPU time was used by sshd, plus when it was last restarted, which at least gives you something to work with.
TIL about systemctl edit, thanks. Note to others: if you want to override an existing setting, rather than just add new settings, you'll need to use 'systemctl edit --full'.
I’d say that fail2ban is a Band-Aid too, if brute forcing passwords is an issue your problem is that you’re not using key based auth.
If someone wants to go to the trouble to break crypto, it would be cheaper to infiltrate the ssh project and implement back doors after a few years of building trust. If that’s your threat model, double wrap your protocols (WireGuard jumphost > ssh type stuff)
As you correctly point out, password authentication should not be used for SSH to begin with, but more importantly fail2ban is not a piece of software written to stop brute force password attacks, it’s a very general tool for triggering network blocking actions based on specific patterns found in logs.
My only point being that there are some great use cases for fail2ban, and while this isn’t one of them it’s important to remember that this does not make fail2ban a misdirected or useless security initiative by default, in case anyone gets that impression from reading these comments.
It doesnt stop determined attackers who have access to a botnet, but it does stop low sophistication attacks.
If brute forcing anything is a concern then addressing that concern should come first, focusing effort on fail2ban is a cost that is probably better spent elsewhere.
I agree. I never understood why people use all this stuff. If you use strong enough passwords, it's impossible to be brute-forced, simple as that. If you can't control people on your servers and afraid of weak passwords, just disable password authentication and that's about it.
Now the issue of logs is real. I tried to configure debian to remove noise from the logs and I couldn't. May be someone with more knowledge would suggest how to do so. Because I'd like to see successful authentications in the logs and only those. So that could be a valid reason to change port, etc. But that's only because logs are lacking proper configurability.
> I tried to configure debian to remove noise from the logs and I couldn't.
Not very elegant (as in, not with built-in tools), but using Logstash would be one way of achieving that. There probably is a better (simpler) way, though.
Edit:
After some research, rsyslog seems like a more appropriate candidate. But the docs don't seem too novice-friendly and the configuration is quite obscure. Tradeoffs.
However, I've worked at a lot of startups, and no matter how much you try to discipline people, someone thinks their circumstance is special (and it somehow requires a password-based login), or someone is just lazy and sets a password for unfathomable reasons. So, for those people, I run fail2ban to allow it to act as a rate-limiter. But tptacek is right, in any sane situation, you don't really need it. But sometimes the situation isn't sane.
> just disable password authentication and that's about it.
a.) for what ever reason, I've had third-parties insist on passwords. (In healthcare¹, too. Ugh.) b.) For quite a while, Azure's "AADLogin" used keyboard-interactive, which meant that you couldn't disable it. Thankfully, that got fixed in a recent version.
¹In addition to fail2ban, we also did firewalls at the cloud level. The password was a good (secure) one, but I'd sleep easier with a key-based login.
Big issue in healthcare is a lot of legacy integration brokers that had SFTP bolted on post-release using the existing UX and config framework for FTP, which had historically been the dominant protocol in healthcare. Depending on the broker sometimes key auth isn't supported at all, it is supported but has to be configured in a strange way that's different from the normal SFTP/FTP config, or it's supported just fine but the company has business processes like paperwork for their integration subcontractor that still assume FTP is used (this latter one does unfortunately come up...).
You'll see the same thing in financial software for basically the same reasons.
> I never understood why people use all this stuff ... Now the issue of logs is real. I tried to configure debian to remove noise from the logs and I couldn't.
Sounds like you understand at least one reason people use stuff like this (including, most prominently, fail2ban). Rate limiting and banning IPs trying to brute force and/or probe open ports (not just SSH) goes a long way to cleaning up your logs, your real time state, as well as other things, like frustrating the attacker, not wasting connections on them.
Having said that, this particular tool looks a bit shit ...
Not disagreeing in general and disallowing password login for root is IMO really a no-brainer.
But, it's nonetheless a fact that a (temporary) nft drop rule in the kernel is much more effective than traffic hitting a user space daemon like SSH.
Some NICs even allow offloading that (e.g., in form of a bpf program) to the card itself, so that such traffic won't ever hit the system (CPU) at all.
So yes, SSH can be made secure and rate-limited without anything else, but that won't be the most efficient solution; now fail2ban/sshguard/... ain't no free lunch either, so one needs to weight the added (IMO small, but still) complexity against benefits in a specific use case.
Just to double check my sshd settings I logged in to a server I have, and while it had "PasswordAuthentication no" set in the normal place in the config file, there was a line added to the very end of it "PasswordAuthentication yes", which I guess overrides the previous one.
I don't think anybody with a sane config need fail2ban or similar. But as a sysadmin who works for clients who don't understand such concepts, sometimes it's easier going "okay, we will install it on your systems for $x" than explaining why it's useless. It's a business politics tool.
Nobody serious thinks the P-curves, which is what we're talking about, were weakened by NIST. I don't think anybody at all thinks the DSA algorithm was "intentionally weakened", but lots of people, reasonably, believe it was incompetently designed.
Ed2519 has the virtue of both a better curve and a better signing algorithm, which is why it's used in preference to ECDSA. If you're using 25519, don't bother keeping the rest of the algorithms enabled.
I think these hardening guides are mostly pretty silly too, for whatever that's worth. The important thing is making sure nobody can log into anything with a password. The rest is just liturgical.
I think it is the issue where ecdsa is a NIST standard that was based on an elliptic curve provided by the NSA. As the NSA has been known to suggest poor cryptography primitives in the past, many groups recommend ed25519 (same process, different and more trusted curve generation process)
Would you recommend a better guide? Is the config suggested by it bad? Not everyone has the skill/time to deeply understand everything immediately, and having a saner starting point is a good thing.
while it is not the best, I do regularly code audit OpenSSH.
For sane default, you can generate your working but sane ssh_config and sshd_config based on my inner code knowledge and network security and as an IDS/IPS architect.
Each and every known OpenSSL (v8.4; v9.0 is underway) settings in the config files have annotations and details and many have additional links detailing why. I also note where each and every config settings are found in the source code by nesting, what protocol state, control state, authentication stage, and lockings (makes code review so much easier for me) also in its comment section of each config setting.
Even has a bash script to let you create these config files (defaults to your subdirectory for pretesting, but can also as an option update /etc/ssh, which I confidently do) into something that would pass an SSH audit for ssh-audit (I’m a contributor), CISecurity and often better.
Also these scripts generates a script containing proper file permission settings based on top generic Linux File System variants (basic variants like APT, pacman, Portage, DNF/RPM) ).
I also offer SSH bastion setup (adjacent to my URL given below) as well.
I also enjoy using certificate-by-user as an authentication mechanism for maximum ease of sysadmin use when it comes to emergency mass-blocking or occasional employee departures.
I mean, yeah, these things are kind of silly. So is, "But it's even scarier: it's written in C. I guess this is from like 2010? Don't run stuff like this."
It depends I guess on where you start the count; some people would say Elias' "Smashing The Stack", I'd say Lopatic's HP-UX exploit, some people would go all the way back to RTM's worm.
Well it does make the developer try harder to have a vulnerability. You can't just forget to bound check an array indexing, you need to actually take instructions from the Internet (e.g. eval, import, or exec in case of fail2ban), but of course the way this happens is usually very indirect (e.g. race condition).
No programming language can stop a determined developer from creating a vulnerability, but some languages make the task easier than others.
Possibly the best tool in this space is blacklistd / blocklistd, which just exposes some reporting APIs and hooks them directly in sshd, rather than trying to parse failed logins out of logs. That said, meh.
Edit: Yeah, there's no race condition; the client sends the offending socket to blocklistd over the blocklistd socket.
(Originally…)
I'm a bit curious; it also seems to imply that it logs the fd number, and then blocklistd queries the kernel to obtain the remote's address. That seems like a race condition:
1. blocklist call is made
2. daemon closes the socket (connection has failed, after all)
3. next client connects
4. lookup is made by blocklistd, and the wrong peer is potentially incriminated
(Actually, perhaps it's sending the fd over the socket. I forget you can do that, and the README is hard for me to follow. The "like syslog" also implied that it's logging over syslog to me, but that's not what the code does.)
Remember when fail2ban had an RCE to execute things on a server via root user and it was as simple as parsing an email string because it was executing the mail binary on the other end? [1]
That's pretty much the reason why I started to write my own guard tools.
I don't need people from China or Russia logging in to my server. I don't understand why tools like this are never built in an ASN aware manner, blocking mechanisms would be so much easier.
I _WANT_ to harvest IP's of PC's trying to SSH into any of my servers. I want them because then I can block ALL traffic coming from their IP's.
At my last gig, I had thousands of domains and hundreds of servers running on every major (and a shitload of minor) service provider. I custom scripted all these servers to compile all the IP's that attempted a connection, send those lists back to HQ, and then distribute back to each node a block-list table that I could dynamically apply to each edge device. One compromised computer even trying to login was completely blocked from 5000+ domains within 15 minutes.
I'm not worried about my ssh login being brute-forced, but if your computer is compromised I want nothing to do with you.
There is a whole Internet worth of random IPv4 addresses being variously used to variously attack various computers. Your blocklist doesn't even approximate it. What's the point of trying to track and block 0.000000001% of the machines that might be used to attack you?
What’s worse, those IPs are probably (at least in part) dynamic, from VPS servers in data centres, or using residential Proxies or compromised machines behind a NAT. A soon as those addresses change or get assigned another customer, you’re blocking potentially legitimate traffic…
I dont see anything wrong in collecting and blocking them for some time, not infinitely of course. Blocking at the firewall effectively means blocking further communication and requests that cost you bandwidth and computing time. Why not save that for the legitimate visitors?
Because you block legitimate visitors. After we ran out of IPv4 space and started to use CGNAT instead of IPv6, the internet has become a bastardized version of itself.
If carriers/ISPs monitored their networks better (or acted on abuse@ emails), their IPs wouldn't wind up on blocklists (either temporarily or permanently).
edit: That isn't to say they should be logging every packet... But if I work for XYZ Hosting Company and spin up a new VM, hand you the IP, and you turn the IP into a mini-shodan scanner... Is the hosting company at least a little responsible for what happens next?
OpenSSH doesn't try to parse untrusted text input that can be fairly easily manipulated by an attacker (i.e. in this case the logs). That's something that should be scary in any language, but C is definitely one of the worst choices possible for this particular task.
>These tools are popular, but I think they're kind of silly.
Really? And how do you block someone who slowly try's (every 2-4 minutes) to bruteforce your smtp? Yes your openSSH has some slowdown mechanisms builtin, but your Imap-Server? Smtp-Server? Your openTTD-Server? However i think using the logs is the wrong approach hence my use of blacklistd* whenever possible.
If your private SSH key is leaked, you can expect the attacker to come up in with a single successful login attempt, sshguard, fail2ban etc. won't help.
The only help against that scenario is a true multi factor setup, or maybe a hardware-based security token that needs a second factor (fingerprint or PIN) to unlock.
hope you don't use github (which discloses all users keys, attached to their usernames there), and/or the same username in multiple places.
SSH also leaks the keys it'll accept, it doesn't use a zero knowledge proof to first determine if the client and the server have mutually acceptable keys.
FWIW, you can make your SSH itself key-auth only and then setup whatever second factor you like via pam. (including fancy stuff if you like, n of m yubikeys ...)
I would probably use something like pam_tally or pam_shield, since it's taking counts directly from the login flow and not via log files.
Mentioned in other comments, protecting from brute force isn't really the big benefit. The benefit is reducing noise in logs so that you can see more serious activity.
One to two lines of iptables limits connection attempts, making a brute force attack take thousands of years. No logging, no monitoring, no extra software.
If you only use SSH keys, no brute force attack will work anyway.
I experienced this at a previous company. Yocto/bitbake wanted to do a lot of git clones over SSH very quickly, but a firewall interpreted this as a brute force because it couldn’t tell that each SSH connection was actually successfully authenticating.
Doesn't Dovecot now have a fairly capable brute force password attack prevention system? Is there any reason that someone would still use something like this anyway?
Couldn't find anything about Exim or Postfix integrated brute force prevention on SMTP connections...
I get keys are better than passwords, but how do you enforce key rotation for non-technical users when they can barely manage to changer their passwords? I much prefer 2FA than keys for non-technical users.
For keys used in authentication, I feel like rotation largely just protects against keys being leaked. If you can enforce keys to exist only on a secure element then to me I feel like rotation is no longer needed in this particular scenario. With the exception of the key strength no longer being enough or something similar.
Do non-technical users know how to operate a secure element? Is that an encrypted home drive on a laptop protected by a weak password?
I appreciate your input, but I still think 2FA is the most secure and usable method for non-technical people. Of course, no SMS (at least outside the US).
There are a lot of commercial solutions out there, but for those of us with macs and without control of our org's IT spend, there's Secretive [1], although it lacks a way to prove that a key is hardware backed.
I'm not sure about the actual situation, but I have an option in my cloudflare dashboard to automatically save my site to the way back machine and fall back to that if the origin is ever down, so that's a totally plausible explanation
PSA: If you are using AWS, you should be using Session Manager for remote access to your EC2 instances if possible - it is free and trivial to enable. Azure offers Azure Bastion, which is exactly what it says. Both of these can keep hostile SSH connections completely away from your servers, and also help to aggregate access logs.
I use crowdsec also nowadays, but I don't trust it like I trust fail2ban. Due to opaque answers from the engine on how it has protected my hosts. Considering going all the way Back to fail2ban again.
Hey, head of community at CrowdSec here. Could you elaborate on your situation and the 'opaque' replies from the agent you're receiving in a mail to klaus at crowdsec dot net? Very interested in understanding your issues and hopefully help you to build trust :-)
The main issue is that on my hosts where fail2ban are running, I see week on week activity and banned hosts.
When I look into cscli decisions or cscli metrics, it gives me the indication that nothing is happening which I don't believe is true.
Maybe it is doing the work it promises, but I can't easily see it.
This could be a false negative, and that there genuinely is less malicious connection attempts. The busy fail2ban are bastion hosts on AWS, while the others are hosted on DigitalOcean. For me as a user though, I wish there was a way to see historically blocked hosts. The last time I looked half a year ago, this was not available in CrowdSec.
Hey and thanks for your reply. This is done easily in the cscli with the 'decisions' command: https://doc.crowdsec.net/docs/next/cscli/cscli_decisions. 'sudo cscli decisions list' lists all local decisions as you request :-). If you want to look into a decision, 'explain' is your friend: https://doc.crowdsec.net/docs/next/cscli/cscli_explain or simply look in /var/log/crowdsec.log (on Linux).
I hope this answers your question. If not, you're welcome to ask again or join our Discord at https://discord.gg/crowdsec (which is by far the best way to get help fast).
Would be nice to have this as eBPF cgroup socket filter, attachable to the systemd service (or OpenSSH could load it itself). Then it's not needed to parse the text logs anymore.
In 2022, I can't think of good reasons to run a Internet-facing ssh server. Wireguard should be your only Internet facing protocol for any private access. Once you establish a Wireguard tunnel, you can ssh to that machine if you like.
And, please don't use passwords, instead use certificates. You can easily have system to create and use a short-term valid login certificates. Also, don't forget to authenticate your hosts to users with host certificates.
All of the above is needed even if it is behind wireguard. That's defense in depth. Ensure there is no shared-fate between the two authentication systems.
Also, consider multi-factor personal authentication – biometric+external auth (FIDO2 device with fingerprint auth) and device-auth (device-secure-enclave bound certificates) where device is unlocked by a password/pin from your memory.
If you do these, you don't need silly hacks like SSHGuard.
SSH certificates don’t have an equivalent in Wireguard. They are a much more scalable way of provisioning authentication.
Defense-in-depth is a good idea, but has its limits. I don’t think there’s too much value to layering SSH over Wireguard.
Agreed on SSHGuard type things not being necessary (for authentication protocol). SSH is not special. People can attempt to bruteforce Wireguard too. In fact, the only difference seems to be SSH making the logs (more easily) accessible.
The main downside to SSH is that it allows in default configuration to use insecure passwords for authentication (but this can be disabled). Wireguard is strictly better since passwords are never an option.
Protocol negotiation is another downside, but I think it’s a smaller one. The SSH devs are fairly good about removing insecure schemes from the defaults.
Sure, but there might be Wireguard vulnerabilities as well. Why not layer SSH over Wireguard over IPSec over OpenVPN just to be safe?
I think that layering may be useful given a certain threat model, but you shouldn’t just layer protocols without actually considering the probability of vulnerability and considering the other tradeoffs you are making. Otherwise the only correct answer would be to layer SSH over every independent VPN protocol.
Considering the security track record of SSH, I don’t see the value to layering, when on the other side, you now need to manage multiple keys and an entirely separate protocol that isn’t as ubiquitous (yet). The time spent doing this might be better spent building robust and testable auto-update infrastructure or architecting the applications on the server to reduce the risk of contagion given an RCE vulnerability.
Defense in depth works on the principle that you will have to find vulnerabilities in multiple heterogeneous and independent systems before you get to the prize.
Reason to put wireguard as the first line of defense before ssh is because of wireguard's simplicity and robustness vs the track record of ssh implementations.
Sure, defense in depth is great, but there needs to be an argument about the tradeoff. Otherwise as I said, everyone should run OpenSSH over Wireguard over IPSec over OpenVPN, because then you need to exploit all of them to do an exploit and that is harder to do than exploiting any smaller subset.
Why is the tradeoff of adding Wireguard worth it? Sure it’s extra security, but is it really worthy managing multiple keys, not-as-ubiquitous protocol, less flexible administration and no CA? I don’t think there is a correct answer (it depends on the threat model), but your original comment made it seem like there was no tradeoff and there was an obvious correct answer.
What setup would you recommend for systems that at some point are owned by a customer (but which you might still need to login to frequently; and you might need to access private repositories from)?
I fixed ssh knocking with having wireguard VPN in front of all of our servers. We still require authentication and all inbound connections (even from the VPN ) are untrusted.
148 comments
[ 3.1 ms ] story [ 149 ms ] threadTools like these are essential if you have Internet-facing SSH services.
If you do not need Internet-facing SSH, do not enable it. It can be an attack vector and there are hundreds of bots that will try to brute-force access within minutes of opening Internet-facing SSH daemon.
Best practices for Internet-facing SSH:
- run on non-standard port (not port 22)
- disable passwords, use SSH passkeys instead: https://www.techrepublic.com/article/how-to-setup-ssh-key-au...
- disable root SSH login
- run fail2ban, sshguard, or similar "block IP addresses for suspicious activity" services
- setup port knocking: https://www.tecmint.com/port-knocking-to-secure-ssh/
edit:
- run WireGuard VPN (https://en.wikipedia.org/wiki/WireGuard) for defense-in-depth
SSHGuard however is much lighter and even less to configure manually.
I used fail2ban in the past and it was helpful.
If I was currently running an Internet-facing SSH daemon, I would give SSHGuard a try.
Nice! Wireguard was not around when I was using Internet-facing SSH in a previous job.
Adding it to the recommendations.
It is also fairly easy to DoS SSH by having too many connections in the authentication state leaving no slots open, which SSHGuard is useful to counter.
Apart from that, SSH's intrinsic security is the same, but if you have password authentication enabled, you are only as strong as the weakest password.
A number of configurations also ship with ssh root login enabled by default. For example if you setup a new Linode VPS. You need to add a user, remember to at least turn off root access and probably also password access. I get why they're doing it because it easier, but yeah ... not a huge fan of this configuration.
Edit: deleted false information about TLS.
And honestly some of the issues that were there are not something preventable by limiting oneself to even 50 LOC. I don’t want to speculate how they came to be though. It’s really baffling.
Here's a good summary:
https://arstechnica.com/gadgets/2021/03/buffer-overruns-lice...
Contrast this with how easily and well Wireguard was integrated into OpenBSD.
Wireguard is just a very simple network bridge. Whoever has the key can send anything over the network. There isn't a robust mechanism required to keep the key, to revoke it, to audit its use, to enable it to only provide access to specific applications. It doesn't have 1/20th the features SSH has, and SSH itself lacks a bunch of security features.
Bottom line is that you should only use Wireguard if you need a simple encrypted network bridge. It does not remove the need for other more complex security products.
Cars and bikes are both fine for point to point travel. They'll each give you a vastly different experience though. It"s prolly better to stick with what you know how to use, in my opinion. You're likely gonna have a bad day if you crash and die.
It's really not. It's a technical question with many technical answers, thankfully they were provided by other commenters before you posted this.
If anything, it's similar to asking if a bike is faster than a car, to which you would reply than a bike might be faster in traffic because of small size but slower over long distances because of propulsion. It is possible to compare apples and oranges over specific axes.
Using telnet makes me feel a little dirty and a lot alive. Using it over wireguard seems like taking a 60s muscle car and adding brakes that actually work before a track day; probably not a bad idea.
Basically security comes with layers of obscurity. Even key based auth is a random string that no one is meant to guess.
If you have more obscurity, the less chance one reaches to the actual attack surface.
If an attacker gains a key, without fail2ban etc, he may try bunch of user names against the key file consecutively but if a login attempt is limited to 3 times every 5 minutes, see it's better than nothing.
These tools are popular, but I think they're kind of silly.
Later
I read some comments below, which compared this to Fail2ban (I had written this tool off as yet another Fail2ban-style script). But it's even scarier: it's written in C. I guess this is from like 2010? Don't run stuff like this.
Well, even when accepting only certificates some brute force bots are dumb enough to keep trying to connect to try passwords, and this can give a lot of unnecessary CPU load due to all the initial connection crypto handshakes.
Though I have to say most of the bots these days are smart enough to quit as soon as password auth is turned off. But 10 years ago most were really dumb. Using fail2ban was more of a DDOS prevention than a security thing.
> Using fail2ban was more of a DDOS prevention than a security thing.
But it was a SPARC server with 1 core at a few hundred MHz and no crypto acceleration. As you can tell this is a while ago :) So yeah I guess these days it's much less of an issue. But the number of possible attempts will have increased too of course.
For the past 30 days, I've seen a combined total of 86k SSH attempts. I've not bothered to filter out my own legitimate sessions, given that they're in the ballpark of ~3 connections a day tops.
Total load on each of these systems has stayed below 0.5, and that's including the other actual work the servers do. I don't have a great way to pull out sshd specifically, but it's safe to say that these servers haven't had their operations affected at all by the nefarious attempts.
For what it's worth, the one thing I'm doing that could potentially impact the resource consumption is hardcoding cipher/keyexchange/MAC selections. Given that I control 100% of the legitimate clients for these SSH daemons, I've hardcoded the settings as follows:
Ciphers chacha20-poly1305@openssh.com
KexAlgorithms curve25519-sha256@libssh.org
MACs hmac-sha2-512-etm@openssh.com
HostKey /etc/ssh/ssh_host_ed25519_key
Over 95% of the spam connection attempts get dropped before my server ever does any crypto, because the connecting system doesn't support that KeyExchange choice.
If it's using systemd, try
If CPU accounting is turned on, you'll see how much CPU time was used by sshd, plus when it was last restarted, which at least gives you something to work with.If not, you can enable it with:
then adding: and restarting the daemon.If someone wants to go to the trouble to break crypto, it would be cheaper to infiltrate the ssh project and implement back doors after a few years of building trust. If that’s your threat model, double wrap your protocols (WireGuard jumphost > ssh type stuff)
My only point being that there are some great use cases for fail2ban, and while this isn’t one of them it’s important to remember that this does not make fail2ban a misdirected or useless security initiative by default, in case anyone gets that impression from reading these comments.
If brute forcing anything is a concern then addressing that concern should come first, focusing effort on fail2ban is a cost that is probably better spent elsewhere.
Now the issue of logs is real. I tried to configure debian to remove noise from the logs and I couldn't. May be someone with more knowledge would suggest how to do so. Because I'd like to see successful authentications in the logs and only those. So that could be a valid reason to change port, etc. But that's only because logs are lacking proper configurability.
Not very elegant (as in, not with built-in tools), but using Logstash would be one way of achieving that. There probably is a better (simpler) way, though.
https://www.elastic.co/guide/en/logstash/current/pipeline.ht...
Edit: After some research, rsyslog seems like a more appropriate candidate. But the docs don't seem too novice-friendly and the configuration is quite obscure. Tradeoffs.
https://en.m.wikipedia.org/wiki/Rsyslog
However, I've worked at a lot of startups, and no matter how much you try to discipline people, someone thinks their circumstance is special (and it somehow requires a password-based login), or someone is just lazy and sets a password for unfathomable reasons. So, for those people, I run fail2ban to allow it to act as a rate-limiter. But tptacek is right, in any sane situation, you don't really need it. But sometimes the situation isn't sane.
> just disable password authentication and that's about it.
a.) for what ever reason, I've had third-parties insist on passwords. (In healthcare¹, too. Ugh.) b.) For quite a while, Azure's "AADLogin" used keyboard-interactive, which meant that you couldn't disable it. Thankfully, that got fixed in a recent version.
¹In addition to fail2ban, we also did firewalls at the cloud level. The password was a good (secure) one, but I'd sleep easier with a key-based login.
You'll see the same thing in financial software for basically the same reasons.
Sounds like you understand at least one reason people use stuff like this (including, most prominently, fail2ban). Rate limiting and banning IPs trying to brute force and/or probe open ports (not just SSH) goes a long way to cleaning up your logs, your real time state, as well as other things, like frustrating the attacker, not wasting connections on them.
Having said that, this particular tool looks a bit shit ...
You don't need fail2ban to rate-limit SSH.
RFTM ... MaxStartups, PerSourceMaxStartups, PerSourceNetBlockSize.
> trying to brute force
We know about the SSH daemon config. You still get all sorts of probing that isn't a brute force on SSH.
https://github.com/fail2ban/fail2ban/tree/master/config/filt...
But, it's nonetheless a fact that a (temporary) nft drop rule in the kernel is much more effective than traffic hitting a user space daemon like SSH.
Some NICs even allow offloading that (e.g., in form of a bpf program) to the card itself, so that such traffic won't ever hit the system (CPU) at all.
So yes, SSH can be made secure and rate-limited without anything else, but that won't be the most efficient solution; now fail2ban/sshguard/... ain't no free lunch either, so one needs to weight the added (IMO small, but still) complexity against benefits in a specific use case.
The server is reinstalling the OS right now...
Still, if you cannot explain how that second line got in there, it is probably prudent to reinstall (perhaps even reflash BIOS).
Do you think using key-based auth and disabling password authentication is enough? or are there other things you would recommend?
OT - nice Fibonacci in your profile :D
If there’s something better please correct me!
I find it interesting that one of the suggestions is to disable the ECDSA host key. I thought ECDSA was one of the "good" algorithms?
Here's the long answer: https://blog.cr.yp.to/20140323-ecdsa.html
Ed2519 has the virtue of both a better curve and a better signing algorithm, which is why it's used in preference to ECDSA. If you're using 25519, don't bother keeping the rest of the algorithms enabled.
I think these hardening guides are mostly pretty silly too, for whatever that's worth. The important thing is making sure nobody can log into anything with a password. The rest is just liturgical.
Take a look at the "concerns" section of the wiki page. https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signatu...
For sane default, you can generate your working but sane ssh_config and sshd_config based on my inner code knowledge and network security and as an IDS/IPS architect.
Each and every known OpenSSL (v8.4; v9.0 is underway) settings in the config files have annotations and details and many have additional links detailing why. I also note where each and every config settings are found in the source code by nesting, what protocol state, control state, authentication stage, and lockings (makes code review so much easier for me) also in its comment section of each config setting.
Even has a bash script to let you create these config files (defaults to your subdirectory for pretesting, but can also as an option update /etc/ssh, which I confidently do) into something that would pass an SSH audit for ssh-audit (I’m a contributor), CISecurity and often better.
Also these scripts generates a script containing proper file permission settings based on top generic Linux File System variants (basic variants like APT, pacman, Portage, DNF/RPM) ).
I also offer SSH bastion setup (adjacent to my URL given below) as well.
I also enjoy using certificate-by-user as an authentication mechanism for maximum ease of sysadmin use when it comes to emergency mass-blocking or occasional employee departures.
https://github.com/egberts/easy-admin/tree/b74765baa450593be...
Why are you scared of C? Also, should I remove sshguard? I dont want my server going down.
Btw, it has been updated after 2010. Are you talking about 2010 release ?
Why would being written in C make it scarier?
No programming language can stop a determined developer from creating a vulnerability, but some languages make the task easier than others.
https://github.com/zoulasc/blocklist
(Originally…)
I'm a bit curious; it also seems to imply that it logs the fd number, and then blocklistd queries the kernel to obtain the remote's address. That seems like a race condition:
(Actually, perhaps it's sending the fd over the socket. I forget you can do that, and the README is hard for me to follow. The "like syslog" also implied that it's logging over syslog to me, but that's not what the code does.)That's pretty much the reason why I started to write my own guard tools.
I don't need people from China or Russia logging in to my server. I don't understand why tools like this are never built in an ASN aware manner, blocking mechanisms would be so much easier.
[1] https://research.securitum.com/fail2ban-remote-code-executio...
if (notMe)
Though I agree, security is a mindset not a tool you can simply download or run.At my last gig, I had thousands of domains and hundreds of servers running on every major (and a shitload of minor) service provider. I custom scripted all these servers to compile all the IP's that attempted a connection, send those lists back to HQ, and then distribute back to each node a block-list table that I could dynamically apply to each edge device. One compromised computer even trying to login was completely blocked from 5000+ domains within 15 minutes.
I'm not worried about my ssh login being brute-forced, but if your computer is compromised I want nothing to do with you.
edit: That isn't to say they should be logging every packet... But if I work for XYZ Hosting Company and spin up a new VM, hand you the IP, and you turn the IP into a mini-shodan scanner... Is the hosting company at least a little responsible for what happens next?
While its similar to sshguard or blocklistd, I didn't know they existed when badips died a couple years back. https://www.nubi-network.com/faq.php
The list file should work for either ingress or egress (or both).
I wanted to post about it on here some time ago, but I'm kindof bad at advertising.
Really? And how do you block someone who slowly try's (every 2-4 minutes) to bruteforce your smtp? Yes your openSSH has some slowdown mechanisms builtin, but your Imap-Server? Smtp-Server? Your openTTD-Server? However i think using the logs is the wrong approach hence my use of blacklistd* whenever possible.
* https://www.freebsd.org/cgi/man.cgi?query=blacklistd&sektion...
* https://man.netbsd.org/NetBSD-7.0/blacklistd.8
Even obscurity has its place (e.g. using a different SSH port)
It's time another prevention system may save your day.
The only help against that scenario is a true multi factor setup, or maybe a hardware-based security token that needs a second factor (fingerprint or PIN) to unlock.
SSH also leaks the keys it'll accept, it doesn't use a zero knowledge proof to first determine if the client and the server have mutually acceptable keys.
Mentioned in other comments, protecting from brute force isn't really the big benefit. The benefit is reducing noise in logs so that you can see more serious activity.
If you only use SSH keys, no brute force attack will work anyway.
https://www.systutorials.com/how-to-use-iptables-to-limit-ra... https://making.pusher.com/per-ip-rate-limiting-with-iptables... https://stackoverflow.com/questions/18616246/iptables-limit-... https://blog.oshim.net/2009/06/iptables-to-rate-limit-incomi... https://stackoverflow.com/questions/47775809/iptables-rate-l... http://blog.serverbuddies.com/using-iptables-to-rate-limit-i...
Limiting Internet-facing SSH connections to key-only is definitely a good idea, though.
Couldn't find anything about Exim or Postfix integrated brute force prevention on SMTP connections...
Perhaps I'm overlooking something?
Do non-technical users know how to operate a secure element? Is that an encrypted home drive on a laptop protected by a weak password?
I appreciate your input, but I still think 2FA is the most secure and usable method for non-technical people. Of course, no SMS (at least outside the US).
https://github.com/maxgoedjen/secretive
I use crowdsec also nowadays, but I don't trust it like I trust fail2ban. Due to opaque answers from the engine on how it has protected my hosts. Considering going all the way Back to fail2ban again.
The main issue is that on my hosts where fail2ban are running, I see week on week activity and banned hosts.
When I look into cscli decisions or cscli metrics, it gives me the indication that nothing is happening which I don't believe is true. Maybe it is doing the work it promises, but I can't easily see it.
This could be a false negative, and that there genuinely is less malicious connection attempts. The busy fail2ban are bastion hosts on AWS, while the others are hosted on DigitalOcean. For me as a user though, I wish there was a way to see historically blocked hosts. The last time I looked half a year ago, this was not available in CrowdSec.
https://www.sshguard.net/docs.html
The setting up link points at https://web.archive.org/web/20180901061425/https://www.sshgu... and the original is 404: https://www.sshguard.net/docs/setup/
why not ufw ? Isn't it widely used?
https://www.daemonology.net/blog/2012-08-30-protecting-sshd-...
And, please don't use passwords, instead use certificates. You can easily have system to create and use a short-term valid login certificates. Also, don't forget to authenticate your hosts to users with host certificates.
All of the above is needed even if it is behind wireguard. That's defense in depth. Ensure there is no shared-fate between the two authentication systems.
Also, consider multi-factor personal authentication – biometric+external auth (FIDO2 device with fingerprint auth) and device-auth (device-secure-enclave bound certificates) where device is unlocked by a password/pin from your memory.
If you do these, you don't need silly hacks like SSHGuard.
Defense-in-depth is a good idea, but has its limits. I don’t think there’s too much value to layering SSH over Wireguard.
Agreed on SSHGuard type things not being necessary (for authentication protocol). SSH is not special. People can attempt to bruteforce Wireguard too. In fact, the only difference seems to be SSH making the logs (more easily) accessible.
The main downside to SSH is that it allows in default configuration to use insecure passwords for authentication (but this can be disabled). Wireguard is strictly better since passwords are never an option.
Protocol negotiation is another downside, but I think it’s a smaller one. The SSH devs are fairly good about removing insecure schemes from the defaults.
Future in-the-wild vulnerabilities in the SSH protocol layer?
I think that layering may be useful given a certain threat model, but you shouldn’t just layer protocols without actually considering the probability of vulnerability and considering the other tradeoffs you are making. Otherwise the only correct answer would be to layer SSH over every independent VPN protocol.
Considering the security track record of SSH, I don’t see the value to layering, when on the other side, you now need to manage multiple keys and an entirely separate protocol that isn’t as ubiquitous (yet). The time spent doing this might be better spent building robust and testable auto-update infrastructure or architecting the applications on the server to reduce the risk of contagion given an RCE vulnerability.
Reason to put wireguard as the first line of defense before ssh is because of wireguard's simplicity and robustness vs the track record of ssh implementations.
Why is the tradeoff of adding Wireguard worth it? Sure it’s extra security, but is it really worthy managing multiple keys, not-as-ubiquitous protocol, less flexible administration and no CA? I don’t think there is a correct answer (it depends on the threat model), but your original comment made it seem like there was no tradeoff and there was an obvious correct answer.
This completely silenced our logs.