Fairly standard stuff, definitely interesting to see all the IoT credentials attempred.
I'd recommend SSHGuard over fail2ban though, I seem to remember the version of fail2ban in the Debian repos completely choking on IPv6 and failing open which is obviously undesirable.
My experience with SSHGuard was it treated monitoring events (specifically from monit) as failed attempts and blocked further attempts marking the service as unreachable.
I'm sure this can be configured but fail2ban does not block such attempts.
I feel like fail2ban failing open is a good thing. If it fails closed you can't SSH into the server. If it fails open it's just as though you aren't using fail2ban.
Actually, my biggest fear with fail2ban is it failing and locking me out of a remote server. If it indeed always fails open then there's no reason for me not to use it.
Though, I don't know how it could be guaranteed to always fail open.
> I feel like fail2ban failing open is a good thing. If it fails closed you can't SSH into the server. If it fails open it's just as though you aren't using fail2ban.
But it fails open whilst you think there's a layer of protection there.
> Actually, my biggest fear with fail2ban is it failing and locking me out of a remote server
Most of these tools will let you whitelist an address - and presumably there's usually a way back in via a serial console/IPMI-KVM interface?
I never understood why people would use fail2ban or similar approaches. Just set proper password and move on. SSH traffic is tiny, it's not a problem. You'll never be hacked with a proper password. Let them waste their time.
The failed attempts make it harder to monitor for other attacks because of the noise in log files, network traffic, etc. and if an attacker IP is blocked early they can't try more effective attacks.
Except actual attacks don't show up in logs anyway, so it's still pointless?
The SSH daemon logs when it successfully rejects an access. A successfully rejected access is inconsequential to your security. If you are using secure passwords or pubkey authentication, it will never log a successful login by an attacker. What remains then is exploitation of the SSH server ... but the SSH server doesn't have a code path that logs "I have been exploited".
fwknop has a cache to prevent replay attacks, that plus the fact the packets are only valid for a certain length of time pretty much thwarts this. Remember, fwknop is _in addition_ to your existing security practices, it is not the only "password."
If it's set up properly, it could mean you don't have log files filled with login attempts since the SSH port won't open to them until a successful knock and the knocking doesn't have to be logged.
While port knocking is really just another layer of obscurity, obscurity works really well on scattershot/random attacks. An attacker dedicated to getting into your specific server is another matter but thankfully far more rare.
"While port knocking is really just another layer of obscurity, obscurity works really well on scattershot/random attacks."
Thank you. Remember - security through only obscurity is a bad idea, but additional layers of obscurity can be very valuable.
Also remember: a typical port knock is a series of three ports that all have to be hit in a certain timeframe - for instance, 2000, 4000, 8000 - that's a big "keyspace" to brute force through at WAN packet speeds ...
Ideally, your port-knocking service is much simpler and easier to audit than your ssh service. It's also much more obscure. So you have a service which is very unlikely to suffer a 0-day vulnerability listening all the time to engage a much more vulnerable service only at those exact times you need it.
I originally set up fail2ban because I got fed up with the noise of the hard disk going tick...tick...tick every 15 seconds or so, as another password guessing bot login attempt got logged and the disk heads had to seek as the logfile was written out to disk...
I have SSH with key auth everywhere, so I have no security worries.
But when sshd ends up pegging an entire core, and I can't login myself anymore, then fail2ban (or even whitelisting IP ranges for ssh) becomes necessary.
Keys, like passwords, can be leaked, so it's possible there are databases of leaked keys out in the world, because unlike PGP or SSL certs, ssh keys don't have a revocation mechanism built in. So they could just try all the known leaked keys - like they try known leaked or default passwords.
Then there was this: https://jblevins.org/log/ssh-vulnkey
(tl;dr - there was a bug in debian that caused ssh-keygen to only produce a very small number of keys rather than keys distributed across the whole keyspace). After that became public the bots were all about trying those keys for a few years after. To this day it's recommended to not used keys from that set.
Point being - the bots will try because it the tiny marginal cost is not high enough to stop the attempt.
It has stopped now (I simply removed port 22 from the whitelist except for connections originating with my home ISP, which is a local ISP where abuse reports actually work)
That would be problematic, but I never encountered that. I'm using cheapest VPS with public IP and without any additional protections. I just checked and its load average is 0.00, so load is non-existant.
Do you ever examine logs? Fail2ban (or even better, using a nonstandard port or both) keep the noise out of lots to the point that you can still examine them manually.
Without it, 99.99% of the average ssh log is failed back attempts.
You aren't infallible. While it's easy enough to secure a minimal server with only SSH access, every time you add another service, provide access to a new user, or change a configuration, there's a chance you'll miss something or make a dumb mistake. Running Fail2Ban is a very cheap and easy method to ensure that, even if you do fuck up somewhere, there's another layer of security that should prevent malicious access in most cases (nothing will save you if you set the pass to admin/admin, e.g.).
I called up my provider and paid a little extra for a fixed IP. Really it was just laziness though because in practice my home IP seemed to only change when I restarted the router.
Just looking toward the future. I think bpf is cool but not aimed at production routing that most network techs can approach. nftables fixes so many issues of iptables and is generally viewed as it's replacement. nftables also has a smaller attack surface due to its youth mostly.
So I made the choice to go ahead and focus on learning nft. It's been enjoyable.
Nothing wrong with this, but a strong password for all Unix accounts (or, ideally, only allowing SSH with public key authentication) is essentially all you need. Unless you're using a dumb password, you're far more likely to be compromised by an application or OPSEC flaw (like someone getting into the email you used to register your server hosting account) than some crazy SSH compromise.
This all is easy if you only get a handful of login attempts. But when you get more, and more, it becomes an issue. On one of my servers it got so bad that SSH was at the end constantly using an entire CPU core.
I don’t have logs from that server, but here are logs – just the failed attempts – of 2 months from a server that was less severely affected: https://s3.kuschku.de/public/failed_ssh [327M]
And that was despite already blocking massive areas of IP space already.
Yeah, I now simply removed 0.0.0.0/0 dport 22 from the whitelist and only added a select few IP ranges. That has calmed the storm significantly, and if an IP from this range (my ISP) tries it anyway I can contact my ISP (a small local ISP) to take that down.
Looks like the graph was generated from the Top 10 only. Obviously every item appears exactly once, so they all have the same 10%. The presentation as a graph is useless.
Which is worse then using cat, from a user perspective.
Imagine if you look for something else, you have to point the cursor in the middle, right after grep and before the file. With cat |grep you press Arrow-up and Alt+Backspace and you are right where you want to be, ready to search for something else.
I would find it okay without cat to start the next line with "grep newPattern " and then fetching the previous filename with alt-. (edit: or !$ of course, but alt-. is more practical) -- not saying it's necessarily always better, but knowing your shell better (and/or using a better one) is often a viable alternative to using another construct.
I don't see how the rest is faster than using Arrow-Up and deleting the last word. It seems more complicated. The most important string is still in the middle.
I just use ^...^... or !gre:... and make my edits that way -- it feels much faster and more intuitive than cursoring around, moving my hand over to the arrow keys etc.
The shells were developed on keyboards that didn't have arrow keys and have affordances for them.
You doubt that they keyboards lacked arrow keys? Just look at an ASR-33 (teletype) keyboard or ADM3A. The latter had arrows printed on the hjkl keys but there was no special way to generate “arrow” key presses — those terminals sent 7-bit ASCII characters directly. Look at what # and @ do in cooked mode — THAT goes back to Multics.
The teletype was a printing terminal and the ADM3A was a “glass teletype” — essentially a printing terminal too.
It’s possinle to write a lot of good code this way
I'm not trying to argue; there are not "arrow" characters in the ASCII character set. sh, csh ksh and bash were all developed on ascii terminals. Brian had an Ann Arbor Ambassador which does do 8-bit ASCII but we always had them configured to set the top bit as a "meta" key, so essentially you had 7-bit ascii as well. All these terminals were connected by serial lines to computers (an unusued Vax 750 in the case of the GNU project; that was the machine he developed Bash on -- for the other shells, developed at Bell labs, ksh was also developed also ib a vax while sh and csh were developed on PDP-11s IMHO). You can see the key layouts available back in the days of ASCII terminals here: http://xahlee.info/kbd/vt100_terminal.html and of course use man ascii.
FWIW anyway Brian wouldn't have thought about those keys as the intended use of Readline would have been emacs keystrokes (emacs, or EMACS as it was in those days, had recently been ported to Unix by RMS).
Interestingly our 36-bit machines (PDP-10s and lispms) had arrow glyphs on their custom non-ASCII keyboards (KTV and space cadet) but they were for mathematical notation, not cursor movement!
I’ll add that bash was the first Unix shell to have interactive editing with history, which comes from me suggesting the design of the readline library to Brian Fox around 1985. I was inspired by the editing JSYS (system calls) in TOPS-20 (VMS had something similar I believe) though they lacked the history mechanism.
I've been taught wrong and I've done it wrong for 15 years O_o
Thank you.
It reminds me of the idiom of putting "# -- coding: utf-8 --" at the top of Python 2.x files. Which is incredibly useful, but nobody remembers, and googles everytime.
This comes from the fact the first famous tutorials on Python where written by emacs users, and this editor recognizes this idiom.
But fun fact, the regex matching this line is "^[ \t\v]#.?coding[:=][ \t]*([-_.a-zA-Z0-9]+)" (see https://www.python.org/dev/peps/pep-0263/) and accepts many variations (including a vim format).
The simplest variation is the dumb "# coding: utf8", which is not only straightforward and easy to remember, but way less magical for anybody reading the code.
Bottom line, if you are stuck in 2.x, use "# coding: utf8".
I'm pretty fanatical about efficiency, but it's kind of annoying to have to remember where in the argument sequence for a whole bunch of commands the input file goes.
Of course once you use the 'cat' trick you then have to remember which comments require '-', '-f -' or something to that effect because whoever programmed them didn't think the primary use case was to work as a filter.
Pipes are one of Unix's most useful features and cat somefile | asequenceofusefulcommands is a shorthand that anybody should be able to use.
Typing speed has never been my handicap. To me this seems like just another case of premature optimization, the fact that you are typing in a command is a pretty good indication it is a one-off.
That's just obviously nonsense? Closing the port does not change anything about the attempts. Nor about the success rate of the attempts, if you aren't being an idiot with insecure passwords.
I think the person above was maybe being too imprecise with their language. Obviously it doesn't affect the frequency of portscans and the like, but whitelisting ports is a reasonable approach to further mitigating your risks.
But then, there is no fundamental difference between a service rejecting unauthorized connections and a firewall rejecting unauthorized connections. If your service is already rejecting unauthorized connections, you don't gain anything by also rejecting the same connections at the firewall, and that is why "No open port = no hacking attempts" is ultimately nonsense: It doesn't change anything about the attempts, and chances are it doesn't fundamentally change anything about the rejections either.
Also, adding a VPN exposes the VPN service to the internet, which thus adds attack surface in a different place. Which might be worth it if you need remote access to otherwise vulnerable services. But the simplistic view of "rejecting connections at the firewall" == "no more hacking attempts!11" is just that: simplistic.
That's not how it works. If a vulnerability is found in the service, and you don't need external access, blocking it from the firewall/router essentially blocks the vulnerability. In your scenario, with a public facing service, you will get exploited by a 0 day.
Except that a VPN endpoint is a public facing service, which might get exploited with a 0-day. And that "internal services" more often than not are reachable via HTML email or just web browsers on the inside, which might be used to exploit your "internal service" with a 0-day.
Closing the port does not change anything about the attempts. Nor about the success rate of the attempts, if you aren't being an idiot with insecure passwords.
How about "No open port = no concern about possibly vulnerable services running on open ports"? I actually worry less about passwords and more about overflows, protocol problems and parse errors these days.
Yeah, sure, but (a) then those problems potentially also affect the VPN endpoint, so it's still a trade-off (b) much of the complex protocol machinery often is behind the authentication barrier, so the risk of just exposing the port to the public isn't necessarily that big, and (c) a firewall doesn't necessarily protect your vulnerable service, often there are browsers on the inside that an attacker could use to access those "protected" services.
My point isn't that blocking off ports at the firewall is always pointless, but that it's usually a trade-off, and keeping a vulnerable service running behind the firewall can still be a risk, and many "hacking attempts" are just irrelevant if you follow general best security practices, so it's pointless to do anything specifically to prevent them.
Changing the SSH port proved to be most successful. Yes, of course you can find the port with a port scan. But it keeps the logs clean. And I don't want to waste CPU cycles on some brute-forcing idiots.
Is using non standard ports a reasonable approach? On one hand, it's kind of a security-by-obscurity measure, and it's also a (very minor) inconvenience to real users. However if it's not being used in place of other reasonable security measures, I'm not really sure what's bad about it, but it does feel a bit janky to me.
Same goes for other non-public services such as VPN.
What ports should be used? I think clearly you don't want to use other well known services (eg, running ssh on port 25 is just going to fill your logs with failed connection errors). Should you use ports >1024? >10000?
I’m so tired of hearing people quote “security by obscurity” in relation to moving ssh ports. It is NOT security by obscurity (SBO). SBO is taking an insecure thing and trying to hide it, and relying on the hiddenness as the ONLY means of security. SSH is already highly secure, so hiding it has nothing to do with the security. You can debate whether it’s worth the inconvenience vs reducing log spam, but SBO does not come into it at all.
I typically just pick a high numbered port. > 10k would work.
I see it as a way to reduce the number of attacks, which is helpful, and may end up being more secure, but I wouldn't count on that alone. Against botnet attacks it will likely help, but against someone specifically targeting you, it likely will only slow them down.
A high numbered port (above 1024) can be bound by non root programs. Use some non 22 port below 1024 if you want to change the port. Or use some RBAC system like selinux to keep anyone from binding to 2222 or whatever.
If it's a new server (like your example) there should be no other user already logged in. If it's not new, then you'll get a warning. And focusing on the "shortlived", so the damage is also shortlived...
I can only speak for myself, if other people are ignoring warnings, it's their problem.
The other question is whether other risks are greater; if a high port number is your only choice, or handled by port forwarding on a router, or you are the only user, or high numbers are scanned sufficiently less often, then maybe you still choose one.
This comes up as a point of confusion often, so allow me to clarify:
A non-standard port as the only security mechanism (e.g., an unauthenticated HTTP admin-interface) would indeed be 'security through obscurity', but a non-standard port as an extra mitigation layer on top of existing security mechanisms (e.g., SSH authentication) is 'defense in depth'.
Really, this sounds like a bad practice to bind on non privileged port.
A user can keep running a program to check if your non privileged port can be bound and if it could, it can start any behaving daemon that could be pretending to be a valid SSH daemon but could have anything like key loggers. Even a single restart of sshd could make it happen.
It doesn't have access to the host keys, so it can't do very much unless you intentionally bypass the "someone is intercepting your connection" error your ssh client will give you in that case.
As I understand the argument, it’s “Changing port number add security, therefore it’s a good idea.” I think nobody argues that it adds security. The problem is that:
1. It adds very little security: 16 bits is not much, and the result is not 256 bits (say) of SSH key plus 16 bits equals 272 bits, but instead effectively still 256 bits, or 256+8×10⁻⁷³ bits.
2. The security it adds is itself bad (sent in cleartext, easily brute-forced)
3. These problems stand against the many drawbacks of this previously discussed (complexity, confusion, etc.).
And the final argument: If increased security is what you want, simply increase your key lengths and/or password lengths, and you will get much more than 8×10⁻⁷³ bits of security, without any of the above problems.
You know what it does do though? Dramatically reduces the amount of log spam :). Running SSH on port 22 vs. even just 2222 significantly reduces the number of failed login attempt messages. They'd always fail anyway unless they happened to somehow get ahold of my ssh private key, but it's sure a lot less annoying when looking for something in the logs to not have to filter through all that noise.
You're looking at this the wrong way, because you seem to assume that non-standard ports adds to security the same way the key length does, which is not the case.
I can’t think of any way it could be otherwise, but please, feel free to enlighten us all how a non-standard port number adds any more security than would an additional 16-bit key.
Also means attackers are much easier to spot in the logs. It's also a point of real interest if an attacker took the time to port scan you first in order to detect which port your ssh is on.
Non-standard ports, port knocking and similar things work before anything reaches sshd; so they can mitigate vulnerabilities in sshd. Obviously a non-standard port is the weakest fellow in the bunch, but that does not mean the effect is zero. An extra 16 bit key length would obviously never mitigate a sshd vulnerability.
An entirely different effect has been brought up by others already; a non-standard port is enough to evade ~99 % of all automated attacks, therefore you can run at higher log levels and might notice actual attackers earlier.
> Obviously a non-standard port is the weakest fellow in the bunch, but that does not mean the effect is zero.
Like I said, nobody argues that it adds zero security, but that, for all the hassle it introduces, it adds way too little security to be worth it. 16 bits is nothing, and can be brute forced quickly.
Regarding port knocking: I’m assuming that most people don’t write their own port knocking software, but use something standard. The configuration of any port knocking scheme is equivalent to an additional separate key of length log2(number_of_possible_port_knocking_configurations). What is the effective “key length” of port knocking schemes?
Also remember that the “key” of port knocking, just like a non-standard port number, is transmitted in the clear, so anybody listening to the traffic can see it. (And if you assume that the attacker can’t listen to the traffic, why are you even using SSH instead of something simpler like telnet?)
It's not security by obscurity; obscurity can be a useful component of security. Something that's well protected and hard to find is less likely to be compromised than something that's well protected and easy to find, because some percentage of attackers won't bother to look for it. "Security by obscurity" occurs when someone tries to use obscurity in place of security, by 'hiding' their insecure thing.
Besides, running a service on a non-standard port offers pretty good protection against a fairly common threat scenario in which a new vulnerability is discovered in a well-known service and attackers start sweeping the internet for open ports in order to exploit it.
Not that you should automatically run your services on non-standard ports; there are tradeoffs as others have mentioned. But if you've got to expose something dangerous to the internet at large (and that's an assumption that you should question frequently), then avoiding the standard ports can be a useful tool.
Years ago (actually probably more than 10 years ago now) I actually switched from port 22 to 622 on my server because I wanted to get rid of the noise of people attempting to bruteforce my SSH regularly. It worked perfectly for years but surprisingly I've had a handful of bruteforce attempts on port 622 in the past year or so. Nothing particularly targeted either, always the same typical default users (root, admin, oracle, mysql etc...). I wonder who bothers doing that.
Since I disabled password logging on all accounts and my key is stored on an HSM I'm not too worried about it though.
Someday I'm going to put all the different pieces together and have things like SMTP & SSH auth failures, hosts hitting my spamtraps, bots scanning for open 23/TCP, and so on, to where I end up with one central host collecting all those events and injecting /32 routes for those IPs towards null0 in my core.
> It absolutely shocked me that it is worth the time of these drive-by attacks to try combinations like root/password, root/root, or root/admin. Apparently they have enough successes using those incredibly insecure combinations that it is worth their time.
It is fairly evident to anyone that has looked at the traffic that a honeypot gets that most of the activity is automated.
Most of it is purely botnets, some of it is automated password guessing followed by manual login. I think the amount of people manually trying passwords is insignificantly small.
One thing I've found amusing is you can cut down SSH hacking attempts to almost zero (at the cost of compatibility with legacy systems) by using very progressive encryption. If you force use of ChaCha20/Poly1305, ED25519 and so forth in sshd_config, you'll see these attempts almost disappear. There's probably an observation to be made about just how overwhelmingly many of these attempts come from fairly old software setups.
It's basically a clone of Mozilla's, except enforcing only the very strictest encryption. You can just lift the first 16 lines out of that and merge them into your sshd_config if you don't want to change your existing sshd_config too much.
I did that and it bit back when it prevented Transmit from connecting to my servers, with only a generic "Unable to connect" message popping up. This was fixed fairly recently[0], but it's a reminder that using a restricted set of ciphers may not only prevent illegitimate connections from going through.
I can confirm from the same experience, however, that it prevents most hacking attempts from ever going to the authentication stage.
- Have a stand alone SSH server that is something like a R-Pi[type B] running an OS that gets patches regularly via unattended-upgrades and reboots itself at least once a week.[could also be a minimal VM like AlpineOS if you need Gbps+ line speed]
- Have this R-Pi and your network gear plugged into a UPS that can withstand at least a couple of hours of power outage.
- Use non-standard ports on your perimeter FW for forwarding to the R-Pi.
- Run fail2ban or something similar. Set a bantime of at least an hour[3600 sec] after no more than 5 attempts in 600 sec. This is mostly to discourage anyone who stumbles across your perimeter listening port.[which enough will] Password login will be disabled anyway, but a bot might not check for allowed auth types.[nmap -Pn -p 22 --script ssh-auth-methods <target>]
- No passwords allowed in /etc/ssh/sshd_config[PasswordAuthentication no]. Use public key based authentication.[PubkeyAuthentication yes] Put the public SSH key[id_rsa.pub] of any machine that you wish to authorize into ~/.ssh/authorized_keys on your R-Pi. This way you can authorize and de-authorize remote machines[with particular users] at will.
- add 'AllowUsers xxxx yyyy' to your /etc/ssh/sshd_config
- No root logins allowed![PermitRootLogin no]
- Maybe limit via perimeter or on the R-Pi which source IP ranges are allowed to access to SSH.[whitelisting]
- Periodically review your logs of perimeter FW, R-Pi FW and /var/log/auth.log to see what's going on. It should be pretty quiet, but look at them anyway once a week.
- Sleep well knowing that you have a simple, layered and robust defense strategy that also has power-event survivability.
Others have mentioned port-knocking which is a cool trick but not something that I typically use in an actual daily defense strategy because I'm not sure how much value it really adds.
Don't use your perimeter device to host SSH services. It will not get patches fast enough when vulns come up.
"Others have mentioned port-knocking which is a cool trick but not something that I typically use in an actual daily defense strategy because I'm not sure how much value it really adds."
I don't know how much it adds, but it is non-zero.
The knockd daemon is rock solid[1] and your ssh port traffic goes down to zero (other than your own use).
Port knocking has no place in security by itself but I think it's a wonderful addition to a layered defense - my favorite one, in fact.
I just use the firewall.. if you make more than 4 SSH connections inside of a minute you're blackholed for 5 minutes. Pretty much drops all automated scanners right on the floor while not adding any steps for normal use.
While I completely agree and encourage people to use all of your suggestions, this single step - not being on port 22 - has easily provided the most benefit. When sshd listened on port 22, I regularly saw attacks that would run through thousands of common passwords at high speed[1]. After moving to a random port, it was over a year before I saw another hostile login attempt.
It may be easy to discover the server with a simple port scan, but in practice the attacks seem to focus on the easy targets.
> fail2ban
Fail2ban is great and highly configurable. I could accept having to manually reset occasional false positives, so my fail2ban will immediately (no retries) ban on the first use of anything that is known to be malicious such as any attempt to login as root with a password, asking to login with an account name like [admin, mysql, squid], or any apache access log matching /\/phpMyAdmin/, etc.
[1] Several logins per second over ADSL could use a large fraction of the upload bandwidth.
I wonder why the author set up a dedicated SSH python server which mimics some unix userspace as a honeypot. Why not a VM with a real standard widespread linux distribution to see what attackers actually do? Obviously their codes are not very sophisticated that they stop when "uname" fails with certain arguments.
Because it's much easier to set up? It's not breaking after any session no matter what the attacker does. Every session is isolated if several run concurrently. Getting this with a VM involves quite some effort. Then you need to make sure the attacker cannot use any commands to cause external damage but still make them seem to work. But the latter is also a problem with the simulated environment. It's quite easy to detect if you're not just blindly sending automated keystrokes without checking the feedback.
Objectively there should be no difference between a honeypot VM and an actual "productive" VM. I acknowledge that it is much harder to setup, maintain and supervise a network of honeypots VMs instead of a dedicated isolated honeypot server process which just simulates a system. However, as mentioned in other comments here, there is no gain of knowledge if you don't simulate a whole system but only parts of it.
My servers generally have an IPv4 address and a whole IPv6 subnet. I like to disable SSH except on a specific IPv6 address.
Many botnets just focus on IPv4 and there is a lot more territory to scan on IPv6. I usually couple that with moving to a non-standard port, enforcing modern encryption (ed25519 or ChaCha20, etcc), fail2ban, require SSH keys, etc
I think that by using obviously easy passwords anyone who really is looking for something is going to go to the next target. After all impossible to believe anything of value is behind 1234567 or changeme as a password. Even 'baseball' makes slightly more sense. Besides someone running an ssh server is not exactly mr computer newbie. Maybe a desktop you pick out of the trash might contain something interesting if that was the password.
In a classic CIA honeypot I would imagine the women are carefully matched to the target. If you are "James Bond" you get a better looking woman to try and trap you. If you are "Wallace Shawn" (in looks) you would probably figure out really quickly it's a trap if the woman looked like Angelina Jolie.
How configurable is fail2ban? Is it service based? Am I in danger of locking out a legitimate user like someone who makes 25 ajax requests within the span of 3 seconds? Can I make it work for authenticated HTTP endpoints as well?
I wrote a little utility to keep ssh ports off unless a totp packet is sent to a UDP port. I haven't worked on it much since the initial proof of concept. There are several things that could be done to improve it (open only for the address that sent the packet, symmetric key for safer secret distribution, etc). Shared key would involve encrypting time and only open if the decrypted time matches current.
The totp packet is a full hash of time + shared code so brute force is completely infeasible.
It's currently in python, it would be much better in something like go with daemon monitoring. It is very simple, and it practically eliminates attempts. If it became widely known there would probably be attempts on the udp port. An additional improvement would be randomize the listening port based on the time+hash:
I looked at that before starting and reference it in the blog post (but should probably include a direct link to the software rather than the original concept post and article that include links).
The similarities are they both use a single packet and they are both an improvement on port knocking.
This is a simpler solution (pairing UDP with TOTP). No TCP connection is required. Making it ip address specific would prevent replay attacks (although even without address specific the replay would only be valid for ~30s).
Another benefit is that a port scan will reveal no ports open (because it is a one-way UDP based protocol).
fwknop is definitely a polished solution where this is just a proof of concept for the TOTP/UDP as a shared key knock.
Thank you for the feedback!
EDIT: I updated the post to include a direct link to fwknop rather than just links to the posts/articles about it.
I found fail2ban is necessary but not sufficient. It requires some initial number of failed attempts before the attacker is blocked.
To prevent even those initial attempts, I use multiple layers of defenses such as country based blocking or blacklist based blocking. If an attacker slips through despite all this, I blacklist them manually in /etc/hosts.deny.
185 comments
[ 3.1 ms ] story [ 225 ms ] threadI'd recommend SSHGuard over fail2ban though, I seem to remember the version of fail2ban in the Debian repos completely choking on IPv6 and failing open which is obviously undesirable.
https://linux.die.net/man/8/ip6tables
Actually, my biggest fear with fail2ban is it failing and locking me out of a remote server. If it indeed always fails open then there's no reason for me not to use it.
Though, I don't know how it could be guaranteed to always fail open.
But it fails open whilst you think there's a layer of protection there.
> Actually, my biggest fear with fail2ban is it failing and locking me out of a remote server
Most of these tools will let you whitelist an address - and presumably there's usually a way back in via a serial console/IPMI-KVM interface?
That's pretty easy to test for.
>will let you whitelist an address
I don't have a static IP address.
>usually a way back in via a serial console/IPMI-KVM interface?
I haven't been able to do this at my provider when password login is disabled. Maybe I just don't know the way.
http://denyhosts.sourceforge.net/
Times have changed...
(kgraft, kpatch)
[0] https://unix.stackexchange.com/questions/105553/how-to-provi...
edit:
quick and dirty notes of my setup [1]
[1] https://pastebin.com/yYzSrM61
The SSH daemon logs when it successfully rejects an access. A successfully rejected access is inconsequential to your security. If you are using secure passwords or pubkey authentication, it will never log a successful login by an attacker. What remains then is exploitation of the SSH server ... but the SSH server doesn't have a code path that logs "I have been exploited".
I much prefer restricting port 22 to a few ip and disable passwords
http://www.cipherdyne.org/fwknop/docs/fwknop-tutorial.html#w...
It is only clear to the routers in your traceroute ... third party attackers cannot snoop on this traffic.
While port knocking is really just another layer of obscurity, obscurity works really well on scattershot/random attacks. An attacker dedicated to getting into your specific server is another matter but thankfully far more rare.
Thank you. Remember - security through only obscurity is a bad idea, but additional layers of obscurity can be very valuable.
Also remember: a typical port knock is a series of three ports that all have to be hit in a certain timeframe - for instance, 2000, 4000, 8000 - that's a big "keyspace" to brute force through at WAN packet speeds ...
But when sshd ends up pegging an entire core, and I can't login myself anymore, then fail2ban (or even whitelisting IP ranges for ssh) becomes necessary.
I mean, why would people even try.
Then there was this: https://jblevins.org/log/ssh-vulnkey (tl;dr - there was a bug in debian that caused ssh-keygen to only produce a very small number of keys rather than keys distributed across the whole keyspace). After that became public the bots were all about trying those keys for a few years after. To this day it's recommended to not used keys from that set.
Point being - the bots will try because it the tiny marginal cost is not high enough to stop the attempt.
Without it, 99.99% of the average ssh log is failed back attempts.
nftables instead of iptables
port-knocking
non-standard port
key+pass access/auth
ip whitelist
good logging
ED25519 wherever possible!!!
TalkTalk UK, jumps around a lot; YMMV.
Anyway, nftables should be nicer to administer. More pf-like.
So I made the choice to go ahead and focus on learning nft. It's been enjoyable.
I don’t have logs from that server, but here are logs – just the failed attempts – of 2 months from a server that was less severely affected: https://s3.kuschku.de/public/failed_ssh [327M]
And that was despite already blocking massive areas of IP space already.
PasswordAuthentication no
^ it should return that. The end.
Imagine if you look for something else, you have to point the cursor in the middle, right after grep and before the file. With cat |grep you press Arrow-up and Alt+Backspace and you are right where you want to be, ready to search for something else.
In scripts you can save the extra process though.
I would find it okay without cat to start the next line with "grep newPattern " and then fetching the previous filename with alt-. (edit: or !$ of course, but alt-. is more practical) -- not saying it's necessarily always better, but knowing your shell better (and/or using a better one) is often a viable alternative to using another construct.
Also, than* (sorry)
I don't see how the rest is faster than using Arrow-Up and deleting the last word. It seems more complicated. The most important string is still in the middle.
The shells were developed on keyboards that didn't have arrow keys and have affordances for them.
I doubt the later one.
The teletype was a printing terminal and the ADM3A was a “glass teletype” — essentially a printing terminal too.
It’s possinle to write a lot of good code this way
FWIW anyway Brian wouldn't have thought about those keys as the intended use of Readline would have been emacs keystrokes (emacs, or EMACS as it was in those days, had recently been ported to Unix by RMS).
Interestingly our 36-bit machines (PDP-10s and lispms) had arrow glyphs on their custom non-ASCII keyboards (KTV and space cadet) but they were for mathematical notation, not cursor movement!
Thank you.
It reminds me of the idiom of putting "# -- coding: utf-8 --" at the top of Python 2.x files. Which is incredibly useful, but nobody remembers, and googles everytime.
This comes from the fact the first famous tutorials on Python where written by emacs users, and this editor recognizes this idiom.
But fun fact, the regex matching this line is "^[ \t\v]#.?coding[:=][ \t]*([-_.a-zA-Z0-9]+)" (see https://www.python.org/dev/peps/pep-0263/) and accepts many variations (including a vim format).
The simplest variation is the dumb "# coding: utf8", which is not only straightforward and easy to remember, but way less magical for anybody reading the code.
Bottom line, if you are stuck in 2.x, use "# coding: utf8".
Of course once you use the 'cat' trick you then have to remember which comments require '-', '-f -' or something to that effect because whoever programmed them didn't think the primary use case was to work as a filter.
Pipes are one of Unix's most useful features and cat somefile | asequenceofusefulcommands is a shorthand that anybody should be able to use.
Why is it that paranoid nerds need to waste so much time masturbating over failed SSH login attempts?
All you need to do is turn off password auth, or perhaps use wireguard.
No open port = no hacking attempts.
Investing a week of on-and-off studying and tinkering with openvpn is really paying off.
That's just obviously nonsense? Closing the port does not change anything about the attempts. Nor about the success rate of the attempts, if you aren't being an idiot with insecure passwords.
But then, there is no fundamental difference between a service rejecting unauthorized connections and a firewall rejecting unauthorized connections. If your service is already rejecting unauthorized connections, you don't gain anything by also rejecting the same connections at the firewall, and that is why "No open port = no hacking attempts" is ultimately nonsense: It doesn't change anything about the attempts, and chances are it doesn't fundamentally change anything about the rejections either.
Also, adding a VPN exposes the VPN service to the internet, which thus adds attack surface in a different place. Which might be worth it if you need remote access to otherwise vulnerable services. But the simplistic view of "rejecting connections at the firewall" == "no more hacking attempts!11" is just that: simplistic.
How about "No open port = no concern about possibly vulnerable services running on open ports"? I actually worry less about passwords and more about overflows, protocol problems and parse errors these days.
My point isn't that blocking off ports at the firewall is always pointless, but that it's usually a trade-off, and keeping a vulnerable service running behind the firewall can still be a risk, and many "hacking attempts" are just irrelevant if you follow general best security practices, so it's pointless to do anything specifically to prevent them.
Is using non standard ports a reasonable approach? On one hand, it's kind of a security-by-obscurity measure, and it's also a (very minor) inconvenience to real users. However if it's not being used in place of other reasonable security measures, I'm not really sure what's bad about it, but it does feel a bit janky to me.
Same goes for other non-public services such as VPN.
What ports should be used? I think clearly you don't want to use other well known services (eg, running ssh on port 25 is just going to fill your logs with failed connection errors). Should you use ports >1024? >10000?
I see it as a way to reduce the number of attacks, which is helpful, and may end up being more secure, but I wouldn't count on that alone. Against botnet attacks it will likely help, but against someone specifically targeting you, it likely will only slow them down.
Basically it boils down to 'why not?' for me.
It's a game of percentages.
Again, percentages. With enough attempts, some will succeed.
I can only speak for myself, if other people are ignoring warnings, it's their problem.
I believe that's been adequately addressed.
Restricting the remote IP range is another option.
This comes up as a point of confusion often, so allow me to clarify:
A non-standard port as the only security mechanism (e.g., an unauthenticated HTTP admin-interface) would indeed be 'security through obscurity', but a non-standard port as an extra mitigation layer on top of existing security mechanisms (e.g., SSH authentication) is 'defense in depth'.
A user can keep running a program to check if your non privileged port can be bound and if it could, it can start any behaving daemon that could be pretending to be a valid SSH daemon but could have anything like key loggers. Even a single restart of sshd could make it happen.
You say it yourself:
> also a (very minor) inconvenience to real users
I would disagree with the “very minor” part. To paraphrase myself (https://news.ycombinator.com/item?id=6617312):
As I understand the argument, it’s “Changing port number add security, therefore it’s a good idea.” I think nobody argues that it adds security. The problem is that:
1. It adds very little security: 16 bits is not much, and the result is not 256 bits (say) of SSH key plus 16 bits equals 272 bits, but instead effectively still 256 bits, or 256+8×10⁻⁷³ bits.
2. The security it adds is itself bad (sent in cleartext, easily brute-forced)
3. These problems stand against the many drawbacks of this previously discussed (complexity, confusion, etc.).
And the final argument: If increased security is what you want, simply increase your key lengths and/or password lengths, and you will get much more than 8×10⁻⁷³ bits of security, without any of the above problems.
Technical protection is likely limited, but then, admin attention is also a limited resource.
An entirely different effect has been brought up by others already; a non-standard port is enough to evade ~99 % of all automated attacks, therefore you can run at higher log levels and might notice actual attackers earlier.
Like I said, nobody argues that it adds zero security, but that, for all the hassle it introduces, it adds way too little security to be worth it. 16 bits is nothing, and can be brute forced quickly.
Regarding port knocking: I’m assuming that most people don’t write their own port knocking software, but use something standard. The configuration of any port knocking scheme is equivalent to an additional separate key of length log2(number_of_possible_port_knocking_configurations). What is the effective “key length” of port knocking schemes?
Also remember that the “key” of port knocking, just like a non-standard port number, is transmitted in the clear, so anybody listening to the traffic can see it. (And if you assume that the attacker can’t listen to the traffic, why are you even using SSH instead of something simpler like telnet?)
Besides, running a service on a non-standard port offers pretty good protection against a fairly common threat scenario in which a new vulnerability is discovered in a well-known service and attackers start sweeping the internet for open ports in order to exploit it.
Not that you should automatically run your services on non-standard ports; there are tradeoffs as others have mentioned. But if you've got to expose something dangerous to the internet at large (and that's an assumption that you should question frequently), then avoiding the standard ports can be a useful tool.
Since I disabled password logging on all accounts and my key is stored on an HSM I'm not too worried about it though.
Somebody that added nmap to their bot script.
It's a big quiet void out there in sixspace.
https://www.blockedservers.com/
Super useful.
It is fairly evident to anyone that has looked at the traffic that a honeypot gets that most of the activity is automated.
Most of it is purely botnets, some of it is automated password guessing followed by manual login. I think the amount of people manually trying passwords is insignificantly small.
[1] https://infosec.mozilla.org/guidelines/openssh#Modern
It's basically a clone of Mozilla's, except enforcing only the very strictest encryption. You can just lift the first 16 lines out of that and merge them into your sshd_config if you don't want to change your existing sshd_config too much.
I can confirm from the same experience, however, that it prevents most hacking attempts from ever going to the authentication stage.
[0] https://library.panic.com/releasenotes/transmit5/#5-1
- Have a stand alone SSH server that is something like a R-Pi[type B] running an OS that gets patches regularly via unattended-upgrades and reboots itself at least once a week.[could also be a minimal VM like AlpineOS if you need Gbps+ line speed]
- Have this R-Pi and your network gear plugged into a UPS that can withstand at least a couple of hours of power outage.
- Use non-standard ports on your perimeter FW for forwarding to the R-Pi.
- Run fail2ban or something similar. Set a bantime of at least an hour[3600 sec] after no more than 5 attempts in 600 sec. This is mostly to discourage anyone who stumbles across your perimeter listening port.[which enough will] Password login will be disabled anyway, but a bot might not check for allowed auth types.[nmap -Pn -p 22 --script ssh-auth-methods <target>]
- No passwords allowed in /etc/ssh/sshd_config[PasswordAuthentication no]. Use public key based authentication.[PubkeyAuthentication yes] Put the public SSH key[id_rsa.pub] of any machine that you wish to authorize into ~/.ssh/authorized_keys on your R-Pi. This way you can authorize and de-authorize remote machines[with particular users] at will.
- add 'AllowUsers xxxx yyyy' to your /etc/ssh/sshd_config
- No root logins allowed![PermitRootLogin no]
- Maybe limit via perimeter or on the R-Pi which source IP ranges are allowed to access to SSH.[whitelisting]
- Periodically review your logs of perimeter FW, R-Pi FW and /var/log/auth.log to see what's going on. It should be pretty quiet, but look at them anyway once a week.
- Sleep well knowing that you have a simple, layered and robust defense strategy that also has power-event survivability.
Others have mentioned port-knocking which is a cool trick but not something that I typically use in an actual daily defense strategy because I'm not sure how much value it really adds.
Don't use your perimeter device to host SSH services. It will not get patches fast enough when vulns come up.
I don't know how much it adds, but it is non-zero.
The knockd daemon is rock solid[1] and your ssh port traffic goes down to zero (other than your own use).
Port knocking has no place in security by itself but I think it's a wonderful addition to a layered defense - my favorite one, in fact.
While I completely agree and encourage people to use all of your suggestions, this single step - not being on port 22 - has easily provided the most benefit. When sshd listened on port 22, I regularly saw attacks that would run through thousands of common passwords at high speed[1]. After moving to a random port, it was over a year before I saw another hostile login attempt.
It may be easy to discover the server with a simple port scan, but in practice the attacks seem to focus on the easy targets.
> fail2ban
Fail2ban is great and highly configurable. I could accept having to manually reset occasional false positives, so my fail2ban will immediately (no retries) ban on the first use of anything that is known to be malicious such as any attempt to login as root with a password, asking to login with an account name like [admin, mysql, squid], or any apache access log matching /\/phpMyAdmin/, etc.
[1] Several logins per second over ADSL could use a large fraction of the upload bandwidth.
Many botnets just focus on IPv4 and there is a lot more territory to scan on IPv6. I usually couple that with moving to a non-standard port, enforcing modern encryption (ed25519 or ChaCha20, etcc), fail2ban, require SSH keys, etc
In a classic CIA honeypot I would imagine the women are carefully matched to the target. If you are "James Bond" you get a better looking woman to try and trap you. If you are "Wallace Shawn" (in looks) you would probably figure out really quickly it's a trap if the woman looked like Angelina Jolie.
https://livesshattack.net
The totp packet is a full hash of time + shared code so brute force is completely infeasible.
It's currently in python, it would be much better in something like go with daemon monitoring. It is very simple, and it practically eliminates attempts. If it became widely known there would probably be attempts on the udp port. An additional improvement would be randomize the listening port based on the time+hash:
https://dhj.io/posts/2017/05/16/totp-knock/
http://www.cipherdyne.org/fwknop/
I looked at that before starting and reference it in the blog post (but should probably include a direct link to the software rather than the original concept post and article that include links).
The similarities are they both use a single packet and they are both an improvement on port knocking.
This is a simpler solution (pairing UDP with TOTP). No TCP connection is required. Making it ip address specific would prevent replay attacks (although even without address specific the replay would only be valid for ~30s).
Another benefit is that a port scan will reveal no ports open (because it is a one-way UDP based protocol).
fwknop is definitely a polished solution where this is just a proof of concept for the TOTP/UDP as a shared key knock.
Thank you for the feedback!
EDIT: I updated the post to include a direct link to fwknop rather than just links to the posts/articles about it.
To prevent even those initial attempts, I use multiple layers of defenses such as country based blocking or blacklist based blocking. If an attacker slips through despite all this, I blacklist them manually in /etc/hosts.deny.
If there is any interest, my scripts are in https://github.com/KamarajuKusumanchi/hosts.deny/