Put a personal VPS (no domain) online today. Took like 10 seconds for the first ssh login attempts to start. It's toxic out there. I put everything behind wireguard these days.
I've had a dedicated server pwned before I even logged in for the first time. Somehow the attackers must have guessed the (purportedly random) initial root password, or exploited a vulnerability in the somewhat outdated OS image.
It was in South Korea. I don't want to blame the host because they were very quick to acknowledge the problem, which is something even well-known companies with worldwide presence tend not to be good at. They wiped the server and reinstalled a clean OS, free of charge, within hours.
If there isn't already, there needs to be something like the "Survivability Onion" from the Military[1], which has the first layer of the onion being: Don't Be There.
For hosting, one of those layers is going to be: Don't expose ports publicly, if you don't want people connecting to them.
Unless you're hosting a public SSH service, you shouldn't expose SSH on a public network.
It protects the VPS from a few avenues of attack:
- SSH vulnerabilities (either current, or because the server has not been updated regularly)
- SSH server misconfiguration
- Credentials leaked/stolen
- Denial of Service (hammering SSH in an effort to either consume resources on the host, and make it harder to fix/patch an issue, or to force your hosting provider to block SSH)
OpenSSH is probably one of the most audited pieces of software running on your system.
> SSH server misconfiguration
Just like with TLS use tools to validate your config (like https://www.ssh-audit.com/) and you'll be mostly safe.
> Denial of Service
Usually it's easier to DoS whatever other service your server is running, like a web backend och mail server or similar.
---
On the other hand you now need to manage two keys, the wireguard one and the SSH one, and you still need to expose your wireguard server (which is not necessarily more secure than OpenSSH).
Just disable SSH password auth, run one of the ssh audit tools like the one I linked above and use hardware keys like the openssh yubikey feature (optional but nice to know that the key can't be cloned).
If you really don't want people trying to ssh to you move it to another port or use port-knocking.
On the SSH points you make before the break - I don't particularly disagree on any part.
But you still have to maintain it, and vulnerabilities do still get found.
---
As for exposing Wireguard, that's not necessarily true - You can have your server dial-out to another host instead.
Additionally, because Wireguard operates on UDP, and doesn't respond unless you pass the right credentials - it's very hard to detect from a cursory scan. You would have to already know what IP and port that Wireguard was listening on.
SSH because it's TCP based can be detected by any random scanner, regardless of it's port. Unless you're using some conditional firewalling, such as like with port-knocking.
Seems like you're just trading one attack surface for another. Your "port" is exposed to the internet either way, you just replaced the service which is listening to it. The question is which you trust more - sshd or VPN.
As mentioned, no, not necessarily. Wireguard is much harder to detect through a network scan, as it only listens on UDP and you have to pass valid credentials first.
Yeah I've been happy with setting DHCP to internally resolve subdomains to local ip addresses, combined with a lets encrypt wildcard cert created with a DNS challenge. Everything secured behind WireGuard and you never for a second need to make your web services public.
I'd added a new hostname to a long-existing domain. Then I added that hostname as a new virtual host to a Caddy server I've been running for a long time. The requests were to that vhost, i.e. using the `Host: my-new-host.example.com` header, not just running `curl http://1.2.3.4`. They were asking for the brand-new host by name.
After hashing it out with some friends on Mastodon, I think it's most likely because Caddy acquired a Let's Encrypt cert, those certs are logged[0], and attackers pounce on new hosts as soon as they're in the logs.
Yes, this is a downside of certificate transparency. Also, using Let’s Encrypt certs for internal hosts, as some do for convenience, means you’re publishing your internal hostnames. Use an internal CA if that worries you.
If you care, you can use a wildcard cert. E.g. instead of getting a cert for foo.example.com, get one for *.foo.example.com, and then use an obscure subdomain. AFAIK, nobody is bruteforcing subdomains on wildcard certs.
Are they brute forcing them, or are they relying on the fact that most users of wildcard certs also have wildcard DNS entries, so it's just that everything is actually available. Wildcard DNS pointing to a web server with a wildcard cert will pass through to the web server, and at that point the web server responds with a 404 if the host header isn't one configured for virtual hosts.
Didn't the wildcard cert still need the alt-names specifying the host for the browser to accept the certificate as fully valid? I remember being annoyed by warnings until I added them a decade ago while playing around with a private CA at a previous job
Great point. This was no big deal for me today. I was prepared for it. I mainly posted this to warn others: if you put a server online, it will be attacked, and it's going to be way sooner than you would have believed. You can't get away with, "yeah, I know I left some extra files laying around, but I'll clean them up and lock everything down as soon as I'm finished with the setup." Nope. That's not an option.
Nothing is impossible, but just having an SSH connection active to a system does not allow the possibility to tunnel backwards through that SSH connection. Unless the person who initiated the connection already configured it for reverse connections.
For non-interactive scripts there probably isn't much you can do.
I've always wanted to setup an interactive honeypot ssh server which prints the standard message for a rejected login and then pretends to close the connection, while running a keylogger to hopefully intercept some command which asks for password. The major obstacle is that people customize their prompts, although many probably use whatever is set by default on Ubuntu.
Back when long-distance calls cost real money per minute, yes. You got the occasional telemarketing call, but it was local and in most areas there wasn't much of it happening.
My humble opinion is that it’s inevitable due to inequality. There is money to be made doing this type of thing. Never underestimate the time-rich, money-poor.
Presumably the former caused your new hostname to be published for all of the Internet to see? That doesn't sound like obscurity to me.
I've had a service running on a high port for many years at the same IP. I've seen it get the occasional "knock" from some scanner or bot, but it has been generally quiet. It probably also depends on your IP, as some parts of the Internet are likely scanned far more frequently and aggressively than others.
Having a webserver listening on ipv4 may get something like that sooner or later. All the tine you get vulnerability scanning by different actors and goals, be aware of that or not.
With hostnames is just another layer, that may have more requirements, but the motivations are similar.
What is missing in our society to actually deal with this? I see stories like this and get the impression that it is a forgone conclusion that it is OK for this to be normal. I realize the author isn't saying that, and I appreciate their warning and think it is great they posted this, but where is the call to action to deal with this? And I don't mean we should all just get stronger safeguards. The people doing this type of thing are causing harm on a large scale so how do we get society to recognize and start really caring about this to the point that it wouldn't be a joke to call some law enforcement agency to tell them you are being hacked?
There's nothing missing from society. This is the natural state of humanity, and one always must be vigilant. The only thing keeping thousands of people from trying every home's physical front door every 5 seconds to see if it's unlocked is that it's not scalable. If it were possible, it would be happening. AND, if it did start happening, it would not be a law enforcement problem unless law enforcement became equally scalable.
This is not a popular opinion and it often gets dunked on reflexively, but I truly believe the default is low-trust. We are one evolutionary step away from chimpanzees, with barely enough "society" grafted on to our world, which mostly doesn't function. You need to assume you are under potential attack at all times and be prepared to defend against it. Whether it be a physical attack, social engineering, scams, advertisements... in cyber-space and in meat-space. Any other security posture, long term will eventually open you up to a breach by an adversary.
I VERY strongly disagree with everything you wrote.
I think it's a tiny minority of people doing this, with the Internet amplifying their reach.
This world runs on trust. For example it's extremely easy to shoplift, yet the vast majority of people don't. Because people are not like that. It's easy to forge things, yet most people don't, it's easy to snatch and grab random objects from people or places, yet, people don't.
The next time you are out, think like a criminal and see just how much stuff you would be able to do, yet you don't, and neither does anyone else.
It's a tiny minority who do abuse that trust.
(And we are not "one evolutionary step", we are several orders of magnitude steps away from chimps. In intelligence chimps are more similar to dogs than they are to humans, chips are smart - for animals. In comparison to humans they just aren't.)
It costs time and money to run those scans, and people wouldn’t be doing it if it wasn’t profitable. I don’t want to blame the victims, but the root is that people put insecure setups online frequently enough to keep the criminals interested. We could require a license or certification to launch a server but that’d be even worse.
I don’t have an answer. This normal isn’t OK. I don’t know what to do about it though.
When I said something like this a few weeks ago, some people replied "well they didn't actually break in, so they didn't do anything against the law". Even when they do, and get shells for miners, ransomware or fraud they say "it's the victim's fault". If there is no moral consideration, that may be true.
I suspect this is one of the overall effects that have come about on account of individuals judging the inequity across the world.
How would you propose we deal with within this exactly? Is there a clear way to define a law that would make this activity illegal? Or something you'd propose other than legal recourse?
Personally I don't really know how to draw a clear line between acceptable and unacceptable here. Sure we can assume intent when requesting a users API from a standard WordPress install or plugin, but can we really say what the person's intent is? There are white hat hackers after all, and a law blocking this would almost certainly stop anyone interested in discovering these vulnerabilities and making them known before someone with bad intent finds them.
I think there are multiple levels here. I agree the law side is a challenge, right now, but as we see new harm arise from new technologies we need to start figuring out those lines and figuring out how as a society we discourage this behavior. Basically, I think this is an example of technology moving faster than society's ability to assimilate it and we need to think of smart ways to catch up. That isn't a call to stop technology, but I do think we need to start growing social antibodies to this type of thing. Consider a comparison to kids vandalizing things. As a society we frown on vandalism even without laws in place forbidding it. At many turns in society kids are discouraged from doing it and most, but not all, grow up knowing they shouldn't do it. Social pressure is there to help keep it in check and that also leads to a better understanding of 'the line' which helps inform more formal things like laws and punishments. Laws, and law enforcement, should come after society has taken notice and started acting so how do we get society noticing and acting? What should society do? I think it is, possibly, partly things like emphasizing the idea that it isn't 'cool' to hack things, which every movie in existence does. I admit that isn't a strong start to the idea thread, but there may be things along those lines that could help.
You and I definitely have the same moral view of hacking, or "offensive hacking" if blackhat hacking isn't a term today?
I'm always very hesitant to see any top-down changes trying to change public sentiment though, whether by some kind of intervention to change norms or through law. Norms and laws should reflect, rather than coerce, public opinion. If a majority of people would, of their own accord, agree that hitting common API endpoints on servers publicly connected to the internet is too dangerous to be allowed, I guess we could try to draw a line between that and hacking or penetration testing with good intent.
Short of that though, is it really our place to convince people that this is wrong? Sure we can absolutely do a better job of raising awareness of what's going on, but at least to me awareness of the facts should be the extent of it. People can make up their own decisions on whether or not something is okay, or if its bad enough that we need to further empower the government to enforce more laws.
The challenge with federal law enforcement is they respond to pressure from elected officials, who are beholden to the lobbyists who pay for reelection campaigns.
The FBI is responsible for super serious and intrastate crimes like terrorism, kidnappings and trafficking. Those resources get redirected to policing copyright violations at the behest of the MPAA and other IP orgs.
ICE resources are redirected from border safety to protecting profit margins impacted by knockoff goods.
Whatever entity gets setup to address your concerns will eventually be captured by major corporate interests.
Crime happens because it is profitable. As far as there is a good chance of winning from hacking, people will hack. So, I'm on the side of good security defaults and practices so that one should invest in crime more than they are likely to earn from it. Harsher punishments won't help because the ability to enforce those punishments is what matters.
same for DDoS. running a service on the Internet means you'll randomly be subject to DDoS attack. unfortunate, but that's just the state of the world these days.
These days certificate transparency logs seems to be a trigger to immediately get slammed with a ton of scanners. Some “legit” commercial (eg paloaltonetworks) some russian/chinese
Yes it would but we dont like to talk about it =) Although if enough people start doing it my guess is the scanners will begin knocking on more than just :22,80,443
Some of these I get and are obvious. Some look like some kind of exploit maybe?
And then there’s .DS_Store.
What’s the point of that? In case you find a Mac to launch more targeted attacks against known bugs? To know if the developer is in a Mac and just copied files without filtering out dot files?
I was working on bug bounty work a few years ago for a large company with lots of subdomains in different parts of the world. I found .git/config on a server and I was able to partially reconstruct the entire git repository.
This led me to paths that I wouldn't otherwise be able to find and a complete server takeover through remote code execution. One of their developers left test code for a website template and an unrestricted file upload form.
Port knocking may be a first line defense here with a port scan attack detector to ban IPs that try to find such ports. See Linux knockd and psad for references. This obscurity doesn’t protect again man-in-the-middle but at least protects from unwanted and opportunistic guests. It also gives more time to indirectly protect from 0-day on sshd (aka the fiasco that could have been the xz incident).
I always wondered who is behind these attacks, they don't seem targeted since they do them on random ips.
I did a bunch of Devops a few years ago on a Startup, and whenever i started a new AWS EC2 instance, i started getting request for Wordpress files, and other common CMS files.
80 comments
[ 0.27 ms ] story [ 49.0 ms ] threadWhat do you mean? You don't connect to ssh in the VPS directly, but through a VPN? How does this protects the VPS from attacks?
Exposing wireguard to the world isn't much better than exposing ssh
For hosting, one of those layers is going to be: Don't expose ports publicly, if you don't want people connecting to them.
Unless you're hosting a public SSH service, you shouldn't expose SSH on a public network.
It protects the VPS from a few avenues of attack:
- SSH vulnerabilities (either current, or because the server has not been updated regularly) - SSH server misconfiguration - Credentials leaked/stolen - Denial of Service (hammering SSH in an effort to either consume resources on the host, and make it harder to fix/patch an issue, or to force your hosting provider to block SSH)
[1] https://en.wikipedia.org/wiki/Survivability#Military (at the bottom of the section)
OpenSSH is probably one of the most audited pieces of software running on your system.
> SSH server misconfiguration
Just like with TLS use tools to validate your config (like https://www.ssh-audit.com/) and you'll be mostly safe.
> Denial of Service
Usually it's easier to DoS whatever other service your server is running, like a web backend och mail server or similar.
---
On the other hand you now need to manage two keys, the wireguard one and the SSH one, and you still need to expose your wireguard server (which is not necessarily more secure than OpenSSH).
Just disable SSH password auth, run one of the ssh audit tools like the one I linked above and use hardware keys like the openssh yubikey feature (optional but nice to know that the key can't be cloned).
If you really don't want people trying to ssh to you move it to another port or use port-knocking.
But you still have to maintain it, and vulnerabilities do still get found.
---
As for exposing Wireguard, that's not necessarily true - You can have your server dial-out to another host instead.
Additionally, because Wireguard operates on UDP, and doesn't respond unless you pass the right credentials - it's very hard to detect from a cursory scan. You would have to already know what IP and port that Wireguard was listening on.
SSH because it's TCP based can be detected by any random scanner, regardless of it's port. Unless you're using some conditional firewalling, such as like with port-knocking.
I'd added a new hostname to a long-existing domain. Then I added that hostname as a new virtual host to a Caddy server I've been running for a long time. The requests were to that vhost, i.e. using the `Host: my-new-host.example.com` header, not just running `curl http://1.2.3.4`. They were asking for the brand-new host by name.
After hashing it out with some friends on Mastodon, I think it's most likely because Caddy acquired a Let's Encrypt cert, those certs are logged[0], and attackers pounce on new hosts as soon as they're in the logs.
[0]https://letsencrypt.org/docs/ct-logs/
Would like to learn more though.
I've always wanted to setup an interactive honeypot ssh server which prints the standard message for a rejected login and then pretends to close the connection, while running a keylogger to hopefully intercept some command which asks for password. The major obstacle is that people customize their prompts, although many probably use whatever is set by default on Ubuntu.
was there some alternative development path or is this inherent in the physical network design?
If it cost some amount of money to connect to a server, all of this would stop almost immediately.
And trying to attach a price to a transaction that makes it unprofitable for bad actors but not be a burden on legitimate users is intractable.
Even if ISPs quickly shut this stuff down the existence of botnets would allow it to continue.
It’s like spam. Just tragedy of the commons.
If you’re relying on obscurity
Presumably the former caused your new hostname to be published for all of the Internet to see? That doesn't sound like obscurity to me.
I've had a service running on a high port for many years at the same IP. I've seen it get the occasional "knock" from some scanner or bot, but it has been generally quiet. It probably also depends on your IP, as some parts of the Internet are likely scanned far more frequently and aggressively than others.
> having an all purpose 1000kloc http server. > serving your source code root > not using any permissions system
With hostnames is just another layer, that may have more requirements, but the motivations are similar.
This is not a popular opinion and it often gets dunked on reflexively, but I truly believe the default is low-trust. We are one evolutionary step away from chimpanzees, with barely enough "society" grafted on to our world, which mostly doesn't function. You need to assume you are under potential attack at all times and be prepared to defend against it. Whether it be a physical attack, social engineering, scams, advertisements... in cyber-space and in meat-space. Any other security posture, long term will eventually open you up to a breach by an adversary.
I think it's a tiny minority of people doing this, with the Internet amplifying their reach.
This world runs on trust. For example it's extremely easy to shoplift, yet the vast majority of people don't. Because people are not like that. It's easy to forge things, yet most people don't, it's easy to snatch and grab random objects from people or places, yet, people don't.
The next time you are out, think like a criminal and see just how much stuff you would be able to do, yet you don't, and neither does anyone else.
It's a tiny minority who do abuse that trust.
(And we are not "one evolutionary step", we are several orders of magnitude steps away from chimps. In intelligence chimps are more similar to dogs than they are to humans, chips are smart - for animals. In comparison to humans they just aren't.)
And IMHO it's better that way, because the alternative --- which some seem to be slowly encouraging us towards --- is dystopia.
I don’t have an answer. This normal isn’t OK. I don’t know what to do about it though.
I suspect this is one of the overall effects that have come about on account of individuals judging the inequity across the world.
Personally I don't really know how to draw a clear line between acceptable and unacceptable here. Sure we can assume intent when requesting a users API from a standard WordPress install or plugin, but can we really say what the person's intent is? There are white hat hackers after all, and a law blocking this would almost certainly stop anyone interested in discovering these vulnerabilities and making them known before someone with bad intent finds them.
I'm always very hesitant to see any top-down changes trying to change public sentiment though, whether by some kind of intervention to change norms or through law. Norms and laws should reflect, rather than coerce, public opinion. If a majority of people would, of their own accord, agree that hitting common API endpoints on servers publicly connected to the internet is too dangerous to be allowed, I guess we could try to draw a line between that and hacking or penetration testing with good intent.
Short of that though, is it really our place to convince people that this is wrong? Sure we can absolutely do a better job of raising awareness of what's going on, but at least to me awareness of the facts should be the extent of it. People can make up their own decisions on whether or not something is okay, or if its bad enough that we need to further empower the government to enforce more laws.
The FBI is responsible for super serious and intrastate crimes like terrorism, kidnappings and trafficking. Those resources get redirected to policing copyright violations at the behest of the MPAA and other IP orgs.
ICE resources are redirected from border safety to protecting profit margins impacted by knockoff goods.
Whatever entity gets setup to address your concerns will eventually be captured by major corporate interests.
This shouldn't be hard. If we can't fix that then good luck tracking down bad actors on the interwebs
an authoritarian global government and cameras you aren't allowed to turn off in every room of your home
And then there’s .DS_Store.
What’s the point of that? In case you find a Mac to launch more targeted attacks against known bugs? To know if the developer is in a Mac and just copied files without filtering out dot files?
This led me to paths that I wouldn't otherwise be able to find and a complete server takeover through remote code execution. One of their developers left test code for a website template and an unrestricted file upload form.
(this is not a prejudice; >80% of auto-bans my servers are issuing are for Tencent IPs. I should grab some exact numbers at some point.)
I did a bunch of Devops a few years ago on a Startup, and whenever i started a new AWS EC2 instance, i started getting request for Wordpress files, and other common CMS files.