58 comments

[ 7.1 ms ] story [ 129 ms ] thread
Bastion host(s) are a useful and important component of a system management infrastructure. A bastion-host, in this context, is actually more properly, but more obscurely, called a jump server. In this post I will simply use the term bastion host. It is the most commonly used term for the system's function: a server, which has undergone security hardening steps, that is the operational and administrative control point for systems and hosts in a datacenter (or AWS Region).
Direct quote from the page at the other end of this link :

"You appear to be using a browser that may not be able to display our site correctly. Generally, this is beause it's an older browser that doesn't support a lot of the newer, awesome features in open Web standards.

Rather than provide an untested and possibly subpar experience of our site, may we suggest downloading one of these fine browsers? (We think you'll find your overall experience of the Web will improve if you do.)"

This is followed with links for Firefox and Chrome. I am running Firefox 24.4 from CentOS repos. This is unskippable, and there's no simple way to bypass it on the page. On a browser from March of this year.

Urgh.

Yeah. Welcome to the world of an HTML 'standard' that is 'living'.
Since it's only a plaintext page, you can read it with curl. Or you could, except the server is broken and sends back gzipped data even if the client doesn't request it.

    curl https://www.pandastrike.com/posts/20141113-bastion-hosts | gunzip | more
I shouldn't have to grab a page curl to see a website. A disclaimer of "Hey we notice $X is your browser, maybe switch to those?" with a "No thanks, I'll deal with it" option on some modal JS dialog.
(comment deleted)
TBH, > 6 months old is hopelessly outdated. Welcome to the future, grampa.
Some of us are using browsers imposed upon us by $EMPLOYER, and may not be able to update of our own volition.
I think Firefox 24 is still supported for a few more weeks, as it is a "long term" release.
The point of those LTS browsers is so that an employer doesn't have to watch for UI changes in their browser every 6 weeks, and then retrain all their staff when the browser moves things around or makes things look different.
Jump boxes are a good idea. A better idea, if you're in AWS, is to use a VPC and just VPN into your environment; the VPN terminator in the VPC serves the same purpose as a jump box.

If OpenVPN scares you, you can still do layer-4 filtering to minimize the impact of losing the OpenVPN server (ie, to get it to "no worse than if we hadn't used VPCs in the first place).

"Single-packet authentication" is, I think, silly, and I've never recommended it to anyone. It's hard to think of a case where hand-rolled authentication in front of SSH has saved anyone with a proper SSH configuration (keys-only-no-passwords). You can skip that step.

Do you have a good reference to administering an OpenVPN server? Setting it up is easy, setting it up with HA is also no problem, but administrating OpenVPN for a startup that doesn't have a dedicated sysadmin would be a very useful guide.
> A better idea, if you're in AWS, is to use a VPC and just VPN into your environment;

Is it though? Is having your (and your co-workers) desktops/laptops/and possibly more directly able to access production servers at a packet level a good idea?

I would posit that it is not. A bastion host can be used to lock down more than firewalls, and reduces the attack surface living on your network considerably.

> VPN terminator in the VPC serves the same purpose as a jump box.

It certainly serves the same network separation purposes a jump box does, but is not able to dictate the access environment nearly as much. If only your jump box has access to servers (and production servers properly filtered), every bit of remote access must go through that single ingress/egress point. And that point can have security policy built-in as well for multi-person teams. For example: I can enforce certain key types being utilized on remote hosts, logging of ssh sessions, etc.

I also agree the "single packet authentication" is silly though. Most bastion hosts these days will be behind VPN already. Just another riff on port knocking really.

Bastion hosts are generally a bad idea. There is not a single implementation of a bastion host that doesn't have significant security holes. This is doubly true if you want to do proper group based access control.

Most notable of these issues is the nature of either needing SSH agent forwarding enabled or storing a copy of the private keys on the bastion host (please don't do that). There are ofcourse ways of dealing with this but they aren't worth it.

It's fundamentally much better to just harden every system and implement proper federated authentication and authorization with LDAP. It also makes it much nicer to implement things like MFA and device level authentication.

Remember, it's very rarely that an attacker is going to be able to successfully walk in or break down the front door. Your security holes are very unlikely to be in your administrative channels like SSH/RDP but rather crappy applications or services you may be running on your systems.

Those crappy bits of software + insider threat are the 2 main things you generally need to deal with. Ensuring systems are sufficiently isolated from one another helps with both of these and bastion hosts just add another layer of shared stuff between workloads that should stay entirely separate.

To a one, every medium- and large- sized application deployment I've tested that failed to set up VPC (or something like it) has fallen to some stupid oversight on some stupid backend, support, or dev machine. Don't rely on individual host hardening for security; you'll fail.
I didn't say forgo the VPN. That would be unnecessarily increasing your attack service. In fact I specifically mentioned doing whatever you can to increase isolation between workloads, this includes ensuring access between dev, support and production machines is either not possible or is via careful controlled and audited means.

I only said bastion hosts are bad and generally decrease overall security because it's almost impossible to craft one that doesn't create more security holes than it closes.

>It's fundamentally much better to just harden every system and implement proper federated authentication

Much better? Yes. More practical? No.

Once you have a network of more than 3 hosts, host security can quickly become difficult to maintain. This means not just keeping up to date on patches but auditing all your configurations and carefully reviewing applications and services.

Every organization in the world has serious host security issues. For example, a company like Google has an excellent security team and presumably world class endpoint security, but there's almost no question that an attacker on some segment of their internal network would be able to pivot and take over hosts (the lateral movement would likely be noticed by their SOC, perhaps within an hour or less, but it would still be after-the-fact).

Proper network segmentation and architecture makes most of those issues much less risky though, as it is much harder to actually get to any of those hosts.

Bastion hosts are generally a bad idea.

It's fundamentally much better to just harden every system

You must be one of those people that don't make mistakes and only use bug-free software across their entire deployment.

For the rest of us, DMZ's and bastion hosts are the only way to limit our attack surface to something that can be realistically managed.

Calling fwknop "hand-rolled auth" seems like quite the stretch. If passing around requests via GPG signed and encrypted messages is "rolling your own crypto", then what isn't hand-rolled?
I don't care what you call it; the point is, it doesn't save you from anything that the correct SSH configuration doesn't already protect you from.

If you really believe that OpenSSH is too scary to expose to the Internet for fear of memory corruption flaws, don't stop at a half-measure like "port authentication". After all: once you "authenticate" the port, the attacker can simply hijack your TCP connection to trigger the flaw.

Instead, write a plugboard proxy, and have it apply a static-keyed AES-CTR transform to the data.

I think this is silly, but if you're trying to protect yourself from OpenSSH bugs, at least do something that actually protects you from OpenSSH bugs.

I personally don't use Single Packet Authentication, for exactly the reasons you cite here, but I can understand some of the motivations. The use case that feels convincing to people that do use it, from what I've been able to understand, is that it does protect them somewhat from misconfiguring OpenSSH. If you fuck up and leave password auth enabled and a password set, and you fuck up and don't have monitoring on failed authentication and something like fail2ban or denyhosts, then fwknop is yet another thing that you'd have to fuck up before you're wide open to the brute-force crawlers. One solution to this is "just don't fuck up", but I can understand people having different levels of paranoia about this. Sure, it also protects you from bugs in OpenSSH, but misconfiguration is WAY more likely for your average OpenSSH installation.
Yes, an AES-CTR transformation would be more secure than any port knocking variety such as SPA, at the cost of being more complicated. But the point in port knocking is not its intrinsic security, which is not worth to talk about: its point is actually the obscurity it adds to some other secure access method such as SSH, and its ease of implementation and usage. With it you get much shorter and less noisy logs to monitor, and it's almost for free. In a way, it can be compared to camouflage: its dirt cheap and quite effective at avoiding hits. Bulletproof vests or armoring are still required, but the question is why would you NOT be camouflaged when you can?
You're not thinking the threat model through. If you're not concerned about the kinds of serious, dedicated attackers who have OpenSSH zero-day, you're fine leaving properly-configured OpenSSH exposed. If you are concerned about those attackers, port knocking is barely a speed bump.

I also dispute that a plugboard proxy is "more complicated" than "cryptographic port knocking".

I agree with what you say regarding the APT scenario, port knocking is mostly useless there. It wouldn't be an effective camouflage in that case, and one probably has bigger problems than a hypothetical OpenSSH zero-day then.

I'm not aware of any available plugboard proxy solution, so it would have the added complication of hacking it together. That's what I meant. And there are diminishing returns there: a non-criptographic port knocking scheme with a few packets could be good enough as a camouflage in many cases. But I still find it to be a time saver in the more general case (not APT). I think having cleaner logs is worth using this simple trick then.

> If you are concerned about those attackers, port knocking is barely a speed bump.

is this because you expect this class of attackers to be able to thwart port knocking more or less trivially ?

> "Single-packet authentication" is, I think, silly, and I've never recommended it to anyone. It's hard to think of a case where hand-rolled authentication in front of SSH has saved anyone with a proper SSH configuration (keys-only-no-passwords). You can skip that step.

I agree. I usually use simple port knocking for that purpose. Surely you can agree that "no ports listening" is a better circumstance than "tcp/22 open to the world, with key-based-auth only"?

A VPN is redundant (and requires special client), whereas port knocking to open a port for ssh (that only supports key-based auth) is a simple matter of a few "nc -w 1" lines in a script before you ssh...

EDIT: Saw your posts below. You presuppose that such OpenSSH-zero-day holding attackers are specifically targeting you, which is not invalid; however a spray-and-pray quickly after public release is much more probable in a lot of cases.

This thread is dead, but: everyone really should enable OpenVPN's tls-auth option. tls-auth simply HMAC's each packet that would otherwise be fed to the SSL protocol, which is really good for reducing the impact of SSL-vulnerability-du-jour.

(tls-auth doesn't protect you from vulnerabilities in the OpenVPN data protocol, but that's a lot simpler.)

Jumphosts have their uses, but in this case, I really don't see it. As others have pointed out, better yet is VPN only access.

As for the port knocking silliness, well, I don't see any reason for it.

http://bsdly.blogspot.com/2012/04/why-not-use-port-knocking....

It's worth noting that single packet auth is distinct from port knocking. fwknop isn't relying on "secret" port combinations: it's using GPG to sign/encrypt requests.
It's not meaningfully different - most of the same flaws still apply.

Say you have a bad actor behind the same NAT'ed IP. Good User A signs their SPA, and the firewall opens up for 30 seconds... during which Malicious User B has access.

What real advantage are you gaining vs. just having restricted firewall rules in the first place? Lowering the window of time an attacker has access? If you're not using password auth that could be brute forced, who cares about the window? Either they have the keys or an exploit needed to get in, or they don't.

The advantage over vanilla restrictive firewalls is ability to come from many different IPs without cluttering logs.

In an enterprise setting, this is probably not helpful. At home or for private servers, this is a nice feature.

(Also at home, it helps reduce stupid ISP imposed rules on not hosting ssh... their scanners just assume "no server here" but I can still get to my home network)

I've been doing this sysadmin thing for a long, long time, and have built some large, high profile platforms,[1][2] and I can tell you that few things have made me happier than port knocking.

Yes, it is a password with a tiny number of "bits". But nobody suggested using only port knocking, so that's not significant.

Yes, if my attacker is sharing the exact same NAT'd IP as me then they have a window to attack ... my fully hardened server, which was no worse than what was presented to them before.

Finally (and this is what makes me happiest) my sshd is not exposed to whatever new exploit might be discovered tomorrow that doesn't require user credentials. It's not exposed to being DOS'd by a buffer overflow vuln. It's not filling my logs with brute forcers all day and night.

I love port knocking.

[1] JohnCompanies (first VPS provider)

[2] rsync.net

just run ssh on a non-standard port and you get the same benefits
That is equivalent to limiting SPA to a 16-bit password.
just run ssh on a non-standard port and you get the same benefits

philjr, that is just complete BS. I've run SSH on non-standard ports and they still get a lot of attempts. You know, there are these things called port scans that easily find the daemon on alternate ports.

Mr. Sysadmin, I have four questions:

1. When was the last time your sshd was hit with a 0day?

2. Why aren't you running grsec to prevent buffer overflows (amongst other things) ?

3. Wouldn't it be easier to DOS your host with excess traffic (like all modern DDOS attacks), rather than find a remote-code-execution buffer overflow 0day and use it solely to starve resources on a bastion host?

4. Why don't you use a VPN? If you used a udp-based VPN nobody would even know the port was open (connectionless) and you could even require a preshared key, which is the more secure equivalent of port knocking.

Regarding item 4, a VPN solution is considerably more complex and error prone than port knocking plus SSH. What do you do if/when your VPN service fails? I think you would usually have a VPN service AND a port knocked SSH backdoor.
Yes, good security is usually more complex, though error prone I don't think is accurate. What do you do when your SSH or port knocking service fails?
I meant its configuration and maintenance is more error prone. Certificates may expire without renewal, for example. I've seen port knocking work without a flaw for over ten years in a row, and the SSH configuration is shorter and almost never changes (unless it is to change the cipher or the user).
A side-by-side comparison would require side-by-side configuration. And you can totally configure both SSH and a VPN the same way.

Both ssh and a vpn support authenticating with public keys. Both require maintenance on security updates, or to gain new features, or to strengthen ciphers used, or allow a new user access.

You mention certificates. You do not need to use certificates with either, though you can. You do not need to validate certificate expiration with either, though you can. You can also set expiration to some year well past when the technology will even be relevant.

You claim that expiration of a certificate is somehow a flaw, yet it exists as a security feature. If what you want is to be more secure, you would be wise to use this feature, and not discard it as an annoying maintenance task. However, happily, you can easily ignore it with a VPN just as you do with SSH.

Yes, good security is usually more complex

Wow. So, in your mind software that is more complex is more secure. OK, I think I understand the origin of a lot of your comments, then.

We completely and fundamentally disagree.

What do you do when your SSH or port knocking service fails?

That's a complete red herring. What do you do when your firewall fails??

The answer to #1 is "never". That's no reason not to be aware of the threat and to guard against it.

What is the state of grsec on FreeBSD ? I'm not sure...

Answer to #3 is yes.

I don't use a full-blown VPN in a lot of cases because it's a lot of complexity that isn't needed. knockd is extremely simple (look at the source) and simple to use.

---

This is the way all discussions about port knocking go:

"I like port knocking"

"But port knocking is not the christ child that will deliver us from all of the evils of the world!"

No, it's not. It simply adds a small amount of incremental security at almost zero cost. All else being equal, my sshd was visible before, and now it's not.

Why do you encrypt your traffic?

If the port knocking feature exists to "hide" your network service, you're saying the security of your service is based on the idea that nobody can see your traffic.

But if someone could see your traffic, and therefore your connection, and thus the port your service runs on, they could also clearly see the contents of your traffic. So port knocking would be useless, and everyone could see your traffic, and probably be able to do anything they wanted with it.

But since you use port knocking, you clearly believe nobody can see your traffic. So it then follows that you don't need encryption.

So clearly you should be using telnet.

So clearly you should be using telnet.

This is just an absurd statement. You have conflated to completely separate issues.

you're saying the security of your service is based on the idea that nobody can see your traffic.

No, it's because no one can find the service, not see the traffic.

Okay, let me see if this explains it better:

1. Anyone who can see your traffic can see where the service is.

2. We assume anyone can see your traffic because you're using an encrypted connection. (The conversation should have stopped here, but i'll continue anyway...)

3. The only thing port knocking does is obscure your service from port scanning.

4. The equivalent method to preventing someone finding your service via port scanning is changing the port number.

5. Changing the port number prevents all non-direct-attacks from finding your service. (Nobody scans every TCP+UDP port of every IP on the internet for one service, because 563.6 trillion network packet send+recvs takes a long time, which doesn't even factor multiple tries or error retransmissions).

6. A direct attack by a committed attacker will eventually find your network service.

7. Port knocking does not add to security. It partially obscures information, but it does not prevent access, and it operates on faulty assumptions about network security; the information is still there, you just have to look for it differently. It is less reliable than even IP whitelisting.

8. In the real world (and not the "gee I think this sounds more secure" theoretical world) every single other network service you have will be exploited before OpenSSH. So trying to hide OpenSSH while not hiding your other services becomes farcical.

1. Anyone who can see your traffic can see where the service is.

Dude, right there you lost me. 99.99999999% of the internet can't see your traffic. The NSA can. Your ISP can. Random threats on the 'net at large cannot see your traffic.

You are radically misinformed. As a hacker, I have about a dozen ways to see your traffic. We don't encrypt credit card transactions because we're afraid the NSA or your ISP's network admins might pick up your card and buy some sneakers on Amazon.

Ways I can see your traffic:

  * fucking with bgp
  * fucking with routers in general
  * fucking with switch port tables
  * fucking with proxies
  * spoofing
  * sniffing wireless networks
  * sniffing wired networks
  * tapping a cage backbone
  * rooting an isp
  * rooting a dslam
  * rooting pops
  * rooting firewalls
  * rooting endpoints on management ports
  * abusing hosted networks
The single easiest way to view someone's traffic is to use the completely fucked-up retarded configuration of an intermediary router. Do you know how many routers sit between you and your destination? Do you know how many of those have 10-year-old configurations enabled via telnet on publically-routable interfaces with default snmp community strings and 6-character passwords?

Don't assume that just because you aren't aware of a particular attack that it doesn't exist or doesn't apply to you. Hackers are only limited by their creativity.

Please don't trust the network.

4. Why don't you use a VPN? If you used a udp-based VPN nobody would even know the port was open (connectionless) and you could even require a preshared key, which is the more secure equivalent of port knocking.

I love this. I absolute hate VPN. It's waaaay too complex and I fear vulnerabilities in that waaay more than other software. So, I'll stick with port knocking. It's much safer, IMO.

This article goes from mediocre to bad when it starts advocating security by obscurity.
Security by obscurity is perfectly fine when it's part of a layered approach. It will help keep attackers from learning the details of your architecture and will also stop skiddies in their tracks.
Exactly.

(Your most secure server) is not as secure as (your most secure server with some obscurity added).

This. Obscurity has too much bad press. It's not to be depended on, but it can be a useful addition sometimes.
Bastion hosts are the wrong approach, as are VPNs. They might have had some merit in the days of handcrafted servers, where "hardening" was something you could do to a single machine. But nowadays, especially on AWS, there's no point in a configuration that isn't automatically reproducible - in which case, why not harden all your machines?

Then you avoid having a single point of failure. If and when a single SSH key is compromised, there are a limited number of hosts that key had access to - and unless that's the log server, you have secured logs so you can figure out what happened. Different keys can have different capabilities, rather than a one-size-fits-all policy on the bastion. It makes it harder for an attacker to know where to get the boss's keys. And you can expand your system geographically without the performance compromises of a VPN; your servers already make encrypted, secured calls to each other (over SSH, SSL or a very small number of vetted protocols), expose only those ports that are actively used. Your attacker could steal your entire system documentation and it wouldn't make any difference, because the system is truly secured, not relying on obscurity.

Better yet. Store the authorized public keys in LDAP or similar system. (AuthorizedKeysCommand makes this super simple these days).

That way you can rotate them easily if the private keys become compromised.

I recently built a system using this technique and synced the keys to S3, very effective.

You're describing an overly optimistic scenario. "Why not just harden all your machines?" Because it's more complicated than that. Yes, you can automate basic hardening (configuration of access controls, limited use of SSH keys), but how do you automate hardening of RPC between one server and another, or hardening of those 3 internal administration web apps which may or may not have some local file disclosure and crypto flaws?

Defense-in-depth is still the only sensible way to handle security. You do what you can on the host side, but bastion hosts, jump hosts, VPNs, etc. provide an additional layer to make lateral movement much more difficult for an attacker.

In the Enterprise world, jump boxes are more typically just a hack implemented by IT workers who don't want to go about properly setting up a firewall and 802.1x/vlan/vpn/ipsec/etc to allow strict control to a protected network resource. (Or because they don't have the hardware capability to do it, or their network admin decided against it).

The only two proper excuses for ever using a bastion host are "I don't have the money to do it right" and "I don't have the time to do it right."

I'm a fan of the VPN terminator as jump point approach.

OpenVPN with tls-auth switched on is a pretty robust front door. If you combine that with enforced key only auth for ssh, ensure all internal machines run their own firewall and logically split different bits of your internal network off in to their own subnets you've got a fairly secure foundation to build upon. Another easy win is switching internal services over to TLS/SSL only (where it makes sense/is possible to do so).

You appear to be using a browser that may not be able to display our site correctly. Generally, this is beause it's an older browser that doesn't support a lot of the newer, awesome features in open Web standards.

Rather than provide an untested and possibly subpar experience of our site, may we suggest downloading one of these fine browsers? (We think you'll find your overall experience of the Web will improve if you do.)