I have two dozen co-located servers to log into. So far I've resisted the idea of setting up a bastion host, because that seems like a way to lock myself out of SSH access if that machine dies. I'm also not sure of the rationale for having a bastion host, other than "big companies do it". How do other non-cloud people handle this access vs. security tradeoff?
I'm also curious what people's preferred fallback method is for preserving access to machines if you lose access to your yubikey(s), assuming you keep your private SSH key stored on one.
I'm not sure I understand what you mean. I run a one-person business and my servers are 110 miles away in Sacramento; what do I do if my yubikey breaks?
It sounds like you're trying to do something like Yubikey OTP or U2F with SSH. Both are doable, but I don't think that's the normal way people use hardware tokens with SSH; what I've seen much more is that people use Y4 keys (which are basically pocket HSMs), generate RSA keypairs with them, and configure SSH to accept the public key on the token.
This doesn't require any support on the serverside; as far as the server's concerned, the Y4 is just another RSA key.
It does require the more expensive Yubikey, but there are some security advantages to using it.
No, my thinking is along the lines of what you're describing (generate the key on the Y5 Yubikey so that it never leaves it). The discussion of one-time stuff is just as backup for scenarios where I lose the key, which I invariably will, or need to give other people "break glass in case of emergency" access to the servers.
You should already have a full backup of your (private) RSA key anyway on an encrypted USB drive in a desk drawer somewhere, so you use that until you can provision a new Yubikey.
Really? Do any compliance standards specifically require this? Or is the specific control defined in the organization itself that requires this?
I've found many compliance standards to be pretty open-ended and function-driven, rather than prescribing specific standards. Client contracts.... they may be a different beast, and often require specific promises by vendors.
I'm pretty sure that at least some of the German financial regulations require that you can audit administrative actions on systems that are relevant for the financial transactions.
Now, you could try to implement that on the systems themselves, but how do you establish a mechanism that logs actions of an administrative user, and at the same time cannot be disabled by the same administrative user?
So instead you use a shell control box as a bastion, and regular administration of the target hosts don't have root access to the shell control box, so they cannot circumvent logging.
I haven't yet any standards that explicitly demand bastions, but it seems to be one of the standard implementations that have proven to pass audits, and so it's a pretty low-risk implementation of the logging requirements.
Easy: on each host you log into, you enter into authorized_keys your Yubikey RSA pubkey and a standard software SSH pubkey. The private key for the software keypair, you stick on a thumb drive and keep in a drawer or a safe or whatever.
Later
Sorry, I see below you're thinking about using Yubikeys in addition to keypairs. This obviously doesn't answer that question.
It's key to setup two or three bastion hosts with similar diversity to your prod fleet (are you in multiple locations? put a bastion in at least two of those; if not, do you have multiple racks? put a bastion in at least two of those, etc) -- you will have network incidents where some hosts are unreachable, you don't want that to hit all your bastions or you'll be very sad.
Bastions are nice, because you can harden them, or put 2fa on them, and not worry as much about the ssh config on the other hosts -- just make sure they don't accept ssh connections from the outside world.
I’ve only ever seen this handled with VPN, and the answer there is that you’d need to launch another vpn server in the right vpc/subnet if one went down (or have failover setup).
It looks like there’s some benefits with bastion hosts around locking down commands & access in a fiber-grained way from vpn, but man what a pain in the ass. I can see a certain perspective that would really care to do things this way but VPN from my cold dead hands.
Once just for fun I setup HAProxy in TCP mode and depending on the host name would direct the SSH to the correct host. But I had some nice ACLs so that if they weren’t in the ACL they would be sent to a honeypot.
That is true (and relatively easy to do with OpenSSH's ProxyCommand) but that was not the implication of the statement that HAProxy was being used in TCP mode to direct incoming TCP sockets to the correct backend SSH server based on the hostname using any SSH client that understands the SSH protocol.
I've thought about doing something like this several times, but the proposition of using any tools/libraries/pam modules/etc not installed by default, custom pam/sshd configs, and generally anything "outside of the box" sort of scares me.
I've used 2FA for SSH at $lastjobatmegacorp, however all that infrastructure was supported by a team of people dedicated to such things. How finicky would setting this up for myself be in practice? (Am I being paranoid for NOT wanting to go the extra mile to ensure that I actually have access to my systems when I need it the most?)
You could run a separate instance of say, dropbear on another port as a failsafe, until you're comfortable. Or maybe firewall it off to a set of fixed source ips.
not a shill but I have great things to say about Yubikey. I'm a longtime linux enthusiast, but never a dojo master. Buying the yubikey and seeing the words "PAM module" made me expect a rodeo, but it was surprisingly easy. just from memory, but it was basically (on arch) 1) install the yubikey package, including a GUI utility to set up the Yubikey and PAM libraries (just .so files). 2) mess around with setting up the keys on the yubi. there are 2 slots that can be set up a variety of ways. 3) configure pam. it was basically one text file, and maybe copying a .so, way easier than I imagined. 4) set up ssh to only accept PAM authentication (delete 2-3 lines, add one line). 5) set up display manager to accept the yubikey OR password.
so this way, my always-on router can ONLY be accessed by a yubikey. That uses "slot 1", which requires me to insert the yubi and long-touch it(it only has one button). and my desktop can be accessed without a password if the yubi is inserted (without pressing it or anything, this is slot0) or I can type my password if I want.
what I thought was going to be a wild rodeo with bucking broncos and PAM module errors, turned out to be riding the coin-operated horse in front of the grocery store.
I wrote a blog post on this recently, using only open-source tools that don't come from big corps. To have TOTP second factor on Debian (like) systems you need only libpam-oath module on the server, and perhaps an open-source app like FreeOTP (RedHat) on a smartphone.
I'm afraid to link it here because the traffic might kill my puny box.
Definitely, all traffic is readable by CloudFlare. Whereas if you have your own SSL certs everything going through the "normal internet infrastructure" is encrypted (in theory your webhost is able to get to it, but definitely not as easily - and this is only an issue if you rent a server).
On top of that there are many arguments against CloudFlare when it comes to an open and decentralized internet.
EDIT: didn't read the Philosophically in your comment. CloudFlare does many things against an open internet, for a long time they blocked (or required excessive captchas) for all TOR users.
Looks great. If it's static, try using netlify and cloudflare; I've used cloudflare with great success for years, and I just tried netlify (works great & free).
> I'm afraid to link it here because the traffic might kill my puny box.
So what if it does?
Presumably it isn't a "critical" host so the worst that could happen is, what, no one can read what your blog posts for a little while? Eventually, the traffic will go away and things will go back to normal, yeah?
I assume that (part of) the reason for writing your blog post was so that others could read it. Yet, by not posting the link, you are ensuring that we don't -- even though pretty much all of us in this comment thread are the target audience.
Alternatively, as someone else mentioned, there are things like the Internet Archive.
(FWIW, several years ago, a page on my blog was linked by The Atlantic one morning and shortly thereafter it also hit the front page of (IIRC) both Reddit and HN. The only way I figured out that "something was up" was due to the huge number of "new follower" notifications I was getting from Twitter as I was driving to work. At the time, I was running WordPress on a little VPS with 1 CPU core and 768 MB of RAM and it handled the ~115,000 page views just fine that day -- although I did have caching and such in place. YMMV.)
I had considered that, but as someone else else pointed out, what have I got to lose if it's taken down?
At least with links to my blog I can use the access logs for an idea of which articles people find interesting (I don't do other metrics/analytics due to my philosophy).
At the moment I'm trying to build a new box to host this on but it seems Caddy has broken source builds (again) and I need plugins for my Git hooks and Hugo build.
Megacorps have their own reasons to select technology. Your best bet is probably to go with a Yubikey (or something like it). It's about $50 and it's what several of the big Linux projects themselves use so it's well tested and documented.
- if you have an opaque enclave on the device (e.g. a black box you can't peer into), you can't know what it's doing
- if you don't, you can read the private key bits out
Your statement was that it has a private enclave you cannot extract key material out of, which resolves the latter half of the question, but not the former, I think?
I had to do a pam config recently. Having only tweaked a few settings previously, it was kind of intimidating as I was doing some security-type stuff and had to rewrite the config. I locked myself out more times than I care to remember; thank God I was working (mostly) in virtual machines. This is especially a problem since the necessary pam settings vary from distro to distro.
If you really want to, test it on a vm that replicates your machine first and make sure you document exactly what to do to get it working. To be clear, it should be fine if you're just adding a module, but sometimes getting the exact settings can be hard, so make sure to do it in a vm first and be careful. Just trying to keep others from going through what I went through.
If all you care about is your SSH key not being stolen, then you can very easily use a YubiKey (or other smartcard..) with SSH via either GPG or PKCS11. Both will mean your key never leaves your YubiKey (or other smartcard...). This doesn't prevent your colleagues from having their key stolen, but does protect yours.
I can use the YubiKey for SSH from Linux, Mac, Android phones without issue, and I keep several YubiKeys with my keys on them so it's extremely unlikely I'll loose them all at once.
Re the paranoia - if you're really the only person who can fix something, and you have lost your closest key, then - the thing in need of fixing can wait till you get home to your second key ;)
I'm less worried about losing a hardware key than I am about something breaking in a mysterious way and locking me out! My initially question was really about personal (unshared) systems. (I don't know - a random auto-update busting a non-standard PAM module and preventing it from loading, or even something just changing the PAM configs on its own, doesn't seem impossible to me). I'd definitely keep a spare if i went the Yubi route.
It's pretty hard to permanently get locked out. In a pinch, you can always mount the disk in another machine, fix whatever is wrong, and boot the original machine back up.
I'd never done anything much with pam before but found it relatively easy to get PrivacyIDEA working on CentOS, it's even easier if you're using Ubuntu. https://www.privacyidea.org/ and Cornelius is really helpful.
Wait, really? They still make these? People still use them? How secure are they really compared to USB tokens like a Yubikey? How do you interface with one of these?
Huh, I guess you are technically correct. They are listed on the Yubikey product page, though, and Yubikey is so strongly associated with Yubico that the distinction is easily lost.
I think it’s pretty impossible to expect people to look at these keys and think “ah yes, this is a Yubico Security Key, not a Yubikey”, given they look like a painted yubikey, are sold by the same company, have some feature overlap, and the docs are commingled with yubikey docs.
The DoD uses them extensively. All military personnel are issued one and it’s used for everything from accessing workstations and email to buying your meals at the DFAC.
CACs are just within the Department of Defense (+ Coast Guard). In the wider Federal government, they are called PIVs and use a slightly different standard:
Just about every Dell and Lenovo laptop from the enterprise lineup still comes with a SC reader option. In the enterprise world, smartcards are the de facto standard. I've heard complaints from $BIGCORP managers that they couldn't have iPads because they don't offer smartcard functionality.
Somehow I managed to snag one of the Thinkpad models that doesn't have an SC reader option (T420p), which is disappointing, because this seemed like a cool thing to look into and experiment with.
In some countries, every national ID card is a smartcard, so millions of people use them. You can authenticate against our IRS site with yours, for example.
Not much as far as SSH is concerned, but you can buy many smart cards and just use one cardreader. For personal use it's more convenient to use Yubikey as a commercial proprietary solution.
I guess it depends. If I’ve got a fleet of users with commodity laptops, and I want them to do CCID auth, I could buy them each a card reader and a card, or I could buy them each a yubikey and use their USB port.
Things are different if this is the DoD or somewhere that’s already got card readers as a core component, but if this is the DoD, I’m already winning because I get to use their PKI/cards and don’t have to build my own PKI :D
If you're using a Vault system, it's also relatively easy and well documented to use Vault for this process. There's two components, which is the OTP configuration of Vault itself:
You can get 2FA over ssh by requiring both key based and password based authentication. That is, you need both your private key and your account password to log in (in addition to your private key passphrase).
disclaimer: not a security expert of any kind. Also apologies
in advance for hijacking the thread.
I loosely remember reading on HN that wireguard could be a replacement for ssh. Is that still the case? When do you think we'd be switching away from ssh to wire guard?
SSH is a service to log in securely to get CLI access. WireGuard is an efficient VPN with low attack surface. SSH can be used to run X programs, as proxy, or VPN.
Development infrastructure like Jenkins has no business being on the internet, so my preferred "second factor" is a VPN, secured with machine specific certificates that offer only VPN connectivity, but not SSH or anything else. This means even if a developer's Git or SSH key is floating around your infrastructure no one without VPN access can get at it, and if a developer loses a laptop there's a good chance you'll hear about that in time to revoke that machine's VPN certificate before anyone has a chance to use it.
That approach is a good defense in depth technique.
Network security by itself doesn't work but entirely neglecting it as some misguided articles I read recently makes me wonder if security is actually moving backwards when it comes to fundamentals and principles.
Granted those supporters are frequently cloud vendors, that dont offer mature enterprise integration. But when smaller shops follow those as best practices I'm getting worried.
Exactly. This is what I employ as one of the safeguards as well. All important resources accessible only to via a jumpbox, and jumbox itself is only accessible in a VPN.
My fear is that if the VPN connection is severed, you're locked out of your environment.
I typically have a public bastion and just shut it down. In AWS you aren't paying for it if it's turned off and you have an "escape hatch" if you need it.
If someone manages to launch another instance or start the stopped one then they would have access, but that's another can of worms.
With AWS, in an emergency you can always use the console to provision access of some sort: SSH bastion with a public IP, second VPN, etc. This definitely implies that explicit 2FA is needed for console access, but I’m pretty comfortable with using smartphones for that.
Just going to throw out there that the company I work at (JumpCloud) is a directory service that makes it super easy to setup 2FA with your SSH. Literally a couple checkboxes, and you can mandate that your whole staff has 2FA setup.
That helps against someone stealing your computer, but not if someone hacks it and manages to get a trojan running - it'll just keylog your passphrase.
There are scenarios where 2FA makes no sense for me. These are local password managers and SSH keys.
Rethink this again: If your ssh key is compromised then you have a problem overall. For my point of view there is no real security gain in 2FA ssh key logins because your private key itself is already a secret! Only thing which is important is to set a good passphrase for your ssh key.
However I can think of one exception of 2FA on top of ssh keys which might be useful: 2FA for gaining root access or sudo commands. This might be okay but the attack vector in this scenario would be that someone can see your keystrokes or clipboard (then they most likely also have your ssh private key if they can do that).
As long as you have a passphrase and don't actually circumvent it (by e.g. an agent) you basically already have 2FA (you need access to the key (on your laptop/pc) and you need access to the passphrase (in your head)). But physical 2FA (like smartphone based ones or smartcard based ones) are nice because you can remove them from your computer or destroy them. You can't "destroy" a passphrase, so passphrase and sshkey are not really "independent".
102 comments
[ 2.9 ms ] story [ 164 ms ] threadI'm also curious what people's preferred fallback method is for preserving access to machines if you lose access to your yubikey(s), assuming you keep your private SSH key stored on one.
We’re doing this successfully where I’m working.
This looks like exactly the answer I needed; thank you!
This doesn't require any support on the serverside; as far as the server's concerned, the Y4 is just another RSA key.
It does require the more expensive Yubikey, but there are some security advantages to using it.
I would be interested in how you think U2F with SSH is doable. Maybe with a custom SSH server and client?
Have you actually see this done in anger?
It's a central place where you can do your logging, which many enterprises must do for compliance reasons.
I've found many compliance standards to be pretty open-ended and function-driven, rather than prescribing specific standards. Client contracts.... they may be a different beast, and often require specific promises by vendors.
Now, you could try to implement that on the systems themselves, but how do you establish a mechanism that logs actions of an administrative user, and at the same time cannot be disabled by the same administrative user?
So instead you use a shell control box as a bastion, and regular administration of the target hosts don't have root access to the shell control box, so they cannot circumvent logging.
I haven't yet any standards that explicitly demand bastions, but it seems to be one of the standard implementations that have proven to pass audits, and so it's a pretty low-risk implementation of the logging requirements.
Later
Sorry, I see below you're thinking about using Yubikeys in addition to keypairs. This obviously doesn't answer that question.
Bastions are nice, because you can harden them, or put 2fa on them, and not worry as much about the ssh config on the other hosts -- just make sure they don't accept ssh connections from the outside world.
It looks like there’s some benefits with bastion hosts around locking down commands & access in a fiber-grained way from vpn, but man what a pain in the ass. I can see a certain perspective that would really care to do things this way but VPN from my cold dead hands.
I've used 2FA for SSH at $lastjobatmegacorp, however all that infrastructure was supported by a team of people dedicated to such things. How finicky would setting this up for myself be in practice? (Am I being paranoid for NOT wanting to go the extra mile to ensure that I actually have access to my systems when I need it the most?)
You could run a separate instance of say, dropbear on another port as a failsafe, until you're comfortable. Or maybe firewall it off to a set of fixed source ips.
so this way, my always-on router can ONLY be accessed by a yubikey. That uses "slot 1", which requires me to insert the yubi and long-touch it(it only has one button). and my desktop can be accessed without a password if the yubi is inserted (without pressing it or anything, this is slot0) or I can type my password if I want.
what I thought was going to be a wild rodeo with bucking broncos and PAM module errors, turned out to be riding the coin-operated horse in front of the grocery store.
I'm afraid to link it here because the traffic might kill my puny box.
Setup Cloudflare, which is free and easy to use. Or, just post an Internet Archive wayback machine copy of your post would be good enough.
> I wrote a blog post on this recently, using only open-source tools that don't come from big corps.
I doubt he wants to use CloudFlare ;)
On top of that there are many arguments against CloudFlare when it comes to an open and decentralized internet.
EDIT: didn't read the Philosophically in your comment. CloudFlare does many things against an open internet, for a long time they blocked (or required excessive captchas) for all TOR users.
So what if it does?
Presumably it isn't a "critical" host so the worst that could happen is, what, no one can read what your blog posts for a little while? Eventually, the traffic will go away and things will go back to normal, yeah?
I assume that (part of) the reason for writing your blog post was so that others could read it. Yet, by not posting the link, you are ensuring that we don't -- even though pretty much all of us in this comment thread are the target audience.
Alternatively, as someone else mentioned, there are things like the Internet Archive.
(FWIW, several years ago, a page on my blog was linked by The Atlantic one morning and shortly thereafter it also hit the front page of (IIRC) both Reddit and HN. The only way I figured out that "something was up" was due to the huge number of "new follower" notifications I was getting from Twitter as I was driving to work. At the time, I was running WordPress on a little VPS with 1 CPU core and 768 MB of RAM and it handled the ~115,000 page views just fine that day -- although I did have caching and such in place. YMMV.)
Here's the link, let's see what happens:
Caveat lector: This is a "beginner" guide targeted at Raspberry Pi users, ignore the Pi related parts.
I intend to publish a second part about using keys instead of passwords.
https://2byt.es/post/totp/
The blog content is open source on GitHub so feel free to raise issues if necessary.
I doubt hn will take down github...
At least with links to my blog I can use the access logs for an idea of which articles people find interesting (I don't do other metrics/analytics due to my philosophy).
At the moment I'm trying to build a new box to host this on but it seems Caddy has broken source builds (again) and I need plugins for my Git hooks and Hugo build.
Searching around for an alternative.
Edit: and thanks for the compliment.
https://www.nitrokey.com
- if you can flash/read everything, so can an attacker
- if you have a blackbox nothing can peer into, how can you trust the device?
1) You generate your own root keys, to deploy onto the device,
2) You can't read the root keys back off the device once it's deployed.
My statement was that:
- if you have an opaque enclave on the device (e.g. a black box you can't peer into), you can't know what it's doing
- if you don't, you can read the private key bits out
Your statement was that it has a private enclave you cannot extract key material out of, which resolves the latter half of the question, but not the former, I think?
If you really want to, test it on a vm that replicates your machine first and make sure you document exactly what to do to get it working. To be clear, it should be fine if you're just adding a module, but sometimes getting the exact settings can be hard, so make sure to do it in a vm first and be careful. Just trying to keep others from going through what I went through.
I can use the YubiKey for SSH from Linux, Mac, Android phones without issue, and I keep several YubiKeys with my keys on them so it's extremely unlikely I'll loose them all at once.
Re the paranoia - if you're really the only person who can fix something, and you have lost your closest key, then - the thing in need of fixing can wait till you get home to your second key ;)
https://github.com/philipWendland/IsoApplet/wiki
https://www.yubico.com/store/#SKY https://www.yubico.com/product/security-key-by-yubico/
I think it’s pretty impossible to expect people to look at these keys and think “ah yes, this is a Yubico Security Key, not a Yubikey”, given they look like a painted yubikey, are sold by the same company, have some feature overlap, and the docs are commingled with yubikey docs.
As far as I can tell first Yubikeys were basically small cardreaders with non-removable smartcards in SIM card formfactor.
> Wait, really? They still make these?
Every credit card with a chip is basically a smartcard, every SIM card too. Smart cards are not going anywhere.
> How do you interface with one of these?
CCID cardreader + OpenSC
Mandated by the US government so yes, lots of people use them.
https://en.wikipedia.org/wiki/Common_Access_Card
https://en.wikipedia.org/wiki/FIPS_201
Things are different if this is the DoD or somewhere that’s already got card readers as a core component, but if this is the DoD, I’m already winning because I get to use their PKI/cards and don’t have to build my own PKI :D
https://www.vaultproject.io/docs/secrets/ssh/one-time-ssh-pa...
You also have to configure the servers to support Vault, using their PAM integration:
https://github.com/hashicorp/vault-ssh-helper
https://github.com/gravitational/teleport
I loosely remember reading on HN that wireguard could be a replacement for ssh. Is that still the case? When do you think we'd be switching away from ssh to wire guard?
Network security by itself doesn't work but entirely neglecting it as some misguided articles I read recently makes me wonder if security is actually moving backwards when it comes to fundamentals and principles.
Granted those supporters are frequently cloud vendors, that dont offer mature enterprise integration. But when smaller shops follow those as best practices I'm getting worried.
I typically have a public bastion and just shut it down. In AWS you aren't paying for it if it's turned off and you have an "escape hatch" if you need it.
If someone manages to launch another instance or start the stopped one then they would have access, but that's another can of worms.
Rethink this again: If your ssh key is compromised then you have a problem overall. For my point of view there is no real security gain in 2FA ssh key logins because your private key itself is already a secret! Only thing which is important is to set a good passphrase for your ssh key.
However I can think of one exception of 2FA on top of ssh keys which might be useful: 2FA for gaining root access or sudo commands. This might be okay but the attack vector in this scenario would be that someone can see your keystrokes or clipboard (then they most likely also have your ssh private key if they can do that).