I was just going to submit this thread, thinking how in the ** could the github host key being changed. Leading to their blog post explaining it.
Their exposed their private RSK key!
ssh-keygen -f ~/.ssh/known_hosts -R github.com
Now it's a ED25519 with +DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
(they don't list this on their blogpost, just their renewed RSA key)
Please before replacing your local fingerprint with the new one, double check it is the expected value. This is an opportune time for man-in-the-middle attackers to strike, knowing everyone has to replace their stored signatures, and that some will be lazy about it with a blind "ssh-keygen -R github.com" command.
It never fails to amaze me how most incident mitigations seem completely oblivious to such security side effects.
"We have no reason to believe that the exposed key was abused, but out of an abundance of caution, we are going to expose 50 million users to a potential MITM attack unless they are extremely careful."
Not a single word in the post about whether this impact was even considered when making the decision to update the key. Just swap the key, and fuck the consequences. Same with the mass password resets after a compromise that some services have done in the past years. Each of those is any phishing operation's dream come true.
There isn't an alternative, really. The private key has been exposed, and presumably it is unknown if or how far it has spread. The SSH keys must be changed, and the sooner the better. All that can be done is to notify people after the change has occurred.
Since the post doesn't mention anything like "we reviewed logs and confirm the key was not accessed", it is very likely that they either don't have logs that are reliable enough to rule it out (e.g. they may be sampled or otherwise incomplete), or that the key was accessed.
No, it wouldn’t be - assuming by “insane” you mean “silly to do”. I build systems at Google that do exactly that.
Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
Presumably, keeping 'last remotely accessed' and 'last remotely modified' for every file (or other stats that are a digest of the logs) is sane for pretty much any system too. Having a handle on how much space one is dedicating to files that are never viewed and or never updated seems like something web companies that have public file access would all want?
What guarantees do these systems provide? Are 100% of requests where data was served guaranteed to either end up in the log or at least create a detectable "logs may have been lost here" event?
Or does it log all the requests all the time as long as everything goes well, but if the wrong server crashes at the wrong time, a couple hundred requests may get lost because in the end who cares?
It's not just GET requests. Someone could have cloned/refreshed the repo using ssh. The repo might have been indexed by github's internal search daemon which might not use the public HTTP API but uses internal access ways however those might look like. You might have purged the database of that daemon but what about backups of it? What about people who have subscribed to public events happening in the github.com/github org via the API?
You'd have to have logging set up for all of these services and it would have to work over your entire CDN... and what if a CDN node crashed before it was able to submit the log entry to the log collector? You'll never know.
One H4X0R gave it to four friends, who in turn gave it to between 9 and 14 friends, who in turn gave it to between one and 6 friends.
If train A leaves New York going 60 miles per hour with 167 people on board and train B leaves Chicago one hour later with 361 people on board going 85 miles per hour, how many people now have the key?
Same as with any other decision: Do a cost/benefit analysis of whether the security risk created by rotating the key is actually outweighed by the security risk of doing nothing, taking into account logs that should tell you whether the exposed key was indeed accessed by unauthorized parties.
To be 100% clear: Both courses of action come with associated security risks. The problem is not choosing one course of action over the other, the problem is thinking you can just skip the cost/benefit analysis because the answer is somehow 'obvious'. It's not obvious at all.
Wrong. A CBA is always needed. If the potential damage from MITM attacks made possible by rotating the key is greater than the potential damage from a rogue key multiplied by the likelihood that someone actually accessed the key, then it is wrong to rotate the key. It's that simple.
The only way a CBA would be unnecessary is if rotating the key didn't have any security risks. But it does.
- if they have evidence that the key was exposed to one person, even with zero usage of the key, failing to rotate the key is tantamount to knowingly accepting widespread compromise at a potential attacker’s whim. At GitHub’s scale, that’s untenable.
- rotating the key is the only correct reaction to that
- they should have better communications in place to help users mitigate MITM
- there really isn’t an option, because they’re critical infrastructure; I’m glad they know that and acted accordingly
- on principle this speculation makes sense, but understanding the threat makes it moot
- you hopefully know that, and it’s good to insist on thoughtful security practices but it’s important to also understand the actual risk
Only if you know for certain that the key has been accessed by a third party.
If you don't know for certain, you have to factor in the likelihood that it has been, and at that point, the two risks aren't equal anymore so that logic doesn't work.
What? This is a terrible way to reason about risks in general. If you don't know for certain, you should assume the worst case scenario, especially since it's impossible for you to calculate the probability distribution of the likelihood of a leak.
You should only keep moving along without key rotation if you know for 100% certainty a leak didn't happen and no one accessed the key (not theoretically impossible if they had the server logs to back it up), but anything minus that and you have to assume it's stolen.
That's why you need certificates and not just a key pair. Certificates make key rotation easier, and you want key rotation to be easy.
I guess the proper way forward is a small utility that gets the latest signature through http+tls, and replaces the line in your known_hosts file, all in the background.
Looking long term, maybe we need to get rid of all the security stuff in ssh and just pipe the rest of its functionalities inside a TLS pipe. Let the os do its certificate management, reuse security bricks that are way more studied, ...
Certificates just add more keys to worry about. The beauty of SSH is that it does not add hugely trusted parties in the name of convenience, while the UX of TOFU (trust on first use) is pretty decent.
The real solution to break out of these UX/security tradeoffs is to put domain names on a blockchain: then you can simply rotate the key in your DNS record, while the blockchain model is such that you need to compromise many parties, instead of "one out of many parties", as with CAs.
Tracking Bitcoin chain for DNS updates is lightweight enough that it can be built into OS alongside other modern components such as secure enclave, TCP/IP stack and WiFi/BT/5G radios.
Those keys can be worried about on a better secured computer, and don't need to be spread out on every frontend ssh server. Also it allows you to have each machine have a different host key pair, so if one leaks, only that single machine may have some trust issues, and not the whole fleet.
Also it's way better than TOFU, you can just add the CA key to known_hosts and avoid TOFU for each machine.
(Nevermind that you'll probably not accidentally commit some semi-ephemeral host key that's rotated often somewhere, because it will not be some special snowflake key you care about, but something handled by your infrastructure software automatically for each machine)
Are there any cert solutions that dont involve having to maintain a revocation list? I only used certs with openvpn years ago and the CRL was a potential footgun.
The alternative would be to use certificate authorities (ssh has CA support) which allow to effectively have private keys at different levels and allow you to keep the root private key in a physical vault and use it very rarely to issue other private keys
And then don't forget to setup key revocation as well, and make sure that an attacker in a position to MITM the connection cannot cause the revocation checks to fail-open.
I hope you don't need that SSH connection to fix your broken CRL endpoint!
Sure, but isn't it more likely that a key that has to be shared by who knows how many ssh load balancer machines at GitHub and can't be easily rotated because it's pinned by millions of users, isn't it more likely that that private key gets eventually compromised or thought to be at risk at being compromised?
We need to compare the relative risks within the same context, namely within a company like GitHub
So it's not relevant to bring up failures of other CAs
Jump into a time machine, go back to the creation of SSH, and adopt SSL-style trusted third-party certificate authorities. Somehow get it adopted anyway, even though loads of people use SSH on internal networks where host-based authentication is difficult; SSH is how many headless machines are bootstrapped; and that you've got to do it 19 years before Lets Encrypt.
Jump into a lesser time machine, go back to when Github were creating their SSH key, and put it into a hardware security module. Somehow share that hardware-backed security key to loads of servers over a network, without letting hackers do the same thing. Somehow get an HSM that isn't a closed-source black box from a huge defence contractor riddled with foreign spies. Somehow avoid vendor lock-in or the HSM becoming obsolete. Somehow do this when you're a scrappy startup with barely any free time.
If you use DNSSEC (cue inevitable rant from Thomas) this just works. If you have DoH (and why wouldn't you?) and your trusted resolver uses DNSSEC (which popular ones do), you get the same benefits.
Unless it has changed recently, you can't have a trust chain of OpenSSH certs though so it's cumbersome that your signing key is not only the root ca but also basically has to be 24/7 accessible to sign any server/client you want to bring up.
I'd guess that most systems aren't using DoH/DoT or end-to-end DNSSEC yet. Some browsers do, but that doesn't help tooling frequently used on the command line.
I suppose you could just accept X.509 certificates for some large/enterprise git domains, but that pokes up the hornet's nest that is CA auditing (the browser vendors are having a lot of fun with that, I'm happy that the OpenSSH devs don't have to, yet).
And where do you maintain the list that decides which hosts get to use TOFU and which ones are allowed to provide public keys? Another question very ill-fitted for the OpenSSH dev team.
There could be an update to the protocol that enables certified keys to be used and allows them to be accepted without warning or with less of a warning.
There could be a well known URL that enables people to fetch ssh keys automatically in a safe manner.
> and adopt SSL-style trusted third-party certificate authorities
So that any large entity can own your servers with easy. (Well, they already can, but not through this vulnerability.)
Anyway, the only thing CAs do is to move that prompt into another, earlier time. It's the same prompt, the same possibility for MITM, and the same amount of shared trust to get wrong. You just add a 3rd party that you have to trust.
SSH does have a CA system. Anybody that isn't managing a large datacenter will avoid it, for good reason.
> So that any large entity can own your servers with easy.
Eh, let's not pretend existing SSL certificate validation is anything to write home about.
Even without any ephemeral servers involved, barely anybody is validating cert fingerprints on first use.
And among people using ephemeral servers, 99% of applications have either baked a certificate into their image (so that any compromised host means a compromise of the critical, impossible-to-revoke-or-rotate key) - or every new server gets a new cert and users have either been trained to ignore certificate warnings, or they've disabled strict host key checking and their known hosts file.
The existing SSL cert validation options are perfect if you're a home gamer or you're running a few dozen bare metal servers with all your SSL users within yelling distance in the same office. But we all know it's a joke beyond that.
Well, at least SSH could allow for signing a new key with the old one. So they could say it's signed, and people would know to accept only a different prompt.
There is DNS verification, but people have been trained all their lives to accept insecure DNS information (and set their systems accordingly), and I really doubt the SSH client checks the DNSSEC data.
They don’t need it. Millions of users are going to blindly trust the “new” GitHub public SSH key they see the next time they connect without checking to see if it matches the published signature.
By pretending to be the host that the user is trying to connect to. You can then present the client with a key you generated yourself. Of course, SSH will warn the user that the fingerprint has changed, but they'll just think "Ah yes, GitHub changed their keys so it's probably fine." This is why updating the key creates a potential MITM risk, unless people actually bother to verify that the fingerprint is correct.
What specifically can you verify that cannot also be spoofed? If I go do this now I (and I’m sure millions of others) have no idea what to look for. I’d be blindly accepting like a sheep if it weren’t for this thread.
if you run a MITM attack today the victim gets the warning in the blog post. Thier most likly action is to google the blog post and see it is expected and and so accept your fake key.
Having said that I dont see what choice Github had, they can't continue to use a leeked key.
Their solution is the second thing in the blog post which is demand into your known host file.
The problem here is that their first command that they advise using is the one that removes the old key and most users are just going to stop right there because it solves the problem of getting the key warning.
The right solution here is to provide a command that most users are going to copy and paste that deletes the old key and adds the new key all at once.
The fact that users have to delete the old Github key from their systems and accept a new one is what could lead to a MITM attack.
If your system doesn't know the public key of an SSH server, when you connect the first time, the SSH client will display a warning and ask you if you accept the server key. An attacker could be between you and Github and if you accept without checking it's the correct key, you would be toast.
Yes, GitHub's announcement provides the correct new public RSA key, and it also provides instructions for a curl invocation which does all the work if you don't trust yourself to copy-paste text or don't understand how.
Only if the https server cert wasn't compromised at the same time as the ssh key. For all we know, this entire announcement of "we have a new key" could be staged.
The key has changed, meaning that every user has to accept a new key.
Meaning that a lot of users will blindly accept whatever new key (even when it might be the one owened by attacker doing MITM) because Github, their college or random person on internet said that that's what you have to do to get rid of error.
> Meaning that a lot of users will blindly accept whatever new key (even when it might be the one owened by attacker doing MITM)
This is less likely because unlike for TOFU the SSH client just rejects the mismatch and insists you take manual action, and the likely manual action will be "Paste this stuff from the announcement".
So an adversary needs to either subvert whatever messaging you see (which is tricky, likely impossible for a random user visiting the github web site wondering what's wrong) or hope that you just try to muddle along and do TOFU again, putting you in the same spot as a whole bunch of users every day at GitHub.
I'm always amazed at this kind of posts. Did these 50 million users (surely none of them use git+https!) check the host key the first time they connected to github? Did you?
Sure, but the difference is that it's now both a plausible moment to go MITM (because they got that key), and furthermore the hypothetical attacker now has good reason to believe users won't be scared by a host-key-change warning, and the hypothetical attacker would know this opportunity exists for a large set of users simultaneously. If some malicious network operator were to try and exploit users, now would be a good moment - they'd likely catch many more people in the time it takes to be discovered than on an average day.
The MITM-at-the-start risk is of course real, but I think this new everyone -restarts-simultaneously risk is qualitatively different enough to be worth at least considering.
Much more concerningly, there is an activated-by-default OpenSSH extension (`UpdateHostKeys`) that allows the server to install new host keys into `.ssh/known_hosts` after every successful server authentication.
The bad guys would also have to have MITMed it every time I connected for the last 15 years, or I would have seen authentication failures when it connected to the real thing. MITMing someone once isn't that hard, but doing it consistently is.
If we're starting from the assumption that the first key was compromised, then you're still vulnerable to MITM. The only solution is communicating the key through a different, trusted way. Which is exactly what github did - inasmuch you can trust that github.com is github.
That's funny, I do it every day. It's frankly easier to install git credential manager (even integrate into WSL) for 2FA authentication on Github (and other git hosts).
I get a bit paranoid when having to deal with Tokens on various CI/CD environments as it stands. And the things that start breaking every year when I forget to update them. Note: this is personal/hobby projects, not corporate stuff, where I'm strictly in the codebase and try to keep my fingers out of CI/CD beyond getting a Docker image built, and someone else configures the keys/auth.
How are you using git credential manager for 2fa on GitHub? They stopped supporting user/password auth for HTTPS git access a while back, and started requiring personal access tokens (which do not require a 2nd factor)
GCM will use an embedded browser so you can authenticate with the UI including your second factor, which will then give you a credential/token that can be used in the git environment over HTTPS. It's still a (differt, oath vs reference generation) token, but you aren't having to go generate, configure and update it yourself.
The point being made here is that because there are millions of users forced to change keys now all at around the same time, and because they are doing so due to seeing an error in Git, this creates a good opportunity to strike. Normally, most users would have connected to GitHub before and a MITM attack has a high chance of failure.
Don't trust corporate PR. They're obviously lying when they say "out of an abundance of caution". The private key was exposed in a public GitHub repo, it could literally be anywhere.
So MITM for some of 50m users is strictly better than MITM for all of 50m users.
> The private key was exposed in a public GitHub repo.
How do you know this?
Github runs scanner for private keys in public and private repos and notifies owner (I did it once so I know ... ;)). So some Github engineer likely would have received such an email if what you say is true. Hilarious.
They provide convenient commands to import the correct keys. It would probably be better to only include the block that contains both the -R and the update command, but at least they do provide them.
It depends on how you found out what the new key value is. By the sounds of your description, you're fine. But in principle there's more than one way people could proceed from here.
If you read the blog post on a verified domain and saw the new key and updated manually, or you deleted the known key and verified the key fingerprint when it warned you about an unknown key, you should be good to go. Here, you trust the people who issue TLS certificates and you trust github to be in control of their domain name, so you can be reasonably confident that the key they advertised on their website is the correct key. If your internet connection was compromised, you would have got an error message when you connected to https://github.blog (because they wouldn't have a certificate from a trusted certificate issuer) or when you connected to the git server (because they wouldn't have the key you just trusted).
If you saw the blog post and then removed the old key and told ssh to save the new key it's receiving without checking it matches the value on the webpage, you might have a problem. The connection to github's ssl could have been compromised, and if you just accepted whatever it told you, you have no trusted intermediate to verify that the key is trustworthy. All you know is that each time you connect to github's server hereafter, you're either connecting to a server with the same key (no error), or you're connecting to one that doesn't have the same key (error message). But whether you can trust that key? You don't know that. You just know it's the same key.
But even if you did the latter, all is not lost. You can look at the known_hosts file (on Linux and MacOS it's ~/.ssh/known_hosts) and check the fingerprint. If it's what they advertise, then you're good to go. If it's different, you should fix it and find people who can help you deal with a security incident.
The reason people are raising a flag is that today, lots of people will be rotating their key. That means if you're looking to target someone, today is the day to do it. Even if 90% of people do it the proper way, by manually verifying the key, that still means there's going to be a lot of people who could be victimised today.
If someone wanted to trick HN users into trusting a phoney key, one way to do that would be to post the phoney fingerprint on HN claiming it to be the real one.
I mean, yes, but you'd also have to have a way to actually MITM the person you are targeting via HN comment, before anyone pointed out it was wrong. It'd be much easier to just use the MITM you already have and not raise the suspicion of posting in a comment.
Why downvote this person! The parent post left plenty of ambiguity in their comment. Are they saying that an actual MITM attack is happening? That the fingerprints shared are actually the wrong ones?
Generally speaking, one would not consider an internet comment directing folks to GitHub's actual SSH fingerprints a "man in the middle" as the phrase in this context usually has a negative implication, where in this case defanor is in fact simply mirroring the actual information that GitHub has officially posted in a way that is much more helpful than yetanotherjosh's "double check it is the expected value". For most of us idiots, we don't know what the expected value is!
So thank you defanor for sharing, and thank you darthrupert for asking for clarification. Y'all contributed to educating myself and others and now we know more because of it.
Ah, okay. I thought this was obvious that the keys in the comment were just for show, and if anyone would need the actual keys, they would be looked via the GH link anyway.
On the other hand, this is a nice TOFU-style double check. The first time HN user "defanor" went to that page, these were the fingerprints; if later someone somehow invades the github documentation server (or somehow MITMs your HTTPS connection to it), and changes the fingerprints there, they will no longer match the ones saved in the comment above.
Indeed, at least for verification. I didn't mean for HN users to trust those, but perhaps should have warned about it: copied the fingerprints primarily for people searching on this page, so that they can follow the link to GitHub (and rely on PKIX to build a trust chain). I did `ssh-keygen -R github.com` myself, and saw the ECDSA key's fingerprint while connecting (which wasn't mentioned in the linked post, and wasn't on this page either), so figured it would be somewhat helpful for others following the same route.
Not all of us are familiar enough with the SSH protocol to understand how to "double check the expected value"? Where can I determine what the expected value should be?
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
Note that the SHA256 present there matches perfectly the one github send. If you don't remember the very first time you connected to github you also had to accept the key. The warning above shows up because the server is saved as a different RSA, for the SSH client it seems that someone setup a server for github but has a different key, which could mean someone is trying to trick you into connecting into the wrong server. This could also mean that github changed their RSA key which is why they published this article.
The fingerprint is a hash of the key, so in theory -- say with a quantum computer -- I could create a key that's different and provides a hash-collision. Is that right?
It would just take many ages of the universe, at present, to calculate a collision, right?
There's a narrow window for that attack. The fingerprint is only used on the first connection, for manual verification. Any later connections would check the ~/.ssh/known_hosts which has the full public key.
If you somehow can MITM an SSH connection on the first connection, you can probably use any key. Most people don't check the fingerprint.
But you are correct, computing an SSH key with a collisionwis expected to take an infeasible amount of time/energy with current understanding of crypto and available computers.
We should have a blockchain for certificates btw. That would be such an amazing solution to this problem. You could advertise ahead of the time that you are changing certificates and we could verify that it was in fact you.
Double-check with what source? The one mentionned in docs.github.com?
I assume it's safe because the SSL cert for docs.github.com is probably not compromised, so it's giving us the right key, and compromising docs.github.com would be extra effort and is unlikely to happen.
However, I wonder what kind of steps an MITM attack would have to perform, I assume one of the easiest would be compromising my local DNS server, since regular DNS does not offer a high level of security, then github.com resolves to the attacker's IP and the attack works. Do you have examples of such attacks that don't involve a virus already active on the end user's PC? Maybe if someone owns an IP previously owned by Github that is still somehow advertised as being Github by some DNS lagging behing?
This is always a concern with SSH as it uses trust on first use. The first time you connect it shows you the fingerprint for out of band verification. That manual step is on you to perform, but most people skip it. Future visits check against the saved fingerprint.
The best practice is to verify the fingerprint out of band using a secure channel. In this case, that's HTTPS and docs.github.com. If (hypothetically) docs.github.com was also compromised, then you don't have a secure channel.
If an attacker has access to the private key, they could use the Host-key rotation feature to migrate you to an attacker-controlled key instead, as the old key is trusted. So, GitHub needs everyone to forcibly untrust the old (exposed) key.
The problem with rotating this particular private key is that it's incredibly disruptive. Everyone who uses GH will see a big scary message from ssh saying the host key changed and something malicious might be going on. A majority of those people probably won't have seen a blog post announcing the change beforehand.
Anyone who's baked the host key in the known_hosts file that gets shipped on their CI systems would start to see jobs failing, and have to manually fix it up with the new host key.
These things are just annoying enough that I think it's perfectly understandable that GH doesn't want to regularly rotate this private key.
Host-key rotation would enable the attacker to continue, but the attacker could be detected simply by diligent people monitoring the github key they use.
The current rotation allows anyone to try to fish the lazy users (like me probably) who will just trust on first use. Probably a bigger risk than key compromise, since they have logs.
It could be a better idea to use Host-key rotation, despite it making the life of a key-thief a bit easier. Just because it exposes people less against opportunistic impersonators.
1. IIRC UpdateHostKeys does not remove the old key, so it would still be there, lurking (I haven't checked the code).
2. It was only added in OpenSSH 6.8, so it missed Ubuntu 14.04 release, and only really turned up in 16.04 LTS that way, plenty of old systems it wouldn't work on.
As other posters noted, a bad actor could rotate the key to their chosen keys just as easily as GitHub could cause the rotation.
I saw once a support person from $COMPANY_I_WORK_IN ask a client to send their private key in email, so they could troubleshoot something. I pressed "Reply all" and called it out, and was yelled at later.
At the end of the day human power structures and trust are heavily important. We are trusting their site didn't get hacked, and if it did that we would find out about it. I mean if you can't trust anything at all, the modern internet is unusable.
Is it weird that this wasn’t in a TPM somewhere? I don’t know how host keys work, are they rotated based on some root cert regularly or is it just the same keypair since the creation of GitHub?
Yes, with ssh you don't use CAs, or certificates at all; just raw asymmetric keys. It does support certificates with all their bells and whistles though, just nobody does really...
Not really weird, no. The host key needs to be used every time someone establishes a SSH connection to Github. HSMs can't handle that volume of requests.
Given that the key is extremely long lived, this has unfortunate implications: If any of these servers is compromised, or decides to go randomly spewing memory content because of a bitflip, or screws up the nonce on a DSA/ECDSA operation, the key can be compromised. This is hard to exploit if you're a random person, but for a global adversary that collects internet traffic at scale (e.g. NSA), it's feasible and I would be surprised if they weren't exploiting such issues. They were collecting HTTPS handshakes for a reason.
Ssh keys are much simpler than the PKI cert infrastructure. This has advantages for development, as well as bug resistance. But it means ssh expects every server behind the same hostname to use the same SSH key.
You can't get two (well functioning) TPMs to have the same key. They come with their own, un-extractable* and unchangeable keys built in.
*TPMs claim this, it is probably not impossible to extract keys just incredibly difficult and requiring specialist knowledge.
There is a diference between the "root" key and the keys you can load into a TPM or HSM. The former is sort of built in, but you can provision several TPM/HSMs with loadable keys and then use those for your crypto implementations - otherwise, every time a TPM/HSM broke, you'd risk data loss.
Are they confident that the private key was not seen by anyone, or did it actually leak? If it leaked, this may be more of an issue than how it appears to be presented.
If you ever push anything to a public Github repository it is immediately mirrored by a host of bots. If it was there for any amount of time, it's there. That said, if I'm honest I think almost no one checks the SSH host key. I would bet that for 9 out of 10 engineers / devops / security people, they will ignore the SSH Host Key and might even set StrictHostKeyChecking=no.
I know I know. All you have to do is to just MITM someone's DNS or IP traffic or whatever and it's all over. Well, I will guarantee you that millions of bitcoin are sitting there waiting for you to take it. It's all yours. Go take it. You could be a millionaire tomorrow. And if you do it right, they'll let you keep some of it if you send most of it back.
> If you ever push anything to a public Github repository it is immediately mirrored by a host of bots.
I'd love to get a copy of the key, so I can setup a demo to encourage education.
> That said, if I'm honest I think almost no one checks the SSH host key. I would bet that for 9 out of 10 engineers / devops / security people, they will ignore the SSH Host Key and might even set StrictHostKeyChecking=no.
I believe you, and I agree this is common. I have been carrying the github host keys in my dot files for more than a decade. I do this somewhat ergonomically by having a Host config that adjusts the known_hosts path to permanent_known_hosts for things I care about.
Setting StrictHostKeyChecking=no is negligence, despite being common there really isn't two ways about that, it's a trivial vector with only a very moderate need for additional information to turn this into an RCE in most cases.
There are big problems with ssh's host keys, I understand and agree, but there are also big problems with passwords and we've made a dent in getting people to use password managers and 2FA. We should not wave this away just because it's a little harder, otherwise we're just relegating ssh to the same status as gpg.
Because "Enterprise". Some C-Level read about Cyber in an inflight magazine and decided "The Firewall" needs to be "locked down" to only allow essential traffic. So https it is!
I would presume because unless you control the GitHub account and the SSH key generation process (making sure to generate on smartcard), any developer can upload any old public key, and then do something like... commit it to a public git repo.
If you're logged in and have a SSH key added to your account, I believe Github UI will show you SSH clone command by default. Therefore I always clone with SSH, even public repos.
You may be reading too much into this ... it was a joke :)
But you've made me curious, what's your workflow like? Do you use a gui git client? For me, the default option provided to me by github is an ssh clone, at least for repos that I can own/can push to. This makes it very convenient to work with, because as long as I'm logged into a shell, I'm authenticated to github. But then again, I almost exclusively use the cli and have an ssh key configured for my gh user...
Indeed. One would assume such a private key to be deeply stashed away behind multiple security borders. Ending up in a GitHub repository seems to imply developers at GitHub somehow had access to it.
Big oof from an ops standpoint. How is a key like this ever accessible to an operator in unencrypted text form?
Funny enough, given that every person (and every bit of tooling) I know blindly approves SSH public key verification anyways, they will likely not even notice this switch.
It's not about the certs. To execute a man-in-the-middle attack the attacker has to literally put themselves in the middle of the route your packet takes to get to github's servers and intercept it.
It could be worse, the user has the old rsa host key present alongside newer ed / ecdsa keys, they may never rotate out the rsa one. A future mitm simply only advertises the rsa key, and mitm passes.
Users will need to actively remove the old rsa key in order to be safe. It's my first question, and a colleague suggested that they believe that the private key was not seen, however, I don't see that in the post - unless I'm missing it - and I really want this stated very clearly somewhere.
I tested this and on a new enough OpenSSH client, the RSA key gets replaced using the mechanism described here: https://lwn.net/Articles/637156/ (if you connect using a key other than RSA).
To be honest, I'd expect something like this to be mentioned in the announcement.
You're absolutely right. It's absurd to frame it this way.
Do they expect people to think "Wow, Github leaked a key, but even without knowing if anyone snagged it, they're still replacing it. Wow, they go above and beyond."
So... uh... why was their private key in a form that could be stored in a Git repo? In the old days, we used to put private keys in hardware co-processors that would never reveal an un-encrypted private key. I guess we don't do that anymore.
Imagine the amount of ssh connections that github must handle every second, I don't think it's so easy to hold the private key in a HSM while maintaining the necessary performance and availability.
Loading it from HSM to memory/keychain is probably fine too. It's certainly odd it found its way to a repo and makes you wonder how that could have even possibly happened. And what that indicates about their security practices in general.
Github is host to a large percent of US tech IP. Pretty concerning if you extrapolate.
The whole point of a HSM is to never reveal the private key, but instead to have the HSM perform the necessary crypto operation for each request. This wouldn't scale to the amount of ssh traffic I would imagine github sees.
I think it's completely reasonable that they have the ssh private key in some sort of configuration management repository, because they need to be able to deploy that key to all their public facing ssh servers. You would hope they would have more than 1 ssh server instance world wide for availability and resiliency.
> The whole point of a HSM is to never reveal the private key
That's not "the whole point". You can have methods to copy a key between HSMs, methods that can scale just fine while being much more secure than a file in a repository.
Or you could just buy HSMs that can do the number of private key operations per second that you require.
I've never seen the inside of GH's network, but I would be surprised to discover they're not distributing the load of SSH termination across a fleet of machines. Just put a HSM on each of the machines that terminates a SSH connection.
HSM is just a computer. May be with crypto accelerators. You can set up OpenBSD boxes, put them into safes and disable any incoming connections other than your custom HSM protocol and that would work better than storing private keys on web frontends.
Yes, it makes things hard and unconventional to set up. But GitHub is not some small website.
That's not all of what an HSM is. Or should be. The beefier ones come with rf shielding to prevent bad guys from trying to reduce the key search space by listening to EM energy coming out of the box. And active key zeroation if it thinks you're trying to drill through the epoxy surrounding the crypto boundary.
Is it technically possible (and/or wise) to duplicate the key across multiple HSMs running in parallel? I'm guessing if you have a super massive CDN like Apple or Meta this is a necessary thing.
>And active key zeroation if it thinks you're trying to drill through the epoxy surrounding the crypto boundary.
Or passive! Probably it wasn't a real product but I recall reading about one that derived its key from the field generated by randomly arranged magnetic particles in the resin, or something like that. The point was to make it impossible to disturb the resin without altering the key.
That sounds like an engineering problem. I have my political opinions on Meta as much as the next person, but every bit of evidence I've seen so far is they take this kind of security thing seriously and they definitely operate on a larger scale than github, although most of their traffic is HTTPS not SSH.
> I have my political opinions on Meta as much as the next person, but every bit of evidence I've seen so far is they take this kind of security thing seriously
HSMs don't handle the entire load of the the connection, only the private key operations. At the 10k per sec sites I worked at, we attached some beefy HSMs to the few (less than 40) machines that terminated TLS.
In fairness, it's better that they rotate the key immediately before even looking at the logs. But we can demand better answers from them from this point.
This seems a reasonable timeframe to me given the scale of the impact.
Security issues are obviously very important, but equally I would be worried if they started acting impulsively immediately after discovering the issue, because there would not have been time to understand the problem and consider the options. If you jump on a solution without giving it a bit of thought first it's too easy to exacerbate the situation.
Its not the time from discovery to rotating the key thats OPs issue. What was the time from key being public until discovery? Thats the issue. And also, while the key was public, did anyone access it?
Maybe I'm missing something, but since all GitHub events can be watched through the API, I don't think it particularly matters how long it was public. Anecdotal but I once accidentally pushed an AWS key (thankfully heavily locked down and not a root account) for all of 30 seconds and it was compromised anyways.
I agree. Even if its only public for one sec it should be considered compromised. However, everything is a risk management question, so knowing the amount of time it was exposed is helpful for other orgs to determine their response to this incident. Same if there was an accesslog.
> How long was it visible for? And no, "briefly" doesn't cut it.
I don't know how long exactly, but in theory you can subscribe to a stream of ALL events happening at GitHub by fetching from this endpoint: https://api.github.com/events
With these events you know what new repositories are created and what changes are pushed, so you can fetch every new change. Once something gets pushed to a public repository, it's very likely that some spider will have fetched it within a few minutes.
Especially something easily identifiable as a SSH private key, you get emails from a variety of security vendors to the address associated with the commit offering their services
Your comment made me curious. I downloaded the complete last month (February). 28 days, every day has a gzipped json file which is on average 86MB. Summarized it is 53GB of gzipped jsons for 28 days. Roughly 700GB gzipped per year. Didn't have the chance to ungzip them all.
> We have no reason to believe that the exposed key was abused and took this action out of an abundance of caution.
This is not verifiable, right? As the authentication method has no public and required revocation source, and given that the key, if having leaked, likely will be acquired by an authoritarian government, they can selectively MITM organizations and users that are not aware of this blog post.
The discrepancy between "briefly" and "this week" stood out to me. It reads like multiple hours or even days would classify as "briefly" here where in this context it would otherwise usually mean in the seconds or maaaaybe single-digit minutes, but even that would be stretching it.
If you don't pre-bake the known-hosts, then you'd allow each new ephemeral run to use whatever github tells you it's key is.
It did briefly break ours, as we pre-bake the known_hosts file into our CI image for convenience and security.
Convenience due to CI not having TTY's, so various tools would get stuck on prompt Y when connecting to github for the first time. (Which is every run, if you are ephemeral CI)
And security, as now everything broke due to github's key changing, which is the desired outcome actually.
We bumped the known_host key entry, merged and all is well again...
You shouldn't start with a blank state, instead you should be querying https://api.github.com/meta . But there are so many repos on Github itself which hardcode the host keys in Github Actions, etc.
> Could anyone with that key have cloned any GitHub repo, private or not?
If you try to push to GitHub but someone is MITMing you with the leaked key, they can say "I'm an empty repo" and the client will helpfully push all the commits to the attacker.
Until everyone updates their .ssh/known_hosts file, a MITM attack can still steal code in this way.
Github had one RSA ssh host key, the most widely supported key format.
It was trusted to clone code into the infrastructure of hundreds of thousands of organizations. People pin it everywhere.
With this key someone could infect the infrastructure of fintech companies and steal billions of dollars. I know this well because I run a security consulting company focusing mostly on that industry. Mainly this is possible because almost no companies check for git commit signing, and Github does not enforce it anyway, and I digress.
This key held enough power over value that some might have used violence to acquire it.
With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right?
Nope. They admit they just stuck it in a git repo in plain text where any engineer could have copied it to a compromised workstation, or been bribed for it, for who knows how many years. Worse, it was not even some separate employee only intranet git repo, but their own regular public production infra and someone had the power to accidentally make it public.
I have no words for this level of gross negligence.
Maybe, just maybe, centralizing all trust and security for most of the worlds software development to a proprietary for-profit company with an abysmal security reputation was not the best plan.
> With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right?
This is unfortunately not how SSH works. It needs to be unlocked for every incoming connection.
You raise valid hypotheticals about the security of the service... but fixing it involves removing SSH host key verification from Github; better OpsSec would not fully resolve this issue.
Hardware security modules can perform key operations without allowing anyone to access the key data. Key material being (accidentally or deliberately) leaked has been a solved problem for a long, long time.
Used in such a way, the entirety of Github's SSH connections would be bottlenecked by this HSM. It wouldn't scale and it would be a single point of failure. As lrvick points out, you'd have to use a certificate-based host key scheme like PKCS#11 to make this scalable. That's fine, but it is a different scheme than RSA host key identification.
PKCS#11 is a protocol for talking to hardware security modules for generic cryptographic operations to keep a key out of system memory.
You can totally take a single host key scheme and use HSMs or TEEs at scale.
Openssh supports using openssl as a backend which in turn can use the PKCS#11 protocol to delegate private key operations to remote enclaves or TEEs. Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else.
> Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else.
Isn't having one key the root of the problem? If you've got one key, it has to be transferred to each HSM module, which means it's going to be in their deploy code. My understanding is that the safe way to scale PKI is to have an HSM generate a key internally, and have that key be signed by a CA in a different HSM, so private key material never leaves any HSM.
I guess you're saying that SSH RSA host keys support this, but I'm only familiar with doing it using looks up correct terminology X.509 certificates like for HTTPS.
> If you've got one key, it has to be transferred to each HSM module, which means it's going to be in their deploy code
There's products out there that allow for keys to be securely shared amongst multiple HSMs, without the key ever existing in clear text outside the HSM.
I am well aware how ssh works. I have written ssh servers and design secure enclave key management solutions for a living.
Even if they wanted the most quick and dirty lazy option with no policy management, they could stick the key in a PKCS#11 supporting enclave every cloud provider supports these days. OpenSSH natively supports them today.
At a minimum they could have placed their whole ssh connection termination engine in a immutable read only and remotely attestable system like a Nitro enclave or other TEE. You do not need to invent anything new for this.
There are just no excuses here for a company with their size and resources, because I do this stuff all the time as just one guy.
Would these secure storage methods for the key be able to scale up to the ssh connection volume an outfit like GH sees? Genuinely asking; I don't know the answer. I just feel like having to hit a HSM or something similar every time a new ssh connection comes in wouldn't work. Or at the very least I would not see the sub-100ms connection time I can get right now.
Easily. You could have only the CA key in the enclave and have it sign throw-away session keys on stateless autoscaling machines, and/or you can have your actual ssh terminations happen in cloud enclaves like nitro enclaves which have all the same specs and cost of a regular cloud server.
Wow, your assessment of the impact here (or even possible impact) is way way way overblown.
In reality, the vast majority of users don't pay attention to SSH host keys at all.
Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections of their target.
Next, they have to decide what they want to do.
If they want to send malicious code to someone doing a 'git pull', they'd have to craft the payload specifically for the repo being pulled from. Not impossible, but difficult.
If they want to "steal" source code from someone doing 'git push' (perhaps to a private repo on GitHub), that's a bit easier, as they can just tell the client "I have no objects yet", and then the client will send the entire repo.
And, again, they'd have to have the ability to MitM some git user's connections. Regardless, there is no way that they could change any code on github.com; this key would not give them any access to GH's services that they don't already have.
So I think your anger here is a bit over the top and unnecessary.
I agree that it's pretty bad that some GH employee even had the ability to get hold of this private key (sure, probably should be in an HSM, but I'm not particularly surprised it's not) in order to accidentally leak it, but... shit happens. They admitted the mistake and rotated the key, even though it's likely that there was zero impact from all this.
End users do not pay attention but their clients pin the key after first use. Also everyone is using gitops these days and almost no one is using dns-over-tls.
Imagine you control the right router on a company wifi, or any home wifi a production engineer works from and suddenly you can cause them to clone the wrong git submodule, the wrong go package, or the wrong terraform config.
If you knew a CI/CD system blindly clones and deploys git repos to prod without signature checks, and that prod is a top 10 crypto exchange with 1b of liquidity in hot wallets, then suddenly a BGP attack to redirect DNS is a good investment. Myetherwallet got taken over for 15 minutes with a BGP so this is not hypothetical.
Should that be the case? Of course not. But the reality is I find this in almost all of the security audits I do for fintech companies. Blind trust in Github host keys is industry standard all the way to prod.
Sure I can imagine that. And in doing so, I imagine this attack is pretty unlikely.
I mean, think of the confluence of things that have to line up for this to work for someone. Many stars have to align in order for someone to successfully exploit this leak. Is it impossible? No, of course not, and so GH was right to rotate the key here, even if their server request logs suggested that no one had accessed it.
If people actually have this sort of attack in their threat model, there are ways to protect against it. Signing commits and verifying them. Pinning to a particular git SHA1 hash. Etc. If people are not doing these things, then it's possible they've made the decision not to worry about this sort of attack. Sure, you can disagree with, but I think you'd probably be in the minority. That's ok, though; you can certainly protect the things you're responsible for in stronger ways.
I have seen attacks along the lines I just outlined and well beyond in my industry many times.
A wildcard TLS cert or an SSH host key in plaintext is a loaded weapon laying around and it will be used on a high value target.
Sad fact is most of the companies that hold billions of dollars of customer assets do development exactly the same way a gaming company might. Those that even attempt security are unicorns. They bank everything on things like DNS, BGP, TLS certs, and ssh host keys working. This is like the medical industry before they learned hand washing was a thing.
I teach every one of my clients how to do things like commit signing, but for every one I help improve there are 100 more on track to be hacked any day now.
I can totally forgive a startup that cannot afford senior security engineers for a mistake like this, but Microsoft can afford that, or at least consultants, and yet they can not even enable optional signing for NPM, properly enforce git commit signing, or even protect an ssh host key trusted by millions in a TEE or HSM.
Yes, but they will have to un-pin the (now compromised) key if they want to continue using Github.
Any compromise would have to isolate them from the "real" Github hosts from today onwards, i.e. plausibly MITM them continuously, or they would just switch to the rotated key to be able to continue working. At least in OpenSSH, this means replacing the compromised trusted key, as there can only be one per host (or even IP in the default configuration).
This is still very bad, but much less catastrophic than e.g. a world in which `.ssh/known_hosts` allows multiple entries, in which case you'd really have a sustained compromise of most clients.
Woah, I looked around some more and it seems like the opposite is true. Multiple trusted keys per domain can exist, and additionally there is this option:
> UpdateHostKeys is enabled by default if the user has not overridden the default UserKnownHostsFile setting and has not enabled VerifyHostKeyDNS, otherwise UpdateHostKeys will be set to no.
This is an OpenSSH extension that allows any host to provide additional host keys the client will then silently add to `known_hosts`. This is really bad in this context as it can allow a one-time MITM to install rogue keys for github.com that can go unnoticed; check your `known_hosts` for any unexpected entries for github.com!
> Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections
This is not hard. If it were hard, we wouldn't need encrypted connections.
If I were a nation state, it would be trivial to position an attack at the edge of a network peering point for a given service. How do I know? Our own government did it for 10+ years.
Cyber criminals often have the same tactics and skills and can find significant monetary reasons to assume heightened risk in order to pull off a compromise.
Random black hats enjoy the challenge of compromising high value targets and can find numerous ways to creatively infiltrate networks to perform additional attacks.
Even without gaining access to a network, virtually anyone on the internet can simply push a bad BGP config and capture traffic for an arbitrary target. Weird routing routinely happens that nobody can definitely say isn't such an attack.
The same ones that terminate TLS for millions. Most are garbage but at least they keep the key offline. Also you can scale horizontally or only use the HSM as a CA to sign short lived host keys.
You could also use things like Nitro enclaves which have all the same specs as a regular EC2 instance.
Take Utimaco Security Server and you'll get 10k+ RSA signatures per seconds. Yes, you'll definitely need dozens of them, but you'll probably want several for high availability and low latency anyway.
> For SSH? Only by having the same private key in all of them, which means it's still around somewhere.
End-to-End encrypted key transfer between HSMs is well established. Ops for such a setup is definitely going to be a pain and lots of manual (and thus expensive) work but it is doable. The banking industry has been operating like that since forever – with symmetric cryptography only. Imagine two people being sent letters with XOR halves of a transport key and physically meeting at an HSM and entering the halves on a PIN pad (not a hexadecimal but a decimal PIN pad, mind you, where you need to press shift for A-F). From a modern perspective it's totally bonkers, but it works.
If I was tasked with building something for large scale companies like GitHub, I would probably pass up on HSMs and use commodity measured boot on a minimal Linux for my trusted key servers. Outside of these key servers the SSH key would only be stored split up with Shamir-Secret-Sharing with few trusted employees which will only restore the key on ephemeral offline systems. Is that overkill? Very much depends on your threat model. Investing in the security of their build chain runners and data-at-rest integrity might have higher pay off. But then again, GitHub has become such a big player that we should also hold them to very high standards. And the setup can be re-used for all their secret management, e.g. TLS certificate private keys.
Thanks! That's hugely helpful - everything I could find myself seemed to be an order of magnitude or so slower than that (and they support ecdsa, too, so I can't even object on the basis of algorithmic support). With hindsight my reply was somewhat flippant - really I just wanted to push back on the idea that this was a problem that could be solved by simply sprinkling HSMs on it rather than something that requires a meaningful infastructural effort. Github is a sufficiently core piece of infrastructure that I agree more should be expected, and I hope this does serve as encouragement for them to do that.
The key has to be in memory on all of their front end servers. Do you think a quorum of engineers should get together every time a front end server boots or reboots?
Genuinely asking because I’ve struggled with this question.
I don't understand how initializing cryptographic keys from an HSM at boot time is an untenable proposition. The quorum would be for accessing the key by human means. You can have a separate, approved path for pre-authorized machines to access cryptographic primitives across an isolated network.
Lots of cloud instances support remote attestation these days which gives you a reasonable path to autoscaling secure enclaves.
1. You compile a deterministic unikernel appliance-style linux kernel with a bare bones init system
2. You deploy it to a system that supports remote attestation like a nitro enclave.
3. It boots and generates a random ephemeral key
4. m-of-n engineers compile the image themselves, get the same hash, and verify the remote attestation proof confirming the system is running the bit-for-bit trusted image
5. m-of-n engineers encrypt and submit shamirs secret shares of the really important private key that needs protecting
6. key is reconstituted in memory of enclave and can start taking requests
7. Traffic goes up and autoscaling is triggered
8. New system boots with an identical account, role, and boot image to the first manually provisioned enclave
9. First enclave (with hot key) remotely attests the new enclave and obtains its ephemeral key (with help of an internet connected coordinator)
10. First enclave encrypts hot key to new autoscaled enclave
The key doesn't have to in memory on all of their front end servers. Any respectable company that cares about security wouldn't put their TLS private key on all of their front end servers anyways. You expose a special crypto oracle that your front end servers talk to; the oracle can be a specially hardened process on a dedicated server or better yet a HSM; the point is, the private key is never in memory on any server that handles untrusted data.
Thoughts with the sysadmins and devops people out there on this wonderful Friday afternoon.
These kinds of changes suuuuuuck. Messing with known_hosts file is not always something easy to do. Might require a image rebuild, if you have access at all.
Is it negligence or just incompetence? I get the sense that security is such a tough problem that all of us, even CISOs and red teamers, are incompetent.
If hospital workers spread disease because they could not be bothered to do the obvious things we -know- prevent this like basic sanitation... then yeah, I would call it negligence.
Do not put long lived cryptographic key material in the memory of an internet connected system. Ever. It is a really easy to understand rule.
In an ideal world this could be avoided by using SSH certificates - an assertion by a trusted party that the key presented by an SSH server is legitimate. The signer of those certificates could be in an HSM, ensuring that the private keys can never be leaked, and the certificates could be sufficiently short-lived that there'd be no need to deal with revocation.
Unfortunately there's no way to make this easy. To associate a CA key with a specific domain you need to add something like:
@cert-authority github.com (key)
to known_hosts, and this (as far as I know) can't currently be managed in the same way that keys are generally added to known_hosts on first use when you ssh to something. So instead we're left with every github.com SSH frontend using the same private keys, and those can't be in an HSM because it simply wouldn't scale to the required load. Which means the private key exists in a form where this kind of thing can occur.
> every github.com SSH frontend using the same private keys
HTTPS has the same problem. Compromise any frontend, and you can MITM traffic to any other front-end till the expiry of the certificate (usually 90 days).
I would really like to see some kind of almost realtime delegation - so that for example, every second a new sub certificate could be generated and signed by a HSM, valid for only 2 seconds.
Clock skew unfortunately makes that scale a little impractical, but yeah it would be great to have mechanisms to delegate endpoint certificate issuance to much shorter timescales
Lots of bad things happen if your front-end is compromised, that's different and it's a high bar. They can persist access with a backdoor. They can exfil historical data, password hashes. They can corrupt and modify data.
With HTTPS certs you usually have at most 90 days of impact when a key is leaked (less if you revoke and software is checking CRL). GitHub used the same RSA key for over a decade, they may have continued using this key for quite some time more had they not noticed the leak this week.
Reading the comments here, I can see several things GitHub could have done to improve the security around their host key. But pragmatically speaking, any change involves changing the host key, which is an extremely disruptive operation. Now that they've bit the bullet and forced everyone to rotate, are they going to use this time to actually generate a new, secured key that never touches persistent memory unencrypted?
Of course not. This is Microsoft we are talking about.
If they did something that expensive for security they would be bragging about it from their blog to earn any possible trust they could at a time when they could really use it.
If you are reading this Microsoft, make me look silly. Please.
I will even give you the enclave designs for free.
500 comments
[ 3.4 ms ] story [ 339 ms ] thread"We have no reason to believe that the exposed key was abused, but out of an abundance of caution, we are going to expose 50 million users to a potential MITM attack unless they are extremely careful."
Not a single word in the post about whether this impact was even considered when making the decision to update the key. Just swap the key, and fuck the consequences. Same with the mass password resets after a compromise that some services have done in the past years. Each of those is any phishing operation's dream come true.
Why would that be unknown? GitHub has HTTP and SSH access logs, right?
Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
Or does it log all the requests all the time as long as everything goes well, but if the wrong server crashes at the wrong time, a couple hundred requests may get lost because in the end who cares?
You'd have to have logging set up for all of these services and it would have to work over your entire CDN... and what if a CDN node crashed before it was able to submit the log entry to the log collector? You'll never know.
If train A leaves New York going 60 miles per hour with 167 people on board and train B leaves Chicago one hour later with 361 people on board going 85 miles per hour, how many people now have the key?
The answer is 31337.
To be 100% clear: Both courses of action come with associated security risks. The problem is not choosing one course of action over the other, the problem is thinking you can just skip the cost/benefit analysis because the answer is somehow 'obvious'. It's not obvious at all.
The only way a CBA would be unnecessary is if rotating the key didn't have any security risks. But it does.
- if they have evidence that the key was exposed to one person, even with zero usage of the key, failing to rotate the key is tantamount to knowingly accepting widespread compromise at a potential attacker’s whim. At GitHub’s scale, that’s untenable.
- rotating the key is the only correct reaction to that
- they should have better communications in place to help users mitigate MITM
- there really isn’t an option, because they’re critical infrastructure; I’m glad they know that and acted accordingly
- on principle this speculation makes sense, but understanding the threat makes it moot
- you hopefully know that, and it’s good to insist on thoughtful security practices but it’s important to also understand the actual risk
Thus rotating is the only logical course of action.
If you don't know for certain, you have to factor in the likelihood that it has been, and at that point, the two risks aren't equal anymore so that logic doesn't work.
You just ignore it and hope for the best ?
Only if you are certain (and better be really sure you haven't missed any cache/cdn, temp files backus etc.) it wasn't accessed you do nothing.
You should only keep moving along without key rotation if you know for 100% certainty a leak didn't happen and no one accessed the key (not theoretically impossible if they had the server logs to back it up), but anything minus that and you have to assume it's stolen.
I guess the proper way forward is a small utility that gets the latest signature through http+tls, and replaces the line in your known_hosts file, all in the background.
Looking long term, maybe we need to get rid of all the security stuff in ssh and just pipe the rest of its functionalities inside a TLS pipe. Let the os do its certificate management, reuse security bricks that are way more studied, ...
The real solution to break out of these UX/security tradeoffs is to put domain names on a blockchain: then you can simply rotate the key in your DNS record, while the blockchain model is such that you need to compromise many parties, instead of "one out of many parties", as with CAs.
Tracking Bitcoin chain for DNS updates is lightweight enough that it can be built into OS alongside other modern components such as secure enclave, TCP/IP stack and WiFi/BT/5G radios.
Even with a certificate authority model, you don't have to trust any CAs if you don't want to. Not having the option to do so is more of a problem.
Blockchain is a perfect solution to this. I wonder why it is not considered yet.
Also it's way better than TOFU, you can just add the CA key to known_hosts and avoid TOFU for each machine.
(Nevermind that you'll probably not accidentally commit some semi-ephemeral host key that's rotated often somewhere, because it will not be some special snowflake key you care about, but something handled by your infrastructure software automatically for each machine)
How easy is it to rotate the keys of your CA?
I hope you don't need that SSH connection to fix your broken CRL endpoint!
We need to compare the relative risks within the same context, namely within a company like GitHub
So it's not relevant to bring up failures of other CAs
Jump into a lesser time machine, go back to when Github were creating their SSH key, and put it into a hardware security module. Somehow share that hardware-backed security key to loads of servers over a network, without letting hackers do the same thing. Somehow get an HSM that isn't a closed-source black box from a huge defence contractor riddled with foreign spies. Somehow avoid vendor lock-in or the HSM becoming obsolete. Somehow do this when you're a scrappy startup with barely any free time.
We also have a way to put SSH host key fingerprints in DNS records already.
https://en.wikipedia.org/wiki/SSHFP_record
I'd guess that most systems aren't using DoH/DoT or end-to-end DNSSEC yet. Some browsers do, but that doesn't help tooling frequently used on the command line.
I suppose you could just accept X.509 certificates for some large/enterprise git domains, but that pokes up the hornet's nest that is CA auditing (the browser vendors are having a lot of fun with that, I'm happy that the OpenSSH devs don't have to, yet).
And where do you maintain the list that decides which hosts get to use TOFU and which ones are allowed to provide public keys? Another question very ill-fitted for the OpenSSH dev team.
There could be a well known URL that enables people to fetch ssh keys automatically in a safe manner.
So that any large entity can own your servers with easy. (Well, they already can, but not through this vulnerability.)
Anyway, the only thing CAs do is to move that prompt into another, earlier time. It's the same prompt, the same possibility for MITM, and the same amount of shared trust to get wrong. You just add a 3rd party that you have to trust.
SSH does have a CA system. Anybody that isn't managing a large datacenter will avoid it, for good reason.
Eh, let's not pretend existing SSL certificate validation is anything to write home about.
Even without any ephemeral servers involved, barely anybody is validating cert fingerprints on first use.
And among people using ephemeral servers, 99% of applications have either baked a certificate into their image (so that any compromised host means a compromise of the critical, impossible-to-revoke-or-rotate key) - or every new server gets a new cert and users have either been trained to ignore certificate warnings, or they've disabled strict host key checking and their known hosts file.
The existing SSL cert validation options are perfect if you're a home gamer or you're running a few dozen bare metal servers with all your SSL users within yelling distance in the same office. But we all know it's a joke beyond that.
There is DNS verification, but people have been trained all their lives to accept insecure DNS information (and set their systems accordingly), and I really doubt the SSH client checks the DNSSEC data.
edit: I found this helpful and honestly had no idea I should be doing this (I’m a hobbiest not a professional) https://bitlaunch.io/blog/how-to-check-your-ssh-key-fingerpr...
The problem here is that their first command that they advise using is the one that removes the old key and most users are just going to stop right there because it solves the problem of getting the key warning.
The right solution here is to provide a command that most users are going to copy and paste that deletes the old key and adds the new key all at once.
If your system doesn't know the public key of an SSH server, when you connect the first time, the SSH client will display a warning and ask you if you accept the server key. An attacker could be between you and Github and if you accept without checking it's the correct key, you would be toast.
Meaning that a lot of users will blindly accept whatever new key (even when it might be the one owened by attacker doing MITM) because Github, their college or random person on internet said that that's what you have to do to get rid of error.
This is less likely because unlike for TOFU the SSH client just rejects the mismatch and insists you take manual action, and the likely manual action will be "Paste this stuff from the announcement".
So an adversary needs to either subvert whatever messaging you see (which is tricky, likely impossible for a random user visiting the github web site wondering what's wrong) or hope that you just try to muddle along and do TOFU again, putting you in the same spot as a whole bunch of users every day at GitHub.
If the key was breached and Github just didn't know it, then a breach happened, then only Github would be to blame.
If Github rotates its key, and somebody suffers a MITM attack, the blame is more diffuse. Why didn't they verify the key out of band?
The MITM-at-the-start risk is of course real, but I think this new everyone -restarts-simultaneously risk is qualitatively different enough to be worth at least considering.
One way to solve this in TOFU is to have a time window where both keys are presented.
well, yes
github doesn't accept https push any more
Maybe you’re referring to how they no longer accept passwords for HTTPS auth? You have to auth for HTTPS push with a personal access token.
I get a bit paranoid when having to deal with Tokens on various CI/CD environments as it stands. And the things that start breaking every year when I forget to update them. Note: this is personal/hobby projects, not corporate stuff, where I'm strictly in the codebase and try to keep my fingers out of CI/CD beyond getting a Docker image built, and someone else configures the keys/auth.
I sure did. Doesn’t everyone?
So MITM for some of 50m users is strictly better than MITM for all of 50m users.
How do you know this?
Github runs scanner for private keys in public and private repos and notifies owner (I did it once so I know ... ;)). So some Github engineer likely would have received such an email if what you say is true. Hilarious.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
Hilarious.
> This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
> leaked in public repo
Me: Yeah, that's why they are doing it.
What do I have to do to ensure I connected to the right server?? I thought just making sure the correct RSA key was in known_hosts would be enough?
If you read the blog post on a verified domain and saw the new key and updated manually, or you deleted the known key and verified the key fingerprint when it warned you about an unknown key, you should be good to go. Here, you trust the people who issue TLS certificates and you trust github to be in control of their domain name, so you can be reasonably confident that the key they advertised on their website is the correct key. If your internet connection was compromised, you would have got an error message when you connected to https://github.blog (because they wouldn't have a certificate from a trusted certificate issuer) or when you connected to the git server (because they wouldn't have the key you just trusted).
If you saw the blog post and then removed the old key and told ssh to save the new key it's receiving without checking it matches the value on the webpage, you might have a problem. The connection to github's ssl could have been compromised, and if you just accepted whatever it told you, you have no trusted intermediate to verify that the key is trustworthy. All you know is that each time you connect to github's server hereafter, you're either connecting to a server with the same key (no error), or you're connecting to one that doesn't have the same key (error message). But whether you can trust that key? You don't know that. You just know it's the same key.
But even if you did the latter, all is not lost. You can look at the known_hosts file (on Linux and MacOS it's ~/.ssh/known_hosts) and check the fingerprint. If it's what they advertise, then you're good to go. If it's different, you should fix it and find people who can help you deal with a security incident.
The reason people are raising a flag is that today, lots of people will be rotating their key. That means if you're looking to target someone, today is the day to do it. Even if 90% of people do it the proper way, by manually verifying the key, that still means there's going to be a lot of people who could be victimised today.
We humans really aren't cut out for this, are we.
Generally speaking, one would not consider an internet comment directing folks to GitHub's actual SSH fingerprints a "man in the middle" as the phrase in this context usually has a negative implication, where in this case defanor is in fact simply mirroring the actual information that GitHub has officially posted in a way that is much more helpful than yetanotherjosh's "double check it is the expected value". For most of us idiots, we don't know what the expected value is!
So thank you defanor for sharing, and thank you darthrupert for asking for clarification. Y'all contributed to educating myself and others and now we know more because of it.
Good clarifications everywhere, yes.
(Not genuinely concerned about this risk, but "Reflections on Trusting Trust" reverberates.)
Not all of us are familiar enough with the SSH protocol to understand how to "double check the expected value"? Where can I determine what the expected value should be?
It should error like this:
Note that the SHA256 present there matches perfectly the one github send. If you don't remember the very first time you connected to github you also had to accept the key. The warning above shows up because the server is saved as a different RSA, for the SSH client it seems that someone setup a server for github but has a different key, which could mean someone is trying to trick you into connecting into the wrong server. This could also mean that github changed their RSA key which is why they published this article.It would just take many ages of the universe, at present, to calculate a collision, right?
If you somehow can MITM an SSH connection on the first connection, you can probably use any key. Most people don't check the fingerprint.
But you are correct, computing an SSH key with a collisionwis expected to take an infeasible amount of time/energy with current understanding of crypto and available computers.
Or the parts below it about updating and verifying in other ways.
https://docs.github.com/en/authentication/keeping-your-accou...
Yes, this assumes the github-hosted docs and your SSL connection to them are not also compromised, but it's far better than not checking at all.
We should have a blockchain for certificates btw. That would be such an amazing solution to this problem. You could advertise ahead of the time that you are changing certificates and we could verify that it was in fact you.
I assume it's safe because the SSL cert for docs.github.com is probably not compromised, so it's giving us the right key, and compromising docs.github.com would be extra effort and is unlikely to happen.
However, I wonder what kind of steps an MITM attack would have to perform, I assume one of the easiest would be compromising my local DNS server, since regular DNS does not offer a high level of security, then github.com resolves to the attacker's IP and the attack works. Do you have examples of such attacks that don't involve a virus already active on the end user's PC? Maybe if someone owns an IP previously owned by Github that is still somehow advertised as being Github by some DNS lagging behing?
The best practice is to verify the fingerprint out of band using a secure channel. In this case, that's HTTPS and docs.github.com. If (hypothetically) docs.github.com was also compromised, then you don't have a secure channel.
https://en.m.wikipedia.org/wiki/Man-in-the-middle_attack has some MITM examples.
https://lwn.net/Articles/637156/
I'm honestly not familiar with anyone actually using host-key rotation?
I mean it seems like its clearly a key that wasn't in an HSM.. and over the lifetime, hundreds? Thousands of Github employees could of accessed it?
Anyone who's baked the host key in the known_hosts file that gets shipped on their CI systems would start to see jobs failing, and have to manually fix it up with the new host key.
These things are just annoying enough that I think it's perfectly understandable that GH doesn't want to regularly rotate this private key.
The current rotation allows anyone to try to fish the lazy users (like me probably) who will just trust on first use. Probably a bigger risk than key compromise, since they have logs.
It could be a better idea to use Host-key rotation, despite it making the life of a key-thief a bit easier. Just because it exposes people less against opportunistic impersonators.
2. It was only added in OpenSSH 6.8, so it missed Ubuntu 14.04 release, and only really turned up in 16.04 LTS that way, plenty of old systems it wouldn't work on.
As other posters noted, a bad actor could rotate the key to their chosen keys just as easily as GitHub could cause the rotation.
It seems like at least a `known_hosts` compromise would be "self-healing" after connecting to the legitimate github.com server once.
Given that the key is extremely long lived, this has unfortunate implications: If any of these servers is compromised, or decides to go randomly spewing memory content because of a bitflip, or screws up the nonce on a DSA/ECDSA operation, the key can be compromised. This is hard to exploit if you're a random person, but for a global adversary that collects internet traffic at scale (e.g. NSA), it's feasible and I would be surprised if they weren't exploiting such issues. They were collecting HTTPS handshakes for a reason.
You can't get two (well functioning) TPMs to have the same key. They come with their own, un-extractable* and unchangeable keys built in.
*TPMs claim this, it is probably not impossible to extract keys just incredibly difficult and requiring specialist knowledge.
All cloud vendors offer the same functionality, if you think about it, so it's not an obscure feature.
If anyone is using a Windows machine with TPM-based bitlocker encryption, you have followed the instructions at https://support.microsoft.com/en-us/windows/back-up-your-bit... I hope?
I know I know. All you have to do is to just MITM someone's DNS or IP traffic or whatever and it's all over. Well, I will guarantee you that millions of bitcoin are sitting there waiting for you to take it. It's all yours. Go take it. You could be a millionaire tomorrow. And if you do it right, they'll let you keep some of it if you send most of it back.
I'd love to get a copy of the key, so I can setup a demo to encourage education.
> That said, if I'm honest I think almost no one checks the SSH host key. I would bet that for 9 out of 10 engineers / devops / security people, they will ignore the SSH Host Key and might even set StrictHostKeyChecking=no.
I believe you, and I agree this is common. I have been carrying the github host keys in my dot files for more than a decade. I do this somewhat ergonomically by having a Host config that adjusts the known_hosts path to permanent_known_hosts for things I care about.
Setting StrictHostKeyChecking=no is negligence, despite being common there really isn't two ways about that, it's a trivial vector with only a very moderate need for additional information to turn this into an RCE in most cases.
There are big problems with ssh's host keys, I understand and agree, but there are also big problems with passwords and we've made a dent in getting people to use password managers and 2FA. We should not wave this away just because it's a little harder, otherwise we're just relegating ssh to the same status as gpg.
Seriously? How that happened is deeply concerning.
And why weren't the other keys exposed?
Most development work, however, uses SSH
The "insteadOf" git config is added to workstations and runners to convert attempted SSH connections over to HTTPS.
Like most corporate environments, “it is what it is” and we do our best to perform our jobs within these constraints.
But you've made me curious, what's your workflow like? Do you use a gui git client? For me, the default option provided to me by github is an ssh clone, at least for repos that I can own/can push to. This makes it very convenient to work with, because as long as I'm logged into a shell, I'm authenticated to github. But then again, I almost exclusively use the cli and have an ssh key configured for my gh user...
I personally never clone using https and I think most of the people I work with do the same
Funny enough, given that every person (and every bit of tooling) I know blindly approves SSH public key verification anyways, they will likely not even notice this switch.
Maybe on initial connect, but who TF ignores that "key has changed danger danger, high voltage" warning? You at least look around and ask a colleague?
1. Various DNS hijacking and cache poisoning attacks 2. three letter agencies in meet me rooms 3. Exploited/hacked routers 4. Wifi hot spots
Users will need to actively remove the old rsa key in order to be safe. It's my first question, and a colleague suggested that they believe that the private key was not seen, however, I don't see that in the post - unless I'm missing it - and I really want this stated very clearly somewhere.
Yea that is my read on it as well. If that is true this is much more severe than the blog post suggests.
To be honest, I'd expect something like this to be mentioned in the announcement.
> GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository
What the... That's not "an abundance of caution". That's the only possible course of action.
Do they expect people to think "Wow, Github leaked a key, but even without knowing if anyone snagged it, they're still replacing it. Wow, they go above and beyond."
It's so ridiculous.
After all it’s them hosting and serving the requests for that (and every other) repo.
Only _reasonable_ course of action. Possible is to do nothing =)
That's the "caution" part.
Github is host to a large percent of US tech IP. Pretty concerning if you extrapolate.
I think it's completely reasonable that they have the ssh private key in some sort of configuration management repository, because they need to be able to deploy that key to all their public facing ssh servers. You would hope they would have more than 1 ssh server instance world wide for availability and resiliency.
That's not "the whole point". You can have methods to copy a key between HSMs, methods that can scale just fine while being much more secure than a file in a repository.
I've never seen the inside of GH's network, but I would be surprised to discover they're not distributing the load of SSH termination across a fleet of machines. Just put a HSM on each of the machines that terminates a SSH connection.
It still shouldn’t have been easily accessible to anyone except the instances running the SSH service.
Yes, it makes things hard and unconventional to set up. But GitHub is not some small website.
Or passive! Probably it wasn't a real product but I recall reading about one that derived its key from the field generated by randomly arranged magnetic particles in the resin, or something like that. The point was to make it impossible to disturb the resin without altering the key.
Meta doesn't own Github, MS does.
- When exactly was the key leaked? Yesterday? A month ago?
- How long was it visible for? And no, "briefly" doesn't cut it.
- Did any internet traffic reach the page while it was exposed? We know you log everything, so it is a yes or no answer.
If any of these answers were pretty, I imagine they would have explicitly included them in the post.
> This week, we discovered...
So they found out about it sometime between Monday and Thursday, and rotated it Thursday evening.
Security issues are obviously very important, but equally I would be worried if they started acting impulsively immediately after discovering the issue, because there would not have been time to understand the problem and consider the options. If you jump on a solution without giving it a bit of thought first it's too easy to exacerbate the situation.
I don't know how long exactly, but in theory you can subscribe to a stream of ALL events happening at GitHub by fetching from this endpoint: https://api.github.com/events
With these events you know what new repositories are created and what changes are pushed, so you can fetch every new change. Once something gets pushed to a public repository, it's very likely that some spider will have fetched it within a few minutes.
Wow I am shocked that they allow "firehose" access not only for free, but without even an API key.
Given enough disk and bandwidth, does this mean you could keep your own copy of all of github? I'd love to be able to grep the whole thing.
Are we talking 1TB/year, 10TB/year, 100TB/year?
"This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository"
> We have no reason to believe that the exposed key was abused and took this action out of an abundance of caution.
This is not verifiable, right? As the authentication method has no public and required revocation source, and given that the key, if having leaked, likely will be acquired by an authoritarian government, they can selectively MITM organizations and users that are not aware of this blog post.
If you don't pre-bake the known-hosts, then you'd allow each new ephemeral run to use whatever github tells you it's key is.
It did briefly break ours, as we pre-bake the known_hosts file into our CI image for convenience and security.
Convenience due to CI not having TTY's, so various tools would get stuck on prompt Y when connecting to github for the first time. (Which is every run, if you are ephemeral CI)
And security, as now everything broke due to github's key changing, which is the desired outcome actually.
We bumped the known_host key entry, merged and all is well again...
> Could anyone with that key have cloned any GitHub repo, private or not?
If you try to push to GitHub but someone is MITMing you with the leaked key, they can say "I'm an empty repo" and the client will helpfully push all the commits to the attacker.
Until everyone updates their .ssh/known_hosts file, a MITM attack can still steal code in this way.
It was trusted to clone code into the infrastructure of hundreds of thousands of organizations. People pin it everywhere.
With this key someone could infect the infrastructure of fintech companies and steal billions of dollars. I know this well because I run a security consulting company focusing mostly on that industry. Mainly this is possible because almost no companies check for git commit signing, and Github does not enforce it anyway, and I digress.
This key held enough power over value that some might have used violence to acquire it.
With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right?
Nope. They admit they just stuck it in a git repo in plain text where any engineer could have copied it to a compromised workstation, or been bribed for it, for who knows how many years. Worse, it was not even some separate employee only intranet git repo, but their own regular public production infra and someone had the power to accidentally make it public.
I have no words for this level of gross negligence.
Maybe, just maybe, centralizing all trust and security for most of the worlds software development to a proprietary for-profit company with an abysmal security reputation was not the best plan.
I will just leave this here: https://sfconservancy.org/blog/2022/jun/30/give-up-github-la...
This is unfortunately not how SSH works. It needs to be unlocked for every incoming connection.
You raise valid hypotheticals about the security of the service... but fixing it involves removing SSH host key verification from Github; better OpsSec would not fully resolve this issue.
PKCS#11 is a protocol for talking to hardware security modules for generic cryptographic operations to keep a key out of system memory.
You can totally take a single host key scheme and use HSMs or TEEs at scale.
Openssh supports using openssl as a backend which in turn can use the PKCS#11 protocol to delegate private key operations to remote enclaves or TEEs. Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else.
Isn't having one key the root of the problem? If you've got one key, it has to be transferred to each HSM module, which means it's going to be in their deploy code. My understanding is that the safe way to scale PKI is to have an HSM generate a key internally, and have that key be signed by a CA in a different HSM, so private key material never leaves any HSM.
I guess you're saying that SSH RSA host keys support this, but I'm only familiar with doing it using looks up correct terminology X.509 certificates like for HTTPS.
E.g. Foo can encrypt the key X with Bars public key and then Bar can import key X.
There's products out there that allow for keys to be securely shared amongst multiple HSMs, without the key ever existing in clear text outside the HSM.
Even if they wanted the most quick and dirty lazy option with no policy management, they could stick the key in a PKCS#11 supporting enclave every cloud provider supports these days. OpenSSH natively supports them today.
At a minimum they could have placed their whole ssh connection termination engine in a immutable read only and remotely attestable system like a Nitro enclave or other TEE. You do not need to invent anything new for this.
There are just no excuses here for a company with their size and resources, because I do this stuff all the time as just one guy.
Yep. Well, certificates exist exactly to bridge the GP's requirement with your reality.
In reality, the vast majority of users don't pay attention to SSH host keys at all.
Even if an attacker got hold of this private host key, they'd have to be able to MitM the connections of their target.
Next, they have to decide what they want to do.
If they want to send malicious code to someone doing a 'git pull', they'd have to craft the payload specifically for the repo being pulled from. Not impossible, but difficult.
If they want to "steal" source code from someone doing 'git push' (perhaps to a private repo on GitHub), that's a bit easier, as they can just tell the client "I have no objects yet", and then the client will send the entire repo.
And, again, they'd have to have the ability to MitM some git user's connections. Regardless, there is no way that they could change any code on github.com; this key would not give them any access to GH's services that they don't already have.
So I think your anger here is a bit over the top and unnecessary.
I agree that it's pretty bad that some GH employee even had the ability to get hold of this private key (sure, probably should be in an HSM, but I'm not particularly surprised it's not) in order to accidentally leak it, but... shit happens. They admitted the mistake and rotated the key, even though it's likely that there was zero impact from all this.
Imagine you control the right router on a company wifi, or any home wifi a production engineer works from and suddenly you can cause them to clone the wrong git submodule, the wrong go package, or the wrong terraform config.
If you knew a CI/CD system blindly clones and deploys git repos to prod without signature checks, and that prod is a top 10 crypto exchange with 1b of liquidity in hot wallets, then suddenly a BGP attack to redirect DNS is a good investment. Myetherwallet got taken over for 15 minutes with a BGP so this is not hypothetical.
Should that be the case? Of course not. But the reality is I find this in almost all of the security audits I do for fintech companies. Blind trust in Github host keys is industry standard all the way to prod.
I mean, think of the confluence of things that have to line up for this to work for someone. Many stars have to align in order for someone to successfully exploit this leak. Is it impossible? No, of course not, and so GH was right to rotate the key here, even if their server request logs suggested that no one had accessed it.
If people actually have this sort of attack in their threat model, there are ways to protect against it. Signing commits and verifying them. Pinning to a particular git SHA1 hash. Etc. If people are not doing these things, then it's possible they've made the decision not to worry about this sort of attack. Sure, you can disagree with, but I think you'd probably be in the minority. That's ok, though; you can certainly protect the things you're responsible for in stronger ways.
A wildcard TLS cert or an SSH host key in plaintext is a loaded weapon laying around and it will be used on a high value target.
Sad fact is most of the companies that hold billions of dollars of customer assets do development exactly the same way a gaming company might. Those that even attempt security are unicorns. They bank everything on things like DNS, BGP, TLS certs, and ssh host keys working. This is like the medical industry before they learned hand washing was a thing.
I teach every one of my clients how to do things like commit signing, but for every one I help improve there are 100 more on track to be hacked any day now.
I can totally forgive a startup that cannot afford senior security engineers for a mistake like this, but Microsoft can afford that, or at least consultants, and yet they can not even enable optional signing for NPM, properly enforce git commit signing, or even protect an ssh host key trusted by millions in a TEE or HSM.
Any compromise would have to isolate them from the "real" Github hosts from today onwards, i.e. plausibly MITM them continuously, or they would just switch to the rotated key to be able to continue working. At least in OpenSSH, this means replacing the compromised trusted key, as there can only be one per host (or even IP in the default configuration).
This is still very bad, but much less catastrophic than e.g. a world in which `.ssh/known_hosts` allows multiple entries, in which case you'd really have a sustained compromise of most clients.
> UpdateHostKeys is enabled by default if the user has not overridden the default UserKnownHostsFile setting and has not enabled VerifyHostKeyDNS, otherwise UpdateHostKeys will be set to no.
This is an OpenSSH extension that allows any host to provide additional host keys the client will then silently add to `known_hosts`. This is really bad in this context as it can allow a one-time MITM to install rogue keys for github.com that can go unnoticed; check your `known_hosts` for any unexpected entries for github.com!
This is not hard. If it were hard, we wouldn't need encrypted connections.
If I were a nation state, it would be trivial to position an attack at the edge of a network peering point for a given service. How do I know? Our own government did it for 10+ years.
Cyber criminals often have the same tactics and skills and can find significant monetary reasons to assume heightened risk in order to pull off a compromise.
Random black hats enjoy the challenge of compromising high value targets and can find numerous ways to creatively infiltrate networks to perform additional attacks.
Even without gaining access to a network, virtually anyone on the internet can simply push a bad BGP config and capture traffic for an arbitrary target. Weird routing routinely happens that nobody can definitely say isn't such an attack.
You could also use things like Nitro enclaves which have all the same specs as a regular EC2 instance.
Tons of options. They clearly chose none of them.
Which ones?
> Also you can scale horizontally
For SSH? Only by having the same private key in all of them, which means it's still around somewhere.
> or only use the HSM as a CA to sign short lived host keys
That would be ideal, except that the user experience around SSH host certificates is currently woeful.
> Which ones?
Take Utimaco Security Server and you'll get 10k+ RSA signatures per seconds. Yes, you'll definitely need dozens of them, but you'll probably want several for high availability and low latency anyway.
> For SSH? Only by having the same private key in all of them, which means it's still around somewhere.
End-to-End encrypted key transfer between HSMs is well established. Ops for such a setup is definitely going to be a pain and lots of manual (and thus expensive) work but it is doable. The banking industry has been operating like that since forever – with symmetric cryptography only. Imagine two people being sent letters with XOR halves of a transport key and physically meeting at an HSM and entering the halves on a PIN pad (not a hexadecimal but a decimal PIN pad, mind you, where you need to press shift for A-F). From a modern perspective it's totally bonkers, but it works.
If I was tasked with building something for large scale companies like GitHub, I would probably pass up on HSMs and use commodity measured boot on a minimal Linux for my trusted key servers. Outside of these key servers the SSH key would only be stored split up with Shamir-Secret-Sharing with few trusted employees which will only restore the key on ephemeral offline systems. Is that overkill? Very much depends on your threat model. Investing in the security of their build chain runners and data-at-rest integrity might have higher pay off. But then again, GitHub has become such a big player that we should also hold them to very high standards. And the setup can be re-used for all their secret management, e.g. TLS certificate private keys.
Genuinely asking because I’ve struggled with this question.
1. You compile a deterministic unikernel appliance-style linux kernel with a bare bones init system
2. You deploy it to a system that supports remote attestation like a nitro enclave.
3. It boots and generates a random ephemeral key
4. m-of-n engineers compile the image themselves, get the same hash, and verify the remote attestation proof confirming the system is running the bit-for-bit trusted image
5. m-of-n engineers encrypt and submit shamirs secret shares of the really important private key that needs protecting
6. key is reconstituted in memory of enclave and can start taking requests
7. Traffic goes up and autoscaling is triggered
8. New system boots with an identical account, role, and boot image to the first manually provisioned enclave
9. First enclave (with hot key) remotely attests the new enclave and obtains its ephemeral key (with help of an internet connected coordinator)
10. First enclave encrypts hot key to new autoscaled enclave
11. rinse/repeat
These kinds of changes suuuuuuck. Messing with known_hosts file is not always something easy to do. Might require a image rebuild, if you have access at all.
Do not put long lived cryptographic key material in the memory of an internet connected system. Ever. It is a really easy to understand rule.
Unfortunately there's no way to make this easy. To associate a CA key with a specific domain you need to add something like:
@cert-authority github.com (key)
to known_hosts, and this (as far as I know) can't currently be managed in the same way that keys are generally added to known_hosts on first use when you ssh to something. So instead we're left with every github.com SSH frontend using the same private keys, and those can't be in an HSM because it simply wouldn't scale to the required load. Which means the private key exists in a form where this kind of thing can occur.
HTTPS has the same problem. Compromise any frontend, and you can MITM traffic to any other front-end till the expiry of the certificate (usually 90 days).
I would really like to see some kind of almost realtime delegation - so that for example, every second a new sub certificate could be generated and signed by a HSM, valid for only 2 seconds.
With HTTPS certs you usually have at most 90 days of impact when a key is leaked (less if you revoke and software is checking CRL). GitHub used the same RSA key for over a decade, they may have continued using this key for quite some time more had they not noticed the leak this week.
If they did something that expensive for security they would be bragging about it from their blog to earn any possible trust they could at a time when they could really use it.
If you are reading this Microsoft, make me look silly. Please.
I will even give you the enclave designs for free.
Do the PR people have technical knowledge or does some tech person write
“I accidentally screwed up and put the key public so we have to replace” and the PR glosses over it and rewrites “Out of an abundance of caution…”?
Here's the list of keys if anyone needs:
https://docs.github.com/en/authentication/keeping-your-accou...
> Host key for github.com has changed and you have requested strict checking.
> Host key verification failed.