GitHub waited 3 months to notify about potential compromise
See full text below:
Hi zwass,
We're writing to let you know that between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC, due to a bug, GitHub Apps were able to generate new scoped installation tokens with elevated permissions. You are an owner of an organization on GitHub with GitHub Apps installed that generated at least one new token during this time period. While we do not have evidence that this bug was maliciously exploited, with our available data, we are not able to determine if a token was generated with elevated permissions.
User privacy and security are essential for maintaining trust, and we want to remain as transparent as possible about events like these. GitHub itself did not experience a compromise or data breach that either created or resulted from this event. Read on for more information.
* What happened? *
GitHub learned via a customer support ticket that GitHub Apps were able to generate scoped installation tokens with elevated permissions. Each of these tokens are valid for up to 1 hour.
GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC.
GitHub Apps generate scoped installation tokens based on the scopes and permissions granted when the GitHub App is installed into a user account or organization. For example, if a GitHub App requests and is granted read permission to issues during installation, the scoped installation token the App generates to function would have `issues:read` permission.
This bug would potentially allow the above installation to generate a token with `issues:write`, an elevation of permission from the granted `issues:read` permission. The bug did not allow a GitHub App to generate a token with additional scopes that were not already granted, such as `discussions:read` in the above example. The bug did not allow a GitHub App to access any repositories that the App did not already have access to.
In order to exploit this bug, the GitHub App author would need to modify their app's code to request elevated permissions on generated tokens.
* What information was involved? *
The following GitHub Apps generated scoped installation tokens during the bug window for your organization(s). We are not able to determine if a token was generated with elevated permissions.
Organization: GitHub Apps <redacted>
* What GitHub is doing *
GitHub immediately began working to fix the bug and started an investigation into the potential impact. However due to the scale and complexity of GitHub Apps and their short-lived tokens, we were unable to determine whether this bug was ever exploited.
We are notifying all organization and user account owners that had GitHub Apps installed and had a scoped installation token generated during the bug window so that they can stay informed and perform their own audit of their installed GitHub Apps.
As a followup to this investigation, GitHub is looking at ways to improve our logging to enable more in-depth analysis on scoped token generation and GitHub App permissions in the future.
* What was the potential impact? *
Due to the variety of GitHub Apps, their possible scopes, and the repositories they may have been given access to, we are unable to advise on any potential impacts as each customer's situation will be unique.
* What you can do *
While we have updated our systems to resolve this bug and no action is required on your end, we do recommend you review your installed GitHub Apps. You can use the following guidance for assessing GitHub Apps, their permissions, and their access to your private organization repositories:
https://docs.github.com/en/organizations/keeping-your-organization-secure
82 comments
[ 5.3 ms ] story [ 163 ms ] threadWhy not after remediation inform users of the flaw and potential impact. Then follow up with detailed impact.
Instead we get this 3 months later all they can say is "Some of your apps refreshed their tokens during a 5 day period" which is not news...
This is also the second time this year there has been significant delay in communication. Granted those involved other third parties so who knows where the delay lived.
Not ideal by any means. I’d be curious to know if my theory is correct or not.
> GitHub learned via a customer support ticket that GitHub Apps were able to generate scoped installation tokens with elevated permissions. Each of these tokens are valid for up to 1 hour.
> GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC.
> GitHub immediately began working to fix the bug and started an investigation into the potential impact. However due to the scale and complexity of GitHub Apps and their short-lived tokens, we were unable to determine whether this bug was ever exploited.
It's a very edge-case issue in Enterprise SSO, so I wasn't really able to generate any blowback with disclosure either. But if you find an org with just the right setup it blows a huge hole into the SSO product, to the point of making it useless.
There also seems to be an asymmetry between the core product and everything else. GitHub Enterprise has issues that aren't even considered UX issues (i.e. notifications showing "3 of 0" notifications if no SAML session exists) that'd warrant bounties if they were in the core product.
[1]: https://notes.acuteaura.net/posts/github-enterprise-security...
That's also likely why issues in the core product are taken more seriously.
This was an accidental find and GitHub has refused to document it.
> Each of these tokens are valid for up to 1 hour.
So these tokens would've been expired for 3 months now, according to when the fix was deployed.
> GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC.
It doesn't sound like there is anything you can or need to do with respect to these tokens (whether they were used to take action with elevated permissions is a different thing, but it doesn't sound like that was the case either)
> ...
> Due to the variety of GitHub Apps, their possible scopes, and the repositories they may have been given access to, we are unable to advise on any potential impacts as each customer's situation will be unique.
Absence of evidence is not evidence of absence.
Repository owners may well have a different level of acceptable risk or legal obligations over the integrity of their source code. For example, if I was maintaining security software or a popular package, it would be entirely appropriate to stop everything and look for abuse. Waiting three months makes that harder.
I'm not sure that's trust building.
Is ‘fulfilling legal requirements’ all you look for in a business relationship?
A restaurant has no legal requirement to make this food tasty but it’s what I’m looking for when choosing where to go.
Unfortunately, people got used to this practice and gladly accept when such companies fulfill all their legal obligations, even when this hurt them or their business.
I think many engineers often overlook the business implication of disclosing security issues, as it would impact multiple business units as well as the board's stance on security, resource allocation, and potentially the stock price too.
>A restaurant has no legal requirement to make this food tasty Food is a core deliverable for a restaurant, whereas information on a potential breach is not for a SaaS service unless it is legally required.
GH has no evidence this was not exploited. They just didn't log enough things to know if it was exploited or not.
I couldn't care less. I want value as a customer. Any company that prioritizes stockholders to customers doesn't deserve my customer money.
Somewhat tangential but I'm not even sure that's entirely true. It gets all sorts of tricky due to the subjectivity, but surely fit-for-purpose laws apply here? I'd be really surprised if a five star restaurant selling $500 tasteless gruel with chunks wouldn't manage to get into trouble if they refused refunds.
[1] - https://www.nbcnews.com/news/world/ants-cod-liver-moss-denma...
Do they keep logs, or is that also not a legal requirement? (See how these things can combine?)
Folks literally should be mindful no oneis obligated to do business with you if they think you're untrustworthy.
- a github app which had read permission on issues could elevate its permission to write
- a github app which had read permissions to discussions could elevate its permissions to write.
So far if the org/user would have been compromise they would have seen with issues or conversations containing content from the app.
Since these are only examples, I can imagine the case with major impact would be a contents:read elevate to content write. But again with commit signing, this would also be caught by the user. What did I miss where the impact would have not been visible to the end user/org ?
What about release artifacts?
[1]: https://github.com/nodejs/node/commits/main
(also ability to do it with content:write is just speculation from my side, they don't make it clear if it is possible, that would need to be confirmed by github)
Vote at the ballot box and with your dollars. Do not reward executives, lawyers, or engineers who dissemble or obfuscate.
yet the same people use google, and facebook, and AWS, like those companies sit upon moral high ground. they do not.
Linux succeeded and defeated Microsoft in every single way that matters to open source people, and the response is to continue to hate Microsoft for their loss? I do not understand.
Just admit your motivation for saying things like this: you hate Microsoft because Slashdot told you to, or tells you to, and you want to let people know about that, unprompted. Nothing about Microsoft's behavior in the 1990s has any bearing on what GitHub does today.
I can’t speak for everyone, but I continue to not like Microsoft because they don’t make a single piece of software that I like, and they’re ruined the ones I did like.
they haven't done EVERYTHING that EVERYONE wants, so lots and lots and lots of people still shit on them like they're still mad, and it still gives nerd cred to shit on them for any reason. as if we're perfect in comparison...
if the same people speak poorly of Facebook, Google, or Amazon in the same ways, the reactions observed are very different. Very different.
People pick on Microsoft because they once earned it, yet we let so many worse things slide, today, because those things are done by companies which are not Microsoft.
flippin' pick an opinion, and stick to it.
I abhor Google, loathe Facebook, have considerable contempt for Musk (as a person) and the products his companies offer, use Amazon sparingly and grudgingly, and only started shopping at Walmart during the Great Recession when a) they became the employer of the majority of my neighbors and b) other vendors in my area closed.
@naikrovek may not now be aware of what happens when one makes assumptions about other people and their actions, beliefs, etc.
So when someone gets a job where they're required to use Windows and spend all day in IE11 and Microsoft Office, let's say at a very large semiconductor company, they're hallucinating?
The goalpost was set at "Linux succeeded and defeated Microsoft in every single way that matters to open source people". Ignore the record of what was actually claimed if you want (the quotes you're throwing around are works of your imagination), but that is not merely a claim that either Windows or Linux is "the most common".
The observation that that there exists some X where P does not hold is precisely the way to counter a claim that for all X, P is true.
To attempt to answer you (I might regret this): what did you take "every single way that matters to open source people" to mean? By gazing through your insults into your comment, trying to find the "∀x statement" you think you saw, I have to assume that you think that means "every one developing open source code isn't using Microsoft?" Is your made-up acquaintance working at a semiconductor company primarily developing open source software there (rather than, say, semiconductors)?
Is the "∀x statement" about "x: way that matter"? Am I to take "non-free software is being used for making semiconductor at a large company" as an example of something that should matter to open source people?
You seem so convinced that there is a logic statement written right there with quantifiers and everything that you don't hesitate to use ridicule on complete strangers and question their grasp of logic. It's funny but couldn't really count as an argument (assuming you were trying to argue rather than just get some bile out).
This is not an uncommon reaction from people who are accustomed to bullshitting their way into arguments and who typically "win" those arguments by being just enough of a nuisance—and where the stakes are just low enough—that the likeliest outcome tends to be that the other party decides to move on rather than exhaustively refuting the bullshit. When you find people doing this with you, you are not "winning". In fact, that it happens is a consequence of how annoying others find it to interact with you.
An example (of someone who was similarly incredulous that everyone didn't just let the lame contrarian quips go unremarked upon—and of the sort of company you're in): <https://news.ycombinator.com/item?id=27906289>
> what did you take "every single way that matters to open source people" to mean?[...] I have to assume that you think that means "every one developing open source code isn't using Microsoft?"
No, you don't have to assume that. The only possible reason to assume something so self-serving is because you refuse to engage in a good faith resolution.
On the issue that is under discussion, there's no ambiguity here—at all.
The original statement—which you distorted twice—is a for all X statement: "Linux succeeded and defeated Microsoft in every single way that matters to open source people":
For all X, where X is some thing that "matters to open source people", the condition C is true, where C is the claim that "Linux succeeded and defeated Microsoft [on those Xs]".
The fact that you are unable to parse this out of a very straightforward passage like the one that appears in the part of the comment I quoted, but that you _are_ able to just, like, make some shit up about where the goalposts were set suggests very much that you are not making any attempt whatsoever to actually understand the issue, and you are just in the business here of issuing low-effort quips (like your first two here[1][2]) that don't actually track the discussion. But setting that aside, let's move on to the actual issue of the claim.
The following scenario results in a contradiction of the statement that I responded to:
It is 1997. Ned is working for a very large semiconductor company. At Ned's company, they are required to use Microsoft products. Ned instead wants to use Linux. He is an open source people, and this is what matters to him.
Now, it is 2022. Ned is still working for that very large semiconductor company where they are still required to use Microsoft products—he's not allowed to use Linux. This still matters to him.
It is therefore undeniably refuted via proof by contradiction; the statement that "Linux succeeded and defeated Microsoft in every single way that matters to open source people" is false.
Having now dealt with that, let's read back the transcript and look at your multiple attempts to move the goalposts, e.g. from "every single way that matters to open source people", to something that you plucked out of the air entirely—that is, whether one of Windows/Linux is merely "the most common". That is absolutely _not_ where the goalposts were set, and that's just the first instance of distortion. The second is where you go on to manufacture a quote: '"there isn't anyone using Windows at all"'—which appears nowhere except in your comment where you present it as a quote. Not only is this bullshit, it is the kind of bullshit that is against the rules here on HN.
Do not do this kind of thing. And certainly don't do this sort of thing while making a big deal about how annoye...
And no one is claiming that Google, Facebook, and Amazon are better. Those are not the examples I would pick to show Microsoft could do better, for sure. This is whataboutism.
That's a weird assumption to make, and definitely incorrect in at least some instances.
> Just admit your motivation for saying things like this: you hate Microsoft because Slashdot told you to ...
Huh? This seems like a really bizarre path to be going down. Not sure how you got there. :/
I don't know, hardware still regularly does not support Linux, the Linux desktop has a risible fraction of the world's user base, popular apps still only exist for winmac.
It won on the server, sure, but that's hardly every single way that matters, at least from what I remember as an open source user in the '00s.
ChromeOS also uses the Linux kernel.
If you are talking about Chrome-the-web-browser, well, then the parent comment is nonsense.
Speak for yourself.
For me, the lesson I learned while growing up with the MS of the 90's was to not trust any big corporation. The power imbalance is too large, individuals have no way to protect themselves and they will take any and every opportunity to exploit that.
This pattern can be seen with in the 90's with Windows, it can be seen today with Github and LinkedIn (talk with recruiters and they will tell you how MS is jacking up the prices and removing functionality) and it can be seen with any of Big Tech in the last 20 years.
“Those who cannot remember the past are condemned to repeat it.” – George Santayana
You have no idea of what tools I or anyone else on here use or don't use. You only know that I use git, which I could be using independently of everything. I could be using git to track my personal thoughts every day and nothing more. I make an effort to source everything I use as ethically as possible, though it is ridiculously difficult in the world of modern technology.
No one died because Microsoft included IE with Windows. no one was enslaved because Microsoft included IE with their OS.
that you think these are all events of the same magnitude has completely invalidated your opinion, and placed you firmly in crackpot territory.