My reading is if you allowed Salesforce/Heroku or Travis CI access to your GitHub account/organization, then there is a chance that that grant was leaked to a malicious third party - and they may have accessed your github account and downloaded private material.
GitHub indicates they are performing an audit and if they find such evidence they will notify each account/org within the next 72 hours.
OAuth properties are like jet plane cockpit buttons. For computer science students there should be one semester OAuth course. And in big companies there should be OAuth Expert. That is how much knowledge OAuth requires. If you don't agree with me try using Keycloak. It has EVERYTHING so you don't need to implement.
> Based on subsequent analysis, we believe this API key was obtained by the attacker when they downloaded a set of private npm repositories using a stolen OAuth token from one of the two affected third-party OAuth applications described above.
Does this imply there was unencrypted AWS credentials stored in an NPM repository and/or possibly GitHub? Seems like a bad idea, though I’m sure it’s hard to get off that practice if you’ve been on it for a long time.
> Does this imply there was unencrypted AWS credentials stored in an NPM repository and/or possibly GitHub?
Large organisation, hundreds of developers. Kind of likely that something is visible somewhere. Still a bit surprising it’s the prod access keys for npm…
Neither Heroku nor GitHub are addressing the key part of this - if Heroku's Dashboard apps were compromised, and thus access was given to connected GitHub repos for download - were the secrets in the Heroku config vars for the app also visible? That is the nightmare scenario for the affected app's owners.
"Salesforce continues to investigate this incident in coordination with GitHub and our retained third-party breach vendor. Once we identify how the threat actor gained access to customers' OAuth tokens, we will immediately take appropriate actions."
Sounds like they simply don't know yet how the actor got access and what else was exposed.
The token isn’t nearly as concerning as what it implies: this stolen token would have been stored in heroku’s DB. However it got out likely has broader implications.
Just going to say that this is the kind of scenario I was discussing in https://mjg59.dreamwidth.org/59353.html . RFC 8705 actually provides a mechanism for dealing with this, so I guess I'm going to be trying to implement that for our infrastructure in the near future.
I still don't totally understand, can anyone read between the lines to explain what's going on here? The attackers stole what sort of OAuth tokens with what sort of access?
I could be wrong but you know when you sign up for services you have option of signing in with github or other oAuth providers? I think they used this to gain access to various private repos, npms, cloud credentials.
So it seems like if your company had some super secret project running on github, heroku, AWS, they had full access to it.
This is my rough understanding just skimming so could be wrong.
I guess here's what I'm wondering: did they somehow get multiple Oauth keys, one per account compromised on github -- from both Heroku and Travis, just coincidentally both heroku and travis exposed their customers keys somehow?
Or did they get one Oauth key each from Heroku and Travis that exposed all of their customers data on github? Are there keys that would do this, set up for what purpose?
If one thing can be gotten from that incident is forcing GitHub to create finer OAuth scopes and per repository.
I am always uneasy when an application even thrust worthy like sentry asks access to ALL my private repositories even when they dont need it.
I feel that in 2022 GitHub is way too critical to many organizations and they should really work on stricter rules. Even google requires special and costly audits from oauth applications to access users critical data like drive and emails.
Only "app installations" from the GitHub marketplace can be limited to specific repositories. For OAuth apps this is not possible sadly (everything that uses GitHub to login or connect accounts, e.g. Travis CI, Terraform Registry, Heroku).
Github has (had) some of the worst wording in the permission prompts as well as too many 3rd party apps asking for blanket permission to do anything and everything in all your repos. It might as well be "hey, give us root on your servers and our new service will do cool things for you!"
And even worse is the 1000s and 1000s of repos that have signed up for those services including all kinds of popular libraries. They've given the keys to hack the libraries at will to 10s or 100s of random companies that integrate with github. It's insane.
> If one thing can be gotten from that incident is forcing GitHub to create finer OAuth scopes and per repository.
This. Limiting OAuth app access to specific repositories. I really wish this was a thing. For this exact reason I'm creating a new GitHub account for every new integration and share the repo with this account. It's technically against the TOS (they only allow single bot account per user) but there is no other way if you want security.
Maybe GitHub will implement this feature now. My hopes are high.
Aren't OAuth tokens scoped to org? Wouldn't it be easier to create an org per repo? I guess it might get expensive depending on your usage/collaborators.
Edit: nevermind OAuth tokens are not org-scoped, only apps are
How _lucky_ that github itself was the subject of the attack in npm.
Unless I'm missing something, this attack could have gone unnoticed for a long time (it would be hard for someone to connect a random breach in their infrastructure to an oauth intrusion affecting two of their service providers).
So this means they also got access to all the private npm package code? Compromising all the organisations using travis or heroku is one thing. But all npm packages quite another.
Nothing can pull the value of secrets from GitHub actions without either running a compromised/modified runner, or the workflow itself encrypting/printing/shipping that actions secret off to somewhere. If you’re really worried, use oidc GitHub actions where possible: https://docs.github.com/en/actions/deployment/security-harde...
Not just any Friday, but Good Friday, which happens to be a legal holiday in many countries, including here in Germany. And since Easter Monday is one too, a lot of people will go on vacation on this "long" weekend (maybe even taking a couple of days off before or after to make it even longer)... Flights and hotel rooms booked, and all that.
It's pretty plain that source code exfiltration occurred, though. It's not clear to me how exactly to confirm that the exfiltration happened to your account or not.
From the other thread:
> GitHub indicates they are performing an audit and if they find such evidence they will notify each account/org within the next 72 hours.
Probably for a lot of us, a source code leak is not the end of the world, but leaked secrets (i.e. config vars) should be addressed immediately. Rotating credentials will take time. E.g. 20 heroku apps with 30 secrets per app could realistically take several hours (even days) to fully rotate creds.
A few days before this attack campaign started, I wrote a guide to help security and engineering teams prioritize and remediate thousands of secrets-in-code incidents.
Hope it can help some of the organizations dealing with this right now!
I was confused because I got an email from Salesforce to my personal email. (We use Salesforce at work.) Email looked phishy (e.g. unsecured (non-https) links). Apparently they acquired Heroku in 2010.
For all the complexity of OAuth, an OAuth token is essentially a free pass for $stuff.
Perhaps we could start validating that a token not only is valid, but also is digitally, freshly signed with a known certificate?
I'd say there's a huge difference between:
* I stole a token
* I stole the token AND am able to keep signing said token with my victim's public certificate
A token can be stolen from many places and is comparatively easy to obtain, while full intrusion into an org's infrastructure is less likely, and even if it happens, eventually will be remediated (and the related certificate will be revoked).
I like this idea makes tons of sense… not only is the token valid but it’s from a trusted source… it is not going to prevent future attacks but it will add one more lock for an attacker to break before gaining access that is a good thing…
I think HTTP request signing will be a non-starter for an interoperable standard like oauth because of the difficulty of canonicalization without an authoritative library. AWS can pull that off because they have an SDK that will canonicalize requests for you, and it's tested in lockstep with the server accepting it. Historically, XML (and JSON to a lesser extent) have been security minefields for this exact reason.
AFAIK according the draft HTTP request signing rfc [1] defines explicit instructions for normalising, serialising and signing HTTP headers (not body). However, you can use an HTTP Digest header for example, that is a hash of the plaintext body to authenticate the body
We're a small org with a github connected to heroku. All of our repos were cloned between April 8 and April 15 with the majority of them having no activity for several years. The audit logs don't show this, you can only see this information in the traffic graphs (/graphs/traffic). If you're seeing cloning of repos that you haven't touched in a while, you've likely been compromised.
69 comments
[ 3.2 ms ] story [ 105 ms ] thread(Submitted title was "Some Salesforce private GitHub repos stolen")
GitHub indicates they are performing an audit and if they find such evidence they will notify each account/org within the next 72 hours.
> Heroku Dashboard (ID: 145909)
> Heroku Dashboard (ID: 628778)
> Heroku Dashboard – Preview (ID: 313468)
> Heroku Dashboard – Classic (ID: 363831)
> Travis CI (ID: 9216)
Developers at Heroku and Travis are going to have a rough Friday
Does this imply there was unencrypted AWS credentials stored in an NPM repository and/or possibly GitHub? Seems like a bad idea, though I’m sure it’s hard to get off that practice if you’ve been on it for a long time.
Large organisation, hundreds of developers. Kind of likely that something is visible somewhere. Still a bit surprising it’s the prod access keys for npm…
"Salesforce continues to investigate this incident in coordination with GitHub and our retained third-party breach vendor. Once we identify how the threat actor gained access to customers' OAuth tokens, we will immediately take appropriate actions."
Sounds like they simply don't know yet how the actor got access and what else was exposed.
The compromised tokens could provide the threat actor access to customer GitHub repos, but not customer Heroku accounts
While the situation can get worse, this token alone may not be enough...
mTLS is definitely an excellent step towards solving this.
So it seems like if your company had some super secret project running on github, heroku, AWS, they had full access to it.
This is my rough understanding just skimming so could be wrong.
Or did they get one Oauth key each from Heroku and Travis that exposed all of their customers data on github? Are there keys that would do this, set up for what purpose?
I am always uneasy when an application even thrust worthy like sentry asks access to ALL my private repositories even when they dont need it.
I feel that in 2022 GitHub is way too critical to many organizations and they should really work on stricter rules. Even google requires special and costly audits from oauth applications to access users critical data like drive and emails.
https://games.greggman.com/game/github-permission-problem/
Github has (had) some of the worst wording in the permission prompts as well as too many 3rd party apps asking for blanket permission to do anything and everything in all your repos. It might as well be "hey, give us root on your servers and our new service will do cool things for you!"
And even worse is the 1000s and 1000s of repos that have signed up for those services including all kinds of popular libraries. They've given the keys to hack the libraries at will to 10s or 100s of random companies that integrate with github. It's insane.
This. Limiting OAuth app access to specific repositories. I really wish this was a thing. For this exact reason I'm creating a new GitHub account for every new integration and share the repo with this account. It's technically against the TOS (they only allow single bot account per user) but there is no other way if you want security.
Maybe GitHub will implement this feature now. My hopes are high.
Edit: nevermind OAuth tokens are not org-scoped, only apps are
Just noting that gitlab has supported per repo tokens for ages. Yes, it's really useful, as this incident can attest.
You’ll also discover this if you try to create a personal access token, it’s all repos, you can’t limit them to a particular repo.
Their sec org is impressive, so these occasional massive things have been surprising
Unless I'm missing something, this attack could have gone unnoticed for a long time (it would be hard for someone to connect a random breach in their infrastructure to an oauth intrusion affecting two of their service providers).
Heroku boasts:
> we will notify affected customers by email without undue delay.
4 days. Really. "without undue delay".
[1] https://github.blog/2022-04-15-security-alert-stolen-oauth-u...
13-14, so "only" 1-3 days later, depending on what GitHub mean and timezones involved.
Some questions that come to mind:
* Would encrypted secrets for GitHub Actions be exposed to these tokens?
* How to best audit for unauthorized access to exposed repos?
* Were Heroku encrypted secrets exposed?
Not just any Friday, but Good Friday, which happens to be a legal holiday in many countries, including here in Germany. And since Easter Monday is one too, a lot of people will go on vacation on this "long" weekend (maybe even taking a couple of days off before or after to make it even longer)... Flights and hotel rooms booked, and all that.
I can't see anything mentioned on the incident page nor config vars page
https://status.heroku.com/incidents/2413
https://devcenter.heroku.com/articles/config-vars
It's pretty plain that source code exfiltration occurred, though. It's not clear to me how exactly to confirm that the exfiltration happened to your account or not.
From the other thread:
> GitHub indicates they are performing an audit and if they find such evidence they will notify each account/org within the next 72 hours.
Hope it can help some of the organizations dealing with this right now!
https://blog.gitguardian.com/a-practical-guide-to-prioritize...
Perhaps we could start validating that a token not only is valid, but also is digitally, freshly signed with a known certificate?
I'd say there's a huge difference between:
* I stole a token
* I stole the token AND am able to keep signing said token with my victim's public certificate
A token can be stolen from many places and is comparatively easy to obtain, while full intrusion into an org's infrastructure is less likely, and even if it happens, eventually will be remediated (and the related certificate will be revoked).
Edit: looks like https://tools.ietf.org/id/draft-ietf-oauth-mtls-09.html would fit the bill.
AWS and Mastodon use HTTP request signing which seems interesting.
[1] https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-mes...
https://developer.mastercard.com/blog/why-mastercard-doesnt-...
For example, my site, serverthiefbait.com might help...