> While running the exploit, CodeRabbit would still review our pull request and post a comment on the GitHub PR saying that it detected a critical security risk, yet the application would happily execute our code because it wouldn’t understand that this was actually running on their production system.
What a bizarre world we're living in, where computers can talk about how they're being hacked while it's happening.
Also, this is pretty worrisome:
> Being quick to respond and remediate, as the CodeRabbit team was, is a critical part of addressing vulnerabilities in modern, fast-moving environments. Other vendors we contacted never responded at all, and their products are still vulnerable. [emphasis mine]
Props to the CodeRabbit team, and, uh, watch yourself out there otherwise!
I did not understand something: why did CodeRabbit run external tools on external code within its own set of environment variables? Why are these variables needed for this entire tooling?
> After responsibly disclosing this critical vulnerability to the CodeRabbit team, we learned from them that they had an isolation mechanism in place, but Rubocop somehow was not running inside it.
Curious what this (isolation mechanism) means if anyone knows.
Unrelated to the article, but the first time I saw them was in a twitter ad with a completely comically bull** suggestion. I cannot take a company seriously that had something like that inside an ad that is supposed to show the best they're capable of.
I've noticed CodeRabbit at times does reviews that are super. It is able to catch bugs that even claude code misses on our Github PRs. Blows my mind at times tbh.
Based on the env vars seems like they're using anthropic, openai, etc. only?
hey, this is Howon from CodeRabbit here. we wish to note that this RCE was reported and fixed in January. it was entirely prospective and no customer data was affected. we have extensive sandboxing for basically any execution of anything now, including any and every tool and all generated code of any kind under the CodeRabbit umbrella.
It is incredibly bad practice that their "become the github app as you desire" keys to the kingdom private key was just sitting in the environment variables. Anybody can get hacked, but that's just basic secrets management, that doesn't have to be there. Github LITERALLY SAYS on their doc that storing it in an environment variable is a bad idea. Just day 1 stuff. https://docs.github.com/en/apps/creating-github-apps/authent...
global scoped installations or keys always scare me for this reason
i believe the answer here was to exchange the token for something scoped to the specific repo coderabbit is running in, but alas, that doesn't remove the "RCE" _on_ the repo
I think that Security fuckups of this disastrous scale should get classified as "breaches" or "incidents" and be required to be publicly disclosed by the news media, in order to protect consumers.
Here is a tool with 7,000+ customers and access to 1 million code repositories which was breached with an exploit a clever 11 year old could created. (edit: 1 million repos, not customers)
When the exploit is so simple, I find it likely that bots or Black Hats or APTs had already found a way in and established persistence before the White Hat researchers reported the issue. If this is the case, patching the issue might prevent NEW bad actors from penetrating CodeRabbit's environment, but it might not evict any bad actors which might now be lurking in their environment.
Oh, it really makes my day when we get hacker post here on the top. This is so well written too, no mystique, just a simple sequence of logical steps, with pictures.
Oh my god. I haven't finished reading that yet, it became too much to comprehend. Too stressful to take in the scope. The part where he could have put malware into release files of 10s of thousands (or millions?) of open source tools/libraries/software. That could have been a worldwide catastrophe. And who knows what other similar vulnerabilities might still exist elsewhere.
This is very similar to a CVE I discovered in cdxgen (CVE-2024-50611), which is similar to another CVE in Snyk's plugin (CVE-2022-24441). tl;dr if you run a scanner on untrusted code, ensure it doesn't have a way of executing that code.
Some ways to prevent this from happening:
1. Don't let spawned processes have access to your env, there are ways to allowlist a set of env vars that are needed for a sub process in all major languages
2. Don't store secrets in env vars, use a good secrets vault (with a cache)
3. Tenant isolation as much as you can
4. And most obviously - don't run processes that can execute the code they are scanning, especially if that code is not your code (harder to tell, but always be paranoid)
That’s why I’m worried about the growing centralization of things such as Chrome, Gmail, AWS, Cloudflare…
It’s very efficient to delegate something to one major actor but we are introducing single points of failure and are less resilient to vulnerabilities.
Critical systems should have defenses in depth, decentralized architectures and avoid trusting new providers with too many moving parts.
Can someone explain how is this not GitHub's fault that they don't allow the end-user to modify the permissions that all these services require? E.g., fine-grained permission control?
For example, why a tool like this code analysis service would need git write permission access in the first place?
The only consolation here is that it'd be difficult to forge git repositories because of the SHA hash conflicts for any existing checkout, although presumably even there, the success rates would still be high enough, especially if they attack front-end repositories where the maintainers may not understand what has happened, and simply move on with the replaced repo without checking what went on.
I cancelled my coderabbit paid subscription, because it always worries me when a post has to go viral on HN for a company to even acknowledge an issue occurred. Their blogs are clean of any mention of this vulnerability and they don't have any new posts today either.
I understand mistakes happen, but lack of transparency when these happen makes them look bad.
One of the problems is that code analyzers, bundlers, compilers (like Rust compiler) allow running arbitrary code without any warning.
Imagine following case: an attacker pretending to represent a company sends you a repository as a test task before the interview. You run something like "npm install" or run Rust compiler, and your computer is controlled by an attacker now.
Or imagine how one coworker's machine gets hacked, the malicious code is written into a repository and whole G, F or A is now owned by foreign hackers. All thanks to npm and Rust compiler.
Maybe those tools should explicitly confirm executing every external command (with caching allowed commands list in order to not ask again). And maybe Linux should provide an easy to use and safe sandbox for developers. Currently I have to make sandboxes from scratch myself.
Also in maybe cases you don't need the ability to run external code, for example, to install a JS package all you need to do is to download files.
Also this is an indication why it is a bad idea to use environment variables for secrets and configuration. Whoever wrote "12 points app" doesn't know that there are command-line switches and configuration files for this.
51 comments
[ 6.0 ms ] story [ 64.9 ms ] threadWhat a bizarre world we're living in, where computers can talk about how they're being hacked while it's happening.
Also, this is pretty worrisome:
> Being quick to respond and remediate, as the CodeRabbit team was, is a critical part of addressing vulnerabilities in modern, fast-moving environments. Other vendors we contacted never responded at all, and their products are still vulnerable. [emphasis mine]
Props to the CodeRabbit team, and, uh, watch yourself out there otherwise!
Curious what this (isolation mechanism) means if anyone knows.
Based on the env vars seems like they're using anthropic, openai, etc. only?
if you want to learn how CodeRabbit does the isolation, here's a blog post about how: https://cloud.google.com/blog/products/ai-machine-learning/h...
i believe the answer here was to exchange the token for something scoped to the specific repo coderabbit is running in, but alas, that doesn't remove the "RCE" _on_ the repo
Here is a tool with 7,000+ customers and access to 1 million code repositories which was breached with an exploit a clever 11 year old could created. (edit: 1 million repos, not customers)
When the exploit is so simple, I find it likely that bots or Black Hats or APTs had already found a way in and established persistence before the White Hat researchers reported the issue. If this is the case, patching the issue might prevent NEW bad actors from penetrating CodeRabbit's environment, but it might not evict any bad actors which might now be lurking in their environment.
I know Security is hard, but come on guys
Some ways to prevent this from happening:
1. Don't let spawned processes have access to your env, there are ways to allowlist a set of env vars that are needed for a sub process in all major languages
2. Don't store secrets in env vars, use a good secrets vault (with a cache)
3. Tenant isolation as much as you can
4. And most obviously - don't run processes that can execute the code they are scanning, especially if that code is not your code (harder to tell, but always be paranoid)
It’s very efficient to delegate something to one major actor but we are introducing single points of failure and are less resilient to vulnerabilities.
Critical systems should have defenses in depth, decentralized architectures and avoid trusting new providers with too many moving parts.
Why would you even grant it such permissions? this is ridiculous.
For example, why a tool like this code analysis service would need git write permission access in the first place?
The only consolation here is that it'd be difficult to forge git repositories because of the SHA hash conflicts for any existing checkout, although presumably even there, the success rates would still be high enough, especially if they attack front-end repositories where the maintainers may not understand what has happened, and simply move on with the replaced repo without checking what went on.
I understand mistakes happen, but lack of transparency when these happen makes them look bad.
Imagine following case: an attacker pretending to represent a company sends you a repository as a test task before the interview. You run something like "npm install" or run Rust compiler, and your computer is controlled by an attacker now.
Or imagine how one coworker's machine gets hacked, the malicious code is written into a repository and whole G, F or A is now owned by foreign hackers. All thanks to npm and Rust compiler.
Maybe those tools should explicitly confirm executing every external command (with caching allowed commands list in order to not ask again). And maybe Linux should provide an easy to use and safe sandbox for developers. Currently I have to make sandboxes from scratch myself.
Also in maybe cases you don't need the ability to run external code, for example, to install a JS package all you need to do is to download files.
Also this is an indication why it is a bad idea to use environment variables for secrets and configuration. Whoever wrote "12 points app" doesn't know that there are command-line switches and configuration files for this.