It gives you a good glance how likely it is that something is a threat - if an overwhelming majority of AV products detect it, it's quite likely that it's bad, and if only a few products find it, it's either very new or really just a bunch of false positives. Like many things, there's no simple "yes" or "no" answer, thus interpreting Virustotal output still requires some literacy on that topic. The site can also help software vendors to find which AV products reject their software, so that they can file false-positive requests to those AV vendors.
A separate advisory says the npm package "rc" is also compromised. That's also a highly popular one, according to the npmjs stats (1,323 dependents; 14.2 million weekly downloads).
What a worthless advisory, how about sharing who could possibly be affected at the very top, or at least anywhere?
Going to the issue, it seems the `preinstall` field was changed to `start /B node compile.js & node compile.js",` which means this would only run on Windows machines, everyone else seems to be unaffected.
Here is how you can find out if you have the affected package on your machine/instance:
That post doesn’t say much about `coa`, besides “new versions started appearing and builds started failing”. The bug report linked from GitHub advisory does a good job of describing the issue, though: https://github.com/veged/coa/issues/99
So if I’m reading this correctly, only Windows hosts are directly affected by the malware. On macOS and Linux one only needs to rollback to a healthy version of the package?
It's sort of a moot point since if you have an affected version you have to assume your whole system is compromised anyway in the case of this type of issue.
What exactly is the malicious code? I assume it's in `compile.js` and only can be found in published (now removed) npm package instead of source code repo?
The thing that should be causing concern is not so much these very loud obvious attacks, but how many better attacks that are harder to detect, are currently happening.
With 1.7M packages and an ecosystem that favours lots of 3rd party package usage, NPM is a large target. Whilst NPM isn't the only repository to have this kind of issue, it's definitely the largest attack surface.
What happened there was that he got the broken update, 2.0.3 which just referenced and used compile.js, but didn't include the file.
Then 2.0.4 came out which included compile.js and compile.bat.
Had he updated a couple of minutes later, this error would not have appeared. Not sure if /Users/ is a MacOS thing, but it is a Windows path structure, which might indicate that he was running this on Windows. And in that case he would have been compromised.
The most common ways these things seem to happen is either password reuse with no 2fa or that the npm token (in ~/.npmrc) was harvested by another compromised package/program. IIRC there were a few that were due to phishing too.
Almost certainly one of these. It's not a typosquatting attack, since it's an existing package. And it's not a repository compromise, since they had to create new versions instead of silently altering an existing version.
TLDR: The attacker injected an attack code as coa's `preinstall` script, which executes an obscurely-named file ("compile.bat"). This file is fully obfuscated, but what it does is basically to pull exploit DLLs from the attacker's server and install 'em.
I think the fortunate part of this accident is that the attacker failed to deploy the malware in his/her first attempt; v2.0.3 only contained the half of the changeset that the exploit needs to work (which accidentally broke tons of CI builds); So some developers could notice that something is wrong a bit early.
ahhh nice, it's been years since i've seen a obfuscated .bat!
very nice use of substring, but a bit too linear... with some input redirects and nested spaced variables it would have become more robust and unpredictable, but i suppose nowadays batch chiselers are rare.
edit: by the way in the article is missing a -useless- decoded line (n.4)
I don't see how this is an NPM issue at all. If the attacker made a Linux version and stuck it inside a Makefile, plenty of devs and build servers would end up executing the malicious code.
On npm, the average application uses hundreds (or thousands?) of dependencies, none of which are audited by the application author. On Linux, ffmpeg uses dozens or hundreds of dependencies, but they're each trusted and audited by the Linux distro packaging the dependency packages, who would hopefully catch malware shipped in a Makefile. And whatever vendored libraries a C/C++ app bundles (so you can build it on distributions that don't ship the library or a new-enough version) are usually self-contained rather than having transitive dependencies. As a result, a C++ app's author count grows linearly with the number of dependencies you bring on (and you generally evaluate each author's trustworthiness, and hopefully skim over the library and build scripts as well), rather than polynomially in the dependency tree's branching factor and exponentially in its depth (you generally don't evaluate the trustworthiness of your transitive dependencies' authors, since there are so many and they're not shown to you when you add a dependency).
Additionally, an app's vendored dependencies don't automatically update until you choose to update them. And when you update a vendored dependency, you have a chance of finding malware when you review the diff before committing the update. The downside is that vendored dependencies introduce friction in updating your dependencies (I've experimented with extracting tarballs, git subtree, and git subrepo, and generally avoid submodules, and they're all somewhat awkward), and don't automatically receive patch/security updates like distribution libraries do.
I'm more familiar with crates.io and PyPI than npm, but I think they all work similarly (unvetted package repository allowing libraries to depend on other libraries, and transitive dependencies get pulled in when you install the library, and can be updated without the library author's knowledge). The primary difference IMO is the degree that packages use transitive dependencies vs. building/copying the subset of code they need (the average Python app uses dozens of dependencies, Rust apps use hundreds, Node apps use thousands).
My wish is that for each language, a team of Linux distribution maintainers or community members would "sign off on" a set of trusted packages, each of which depends on only trusted packages. And when a trusted package gets updated, the new version isn't marked as trusted until the maintainer verifies the diff isn't malicious. Then the language package manager would add a mode where apps or libraries can choose to only depend on trusted packages (which only depend on trusted packages, etc.).
(In the Rust world, where the compiler and package manager are developed by the same non-commercial organization, I hope the Rust organization will create the team handling "trusted packages"; sadly they seem opposed to it because they consider it favoritism, but IMO it's a good thing for Arch to package wget and curl, but not "joe's 1337 haxxor tool". In the Node world, V8 and node.js and npm are different groups/companies I think, so I don't know which organization is community-led and neutral enough to make a trustworthy host for this effort.)
Crev takes the "web of trust" approach, where everyone can choose who to trust. However, it doesn't ship a "deny untrusted dependencies" mode as part of Cargo. And as with PGP webs of trust, I haven't met any library developers in person, don't know which developers sign off on packages, and would prefer having semi/centralized institutions (like Linux distributions) who stake their reputation on the non-maliciousness of packages they ship, and allow different people to agree on which packages to trust. Having institutions reduces the responsibility placed on the individual app developer. Right now, you need to trust an distro to take responsibility for an OS, and a languag...
Yes, but good luck getting kernel.org to distribute that. They're not perfect (nothing is, and kernel.org has been compromised in the past, though quite long ago, and even then it did not lead to compromise of the Linux kernel sources), but to put the Linux kernel maintainers on the same level as the people maintaining the NPM distributions is a bit much.
I would say that it's not that different from others I've seen, just more visible because of the size and activity of the repository.
One thing NPM does (and I believe Python too) is to allow install scripts -- this has been a reliable vector for attackers to steal credentials. Not every package repository system has that.
authors cannot revoke their compromised keys to immediately halt all distribution, and you don't have any process to verify package<->author ownership beyond the upload secrets.
I know npm feels like wild west, but you can audit. Its quite a challenge to review many of the C, C++, libraries out there that are just a .zip file stored in a website.
My point is that: Npm is auditable, trackable.
I'm not challenging the bug itself, neither the security issue..
The receiving server would trigger it. The first review should be done by the owner, who should be contactable via email and authenticate via 2FA and acknowledge the modification.
Then I'd guess that Microsoft has enough information with NPM's history to train an AI. Specially the modifications made in these versions could easily trigger suspicious activity.
Did you look at the diffs?
Also, three years of inactivity and then a sudden upload should easily trigger a manual review, even if it is by automatically opening an issue with a review request on that project's GitHub page.
I know npm feels like wild west, but you can audit. Its quite a challenge to review imany of the C, C++, libraries out there that are just a .zip file stored in a website.
My point is that: Npm is auditable, trackable. I'm not challenging the bug itself, neither the security issue..
It seems that all of these should be cryptographically signed by a developer's private key before publication and then verified by others before use. Is that not the case?
I agree that all software package repositories -- NPM, Rubygems, PyPI, Maven, NuGet, Crates, etc -- should have two minimum baseline security policies:
1. All accounts are MFA, no exceptions.
Only CI/CD usecases justify some laxity. Only push tokens should be non-MFA-able for CI/CD purposes, and they should only be usable for push. Tokens should only be obtainable with MFA.
Ideally these too would be part of a multi-step OAuth2 or OIDC refresh token to access token flow, so that any given access token is only used once and CI/CD jobs never get to see the refresh token.
And while we're on tokens: they should all expire, after a short period of time, without exception.
2. Signing packages is opt-out, not opt-in.
The two reasons package signing is relatively rare are that (1) in most ecosystems it is unwieldy, (2) not actually that effective at providing guarantees (self-signed certs are no better than a pinky promise).
But once these are solved, and I believe they can be, making signing opt-out means that we'd see signing rate percentages in the 90s, instead of languishing in single digits as they do today.
If you work on NPM and would like to swap notes, I would love to talk to you with my professional hat on -- email in profile.
This makes me appreciate Deno's focus on security. Having things like file and network access 'opt in' seems like a no brainer when we see how easy it is to simply install an npm package and find yourself vulnerable to malware.
Any protection offered by Deno isn't robust. Only OS-level protection is secure. Any tooling will require access to file system - and file system access is enough to compromise developer system.
Let's assume that you have Deno compiler for other language. You run it through seemingly innocent deno run "https://..." --allow-write=. src/ (you use optional parameter to --allow-write, right?).
Unfortunately, webpage hosting script was compromised. Now our compiler can write to .git/hooks, .npmrc (npm can do arbitrary script execution in version 6 or lower, even on npm --version), .idea/ etc.
Reminder that people should seriously consider disabling the install-scripts.
Personal system-wide config:
npm/yarn config set ignore-scripts true -g
and add & commit a .npmrc/.yarnrc file with
ignore-scripts true
Yes, this will cause headaches in some (increasingly rare) cases where some package actually needs those scripts. You can fix this with custom install scripts that take care of running install for those specific packages.
And yes yes, as people love to point out, this isn't exactly a bulletproof solution either. The attacker could just put the malicious code inside the package's code and wait for it to be actually executed. But again and again, they don't, they choose to use the package's install scripts as the place to do their dirty work.
So in practice this policy would've alrady protected you from who knows how many of these attacks, and my guess is that it'll continue to do so.
And to clarify, you should run this once by hand to collect the list, don't add the grepping to the actuall installation script or you are back to square one :D
Here's what I use with yarn, install.sh:
#!/usr/bin/env bash
yarn
function run_install() {
local dependency="node_modules/$1"
if [[ -d "$dependency" ]]; then
yarn --cwd "$dependency" run postinstall
fi
}
run_install 'esbuild'
run_install 'other_package'
...
NB: Setting “ignore-scripts” globally means you can’t even “npm run” scripts manually, for packages you’re authoring (or at least that appears to be happening for me).
Edit: Actually, on NPM 8.1.0, setting "ignore-scripts" globally still allows you to run scripts manually with "npm run".
The behavior I described (where it just blanket ignores ALL scripts, even ones you try to run) existed on NPM 6 (which is what I was using until a few minutes ago). I couldn't find the exact version where this behavior was fixed.
There are just too many micropackages to properly look at. A basic react app created using official method (CRA) was 200MB something last time I tried.
Are other languages/runtimes also that risky as Node with npm?
npm packages seem like a cardhouse.
I know that the node_modules folder is often times criticized for its sheer amount of 3rd party libraries. Is it because of JavaScripts "missing" standard library?
I will be downvoted to hell for saying this. But javascript ecosystem is where most newbies come. (Low barrier to entry and it also seems hip). With no regards to security, maintainability or reliability, fashion chasing blog-happy hipsters.
The amount of churn in JS ecosystem, security incidents like this and general crappiness of websites can be generally explained by how immature these hipsters are.
Not disagreeing with organization around frameworks in PHP, but apparently PHP suffered a lot from low barrier, in terms of security especially. That was the time package management wasn't that widespread yet, which IMO limited this kind of stuff. But there were, for sure, many applications where PHP could be blamed for security incidents and general unreliability. PHP improved quite well though.
Now PHP isn't hip anymore, node js is super popular hip thing, and every tom dick and harry from art school in US or 3rd tier engineering college in India will slap together three todo list applications on Resume and wants to call himself full stack developer. Internet is fast, hardware is fast, no one cares about pile of dependencies sitting beneath them.
Add to that resume driven development where every JS wants to write libraries and become github-famous. 30-line libraries will be considered a joke and it will be shameful to brag about such things in any other ecosystem.
In 2019, 11% of all vulnerabilities listed by the National Vulnerability Database were linked to PHP; historically, about 30% of all vulnerabilities listed since 1996 in this database are linked to PHP. [1]
Also early this year there were news that git.php.net was compromised and a backdoor was introduced into PHP, but lucky enough the backdoor was catch before a production release.[2][3]
Npm is pretty unique in the low bar it sets for security. What is really frightening is how these cases are discovered, more or less by accident, rather than by some kind of verification process that ensures this simply can not happen before QA catches it on the way to a release.
My read of the headline was that this was malware in embedded hardware electronics systems, or describing some exploit / attack surface for same.
May I suggest that a clearer phrasing would be ‘Malware embedded in Coa’? Or is ‘embedded malware’ a somewhat confusing term-of-art in the cybersec community?
If I'm reading this correctly, the malicious code was new (higher) versions of the releases.
Would this mean any project using a package.lock/yarn.lock was 'safe' going through deploys? So only new installs and builds without lock files could have grabbed the higher version?
If so, I wonder if it's hard or impossible to swap a release version on NPM. Seems like that would hit a much wider audience before being detected.
(Of course this malware was in a preinstall script, which should also be disabled... but any module you import in a node app can do bad things when you run your app, preinstall script or no.)
85 comments
[ 1.9 ms ] story [ 79.5 ms ] threadThere's signal if there's detections
There's signal if something is old and there's no detections
There's no signal if something is recent with no detections
https://github.com/advisories/GHSA-g2q5-5433-rhrf (" Embedded malware in rc" "critical severity")
Notable that both advisories link to the virustotal entry for the same file hash (same malware).
@dang Could the title be updated to include the names of other affected packages?
Going to the issue, it seems the `preinstall` field was changed to `start /B node compile.js & node compile.js",` which means this would only run on Windows machines, everyone else seems to be unaffected.
Here is how you can find out if you have the affected package on your machine/instance:
Assumes you have `find`, `xargs` and `jq` installed, will print all versions of coa it can find. Seems any version above 2.0.3 is bad.Edit: is anyone sitting on the source for `compile.js` as mentioned? Would be interesting to see.
https://www.bleepingcomputer.com/news/security/popular-coa-n...
locate "/coa/package.json" | xargs -I {} jq .version {} 2>/dev/null
Is it possible to check how many downloaded the compromised versions?
https://www.npmjs.com/package/coa
https://www.bleepingcomputer.com/news/security/popular-coa-n...
>"Based on our analysis and information seen thus far, the malware is likely the Danabot password-stealing Trojan for Windows."
The thing that should be causing concern is not so much these very loud obvious attacks, but how many better attacks that are harder to detect, are currently happening.
With 1.7M packages and an ecosystem that favours lots of 3rd party package usage, NPM is a large target. Whilst NPM isn't the only repository to have this kind of issue, it's definitely the largest attack surface.
[0] -> Error: Cannot find module '/Users/me/.npm/_npx/27078/lib/node_modules/@svgr/cli/node_modules/coa/compile.js'
What happened there was that he got the broken update, 2.0.3 which just referenced and used compile.js, but didn't include the file.
Then 2.0.4 came out which included compile.js and compile.bat.
Had he updated a couple of minutes later, this error would not have appeared. Not sure if /Users/ is a MacOS thing, but it is a Windows path structure, which might indicate that he was running this on Windows. And in that case he would have been compromised.
[0] https://github.com/veged/coa/issues/99
https://github.com/veged/coa/issues/99#issuecomment-96153687...
TLDR: The attacker injected an attack code as coa's `preinstall` script, which executes an obscurely-named file ("compile.bat"). This file is fully obfuscated, but what it does is basically to pull exploit DLLs from the attacker's server and install 'em.
I think the fortunate part of this accident is that the attacker failed to deploy the malware in his/her first attempt; v2.0.3 only contained the half of the changeset that the exploit needs to work (which accidentally broke tons of CI builds); So some developers could notice that something is wrong a bit early.
very nice use of substring, but a bit too linear... with some input redirects and nested spaced variables it would have become more robust and unpredictable, but i suppose nowadays batch chiselers are rare.
edit: by the way in the article is missing a -useless- decoded line (n.4)
What is missing is more automated and recurrent checks in all the packages and downstream dependencies.
However, that is dependent on first finding and flagging issues, which is exactly what this post is about.
Additionally, an app's vendored dependencies don't automatically update until you choose to update them. And when you update a vendored dependency, you have a chance of finding malware when you review the diff before committing the update. The downside is that vendored dependencies introduce friction in updating your dependencies (I've experimented with extracting tarballs, git subtree, and git subrepo, and generally avoid submodules, and they're all somewhat awkward), and don't automatically receive patch/security updates like distribution libraries do.
I'm more familiar with crates.io and PyPI than npm, but I think they all work similarly (unvetted package repository allowing libraries to depend on other libraries, and transitive dependencies get pulled in when you install the library, and can be updated without the library author's knowledge). The primary difference IMO is the degree that packages use transitive dependencies vs. building/copying the subset of code they need (the average Python app uses dozens of dependencies, Rust apps use hundreds, Node apps use thousands).
My wish is that for each language, a team of Linux distribution maintainers or community members would "sign off on" a set of trusted packages, each of which depends on only trusted packages. And when a trusted package gets updated, the new version isn't marked as trusted until the maintainer verifies the diff isn't malicious. Then the language package manager would add a mode where apps or libraries can choose to only depend on trusted packages (which only depend on trusted packages, etc.).
(In the Rust world, where the compiler and package manager are developed by the same non-commercial organization, I hope the Rust organization will create the team handling "trusted packages"; sadly they seem opposed to it because they consider it favoritism, but IMO it's a good thing for Arch to package wget and curl, but not "joe's 1337 haxxor tool". In the Node world, V8 and node.js and npm are different groups/companies I think, so I don't know which organization is community-led and neutral enough to make a trustworthy host for this effort.)
Crev takes the "web of trust" approach, where everyone can choose who to trust. However, it doesn't ship a "deny untrusted dependencies" mode as part of Cargo. And as with PGP webs of trust, I haven't met any library developers in person, don't know which developers sign off on packages, and would prefer having semi/centralized institutions (like Linux distributions) who stake their reputation on the non-maliciousness of packages they ship, and allow different people to agree on which packages to trust. Having institutions reduces the responsibility placed on the individual app developer. Right now, you need to trust an distro to take responsibility for an OS, and a languag...
One thing NPM does (and I believe Python too) is to allow install scripts -- this has been a reliable vector for attackers to steal credentials. Not every package repository system has that.
My point is that: Npm is auditable, trackable.
I'm not challenging the bug itself, neither the security issue..
https://my.diffend.io/npm/coa/2.0.3/2.0.4/
In 2021, why on earth does such a change not trigger a review before release?
Then I'd guess that Microsoft has enough information with NPM's history to train an AI. Specially the modifications made in these versions could easily trigger suspicious activity.
Did you look at the diffs?
Also, three years of inactivity and then a sudden upload should easily trigger a manual review, even if it is by automatically opening an issue with a review request on that project's GitHub page.
My point is that: Npm is auditable, trackable. I'm not challenging the bug itself, neither the security issue..
[0] https://www.change.org/p/npm-please-secure-package-releasing
1. All accounts are MFA, no exceptions.
Only CI/CD usecases justify some laxity. Only push tokens should be non-MFA-able for CI/CD purposes, and they should only be usable for push. Tokens should only be obtainable with MFA.
Ideally these too would be part of a multi-step OAuth2 or OIDC refresh token to access token flow, so that any given access token is only used once and CI/CD jobs never get to see the refresh token.
And while we're on tokens: they should all expire, after a short period of time, without exception.
2. Signing packages is opt-out, not opt-in.
The two reasons package signing is relatively rare are that (1) in most ecosystems it is unwieldy, (2) not actually that effective at providing guarantees (self-signed certs are no better than a pinky promise).
But once these are solved, and I believe they can be, making signing opt-out means that we'd see signing rate percentages in the 90s, instead of languishing in single digits as they do today.
If you work on NPM and would like to swap notes, I would love to talk to you with my professional hat on -- email in profile.
Let's assume that you have Deno compiler for other language. You run it through seemingly innocent deno run "https://..." --allow-write=. src/ (you use optional parameter to --allow-write, right?).
Unfortunately, webpage hosting script was compromised. Now our compiler can write to .git/hooks, .npmrc (npm can do arbitrary script execution in version 6 or lower, even on npm --version), .idea/ etc.
Personal system-wide config:
and add & commit a .npmrc/.yarnrc file with Yes, this will cause headaches in some (increasingly rare) cases where some package actually needs those scripts. You can fix this with custom install scripts that take care of running install for those specific packages.And yes yes, as people love to point out, this isn't exactly a bulletproof solution either. The attacker could just put the malicious code inside the package's code and wait for it to be actually executed. But again and again, they don't, they choose to use the package's install scripts as the place to do their dirty work.
So in practice this policy would've alrady protected you from who knows how many of these attacks, and my guess is that it'll continue to do so.
Here's what I use with yarn, install.sh:
I wish package.json had an option where I could explicitly mark which packages can run postinstall scripts.
Here's an RFC on exactly that: https://github.com/npm/rfcs/discussions/80
The behavior I described (where it just blanket ignores ALL scripts, even ones you try to run) existed on NPM 6 (which is what I was using until a few minutes ago). I couldn't find the exact version where this behavior was fixed.
Having NPM just be a free-for-all is a ticking time bomb. It is only a matter of time before an event like this results in something very serious.
NpmJS echo system is cancer.
npm packages seem like a cardhouse.
I know that the node_modules folder is often times criticized for its sheer amount of 3rd party libraries. Is it because of JavaScripts "missing" standard library?
The amount of churn in JS ecosystem, security incidents like this and general crappiness of websites can be generally explained by how immature these hipsters are.
Perhaps because dependencies are more curated in PHP due to clusters of dominant frameworks, rather than a proliferation of smaller libraries.
Now PHP isn't hip anymore, node js is super popular hip thing, and every tom dick and harry from art school in US or 3rd tier engineering college in India will slap together three todo list applications on Resume and wants to call himself full stack developer. Internet is fast, hardware is fast, no one cares about pile of dependencies sitting beneath them.
Add to that resume driven development where every JS wants to write libraries and become github-famous. 30-line libraries will be considered a joke and it will be shameful to brag about such things in any other ecosystem.
Until composer, there wasn't much of a culture around creating packages full stop, apart from PEAR, which beginners didn't create packages for.
Also early this year there were news that git.php.net was compromised and a backdoor was introduced into PHP, but lucky enough the backdoor was catch before a production release.[2][3]
1: https://en.wikipedia.org/wiki/PHP#Security
2: https://arstechnica.com/gadgets/2021/03/hackers-backdoor-php...
3: https://flast101.github.io/php-8.1.0-dev-backdoor-rce/
The git.php.net server was also not compromised, as far as I know, from a PHP vulnerability that was active at the time.
May I suggest that a clearer phrasing would be ‘Malware embedded in Coa’? Or is ‘embedded malware’ a somewhat confusing term-of-art in the cybersec community?
Would this mean any project using a package.lock/yarn.lock was 'safe' going through deploys? So only new installs and builds without lock files could have grabbed the higher version?
If so, I wonder if it's hard or impossible to swap a release version on NPM. Seems like that would hit a much wider audience before being detected.
(Of course this malware was in a preinstall script, which should also be disabled... but any module you import in a node app can do bad things when you run your app, preinstall script or no.)