Does anyone know how this was done?
Was it some sort of automatic AI system? Dedicated people watching update feeds 24/7? Random person noticing github commit and notifying npm?
Wouldn't be too hard to monitor drastic changes to packages, or monitor for packages like cryptomining being added to an otherwise bland set of dependencies.
The bottom line is NPM should start warning people that this is a COMMON occurrence, how many NPM users are novices or bootcamp grads who don't even know that it happens?
Obvious way to do this would be with some variation of ranking the edit distance or simple entropy differences, and alerting/prioritizing significant updates for further scrutiny in another workflow pipeline. Bug fixes are relatively low distace, where a new module would show up as a bump. It would FP on code cleanups, but who actually does those. There are more sophisticated methods than these used in products today, but this is something an amateur could implement. Compute is huge, but cost/benefit is a product problem.
Some string searches would be easy wins as well, given wallet file names, detecting new public keys, etc. Can think of a few AV products that would benefit users by being able to state an app has added an update that resembles a new module with functionality they might not have signed up for in the original EULA.
If you are an enterprise customer, I'd ask your AV vendor for it as a feature. Realistically, the best control is don't put a cryptocurrency wallet on a device that has automatic updates from untrusted sources.
"The attack was carried out by using a pattern that is becoming more and more popular; publishing a ‘useful’ package (electron-native-notify) to npm, waiting until it was in use by the target, and then updating it to include a malicious payload."
> This is exactly what many of us predicted when NPM introduced a world of thousands of separate dependencies per app. The main reactions to that skepticism were "stop being so negative" and "don't try to stop progress."
I'm sure "many of us" predicted planes would crash when they first took flight, but nonetheless the benefits of commercial air travel vastly outweigh the risks.
npm, having been a huge benefit to developers and the JS ecosystem for many years, should not have its problems dismissed with an "I told you so" attitude.
The problems with the JS ecosystem probably could have been solved in a way that doesn't involve every nontrivial project dynamically downloading thousands of transitive dependencies by god-knows-who. It's not like package repositories and dependency management were uncharted waters before npm came along.
And with github's recent "automatically create dep update PR" feature for security notifications, it would presumably be even easier to get people to update to a malicious version. Just claim the old version had a security vulnerability!
NPM foils plot that was only possible due to ridiculous house of cards built by NPM.
I notice that the NPM blog post also fails to mention that it was actually successful and resulted in about 1 million KMD (current value ~$1.7M) being stolen. The Komodo blog post contains that information: https://komodoplatform.com/update-agama-vulnerability/
Even worse, they had also successfully stolen about 9 times more than that. The only reason it was prevented was because the seeds were being sent to a public server, and Komodo was able to access and use them to "steal" the 8 million coins (and 96 BTC) from those wallets before the attacker did.
This was a successful $10M+ theft that NPM is somehow trying to spin into a positive.
I don’t think it’s fair to single npm out here. The truth is any package manager that lets anyone publish package updates is vulnerable to this type of attack.
Dependency ranges mean package updates happen automatically. So a minor release could contain malicious code without anyone noticing. I still think semver is worthwhile even if it a little bit more vulnerable to this kind of attack.
npm encourages an auto-upgrading versioning scheme: take version x or higher.
It's great for keeping dependencies up-to-date, but this makes it immensely vulnerable to the kind of malware injection, as publishing a library version with payload will immediately propagate throughout the ecosystem.
In most other version managers (e.g. maven) every library and application specifies a hard-coded version, which requires users to manually upgrade, giving a window were a malicious version can be detected before it affects anyone.
For a security critical app not locking down the dependencies to audited versions seems very dangerous indeed and I believe there is some blame to be had there
You’re still vulnerable if you lock down dependencies during the period the malicious code is wild though. You have to actually audit your locked dependencies to consider yourself safe.
Isn't that expecting the car to be invented before the wheel?
From what I've seen, quite a few of these have been targetted attacks - the assailants have targetted specific variables that existed in a specific codebase that they knew imports their module. Of course, if you've only just started writing that codebase, your dependencies have no way of knowing where you're storing private keys (short of somehow scanning the object space for variable names like "private_key")
But yes, in theory I would love for any *-sensitive code to go through a thorough audit after every dependency update before a version gets released in the wild. I'm guessing that next-to-nobody has the resources for that kind of an effort though.
Go's official package management solution actually only fetches the minimum specified version of each dependency. So auto-upgrades never happen except when you explicitly update a dependency to a newer version.
What they did was to allow komodo to, vigilante style, outright steal the coins, and hold them hostage until you entertain them with proof of ownership/identity and whatever other ridiculous requirements they come up with.
But it's npm and kleptocurrency. Nothing of value lost.
I’m honestly amazed this hasn’t happened more often; it’s ridiculous that a developer has to install thousands of lines of code to create a simple “hello world” website.
Maybe the post has been edited, but just now it reads '"hello-world" website'. When I search for "create a website with node.js" the top results all talk about using Express.js.
If I `npm init` and then `npm install express` as suggested, the result is 50 packages installed containing 21707 lines of javascript.
edit: trying to show what people are steered towards when looking up how to do this themselves.
"Before you even open a text editor, your project contains 1.5 million lines of code. Most of it contributed by volunteers and enthusiasts. No formal review or release process."
Take any other language with a web framework---Ruby with Rails, Java with Play/Spring, any web server which exposes a CGI interface, Python with Django/flask/whatever---and you will find a colossal amount of framework code behind the simple "hello, world!" demo.
Web apps are complex things, especially when the framework has to account for common cross-cutting concerns and patterns elegantly and succinctly. The fact that the JS ecosystem with NPM exposes that complexity to the user directly with an (albeit massive) node_modules installation isn't a strike against node per se: other languages have this code too, they just don't stash it all in the same folder as your project.
If you want to criticize the JS ecosystem, doing it on the basis of sheer SLOC downloaded by an `npm install` isn't too great.
We can talk about the quality of these packages, or about code review standards, or about the unique challenges auditing code from many sources for bugs and security problems. But talking about how the framework code gets installed seems counterproductive.
No need to repeat what others said, but I'll also note that create-react-app is a website boilerplate generator and includes preconfiguration for tons of stuff including test frameworks etc.
So yes, before you even open a text editor, you have eslint, jest, typescript etc all set up. Duh-doy, that's a lot of lines of code.
Hm... to claim the coins they saved, you have to receive and send a small transaction from the compromised wallet to verify ownership. But how do they make sure that it's not claimed by the hackers who may have the seed of these wallets? The only thing they say is that they don't send it if there is multiply claims (probably one from hacker and one from real owner).
I posted this a couple of days ago, but I guess at the wrong time of day - when I saw it got no interest, I thought "I'll bet this is one of those articles that someone else posts later, that the mods let through"!
This comment is a bit meta, but I actually really like this "second chance" feature of hacker news. Mods could in theory "boost" such stories by making them sticky, but I like this kind of hands-off, no-meddling approach :)
I received an email from Dan once recommending I repost a Show HN submission I had made that had no hits. Hacker News is pretty well moderated overall.
I get where you’re coming from, but doesn’t that mean you need at least one full time security auditor just to keep pace? Node projects seem to pretty quickly hit hundreds or thousands of dependencies, which means updates happen most days.
Here's a free idea for somebody at NPM or any other third party security provider: Tracking changes to network or fs access across versions
It should be possible to scan JS modules to determine either the fixed set of dependencies, the fixed set of system dependencies (ex: require('fs')), or whether it's not determinate (code contains "eval" or otherwise invokes require(...) with a non-constant). Including a package that has fs or net access would automatically include that taint as well.
The idea for this is that a malicious package would be more easily noticed as something like leftpad(...) suddenly requiring net access should be flagged. It's not a panacea as something malicious that already has fs or net access could do something new, but it would help add some sanity checks and give a smaller set of packages and versions to manually review.
I think deno (Node.js creator's new project using Typescript + Golang) has a similar idea but built in to the individual packages themselves where they need to explicitly include permissions for fs, net, etc. That'd be a great idea but getting to that point from the current (and growing) Node.js/NPM world is going to take a while.
This is a losing game (think halting problem). This exact scenario was played by antivirus software versus malware in the 90s/00s. The antivirus was trying to figure out if the binary did something like modify a file on disk, while the virus was trying to obfuscate that or find innovative ways of doing it.
A isolation/capability solution is the only one that could work, leftpad shouldn't have access to anything but basic CPU compute.
Doesn't add up if you look at the dates they claim the malware was injected and the dates of the most recent transactions of all the addresses swept. Some of those addresses hadn't been used in over a year.
These issues are entirely caused by NPM in the first place. Anyone who starts a node project ends up with a massive web of unaudited code which can only be dealt with my moving off NPM and adding packages to your own private repo after youve audited them yourself. Which is an almost impossible task.
60 comments
[ 3.4 ms ] story [ 105 ms ] threadThe bottom line is NPM should start warning people that this is a COMMON occurrence, how many NPM users are novices or bootcamp grads who don't even know that it happens?
Some string searches would be easy wins as well, given wallet file names, detecting new public keys, etc. Can think of a few AV products that would benefit users by being able to state an app has added an update that resembles a new module with functionality they might not have signed up for in the original EULA.
If you are an enterprise customer, I'd ask your AV vendor for it as a feature. Realistically, the best control is don't put a cryptocurrency wallet on a device that has automatic updates from untrusted sources.
Gary Bernhardt has an interesting comment: https://twitter.com/garybernhardt/status/1137459482135416832
> This is exactly what many of us predicted when NPM introduced a world of thousands of separate dependencies per app. The main reactions to that skepticism were "stop being so negative" and "don't try to stop progress."
I'm sure "many of us" predicted planes would crash when they first took flight, but nonetheless the benefits of commercial air travel vastly outweigh the risks.
npm, having been a huge benefit to developers and the JS ecosystem for many years, should not have its problems dismissed with an "I told you so" attitude.
I notice that the NPM blog post also fails to mention that it was actually successful and resulted in about 1 million KMD (current value ~$1.7M) being stolen. The Komodo blog post contains that information: https://komodoplatform.com/update-agama-vulnerability/
Even worse, they had also successfully stolen about 9 times more than that. The only reason it was prevented was because the seeds were being sent to a public server, and Komodo was able to access and use them to "steal" the 8 million coins (and 96 BTC) from those wallets before the attacker did.
This was a successful $10M+ theft that NPM is somehow trying to spin into a positive.
It's great for keeping dependencies up-to-date, but this makes it immensely vulnerable to the kind of malware injection, as publishing a library version with payload will immediately propagate throughout the ecosystem.
In most other version managers (e.g. maven) every library and application specifies a hard-coded version, which requires users to manually upgrade, giving a window were a malicious version can be detected before it affects anyone.
From what I've seen, quite a few of these have been targetted attacks - the assailants have targetted specific variables that existed in a specific codebase that they knew imports their module. Of course, if you've only just started writing that codebase, your dependencies have no way of knowing where you're storing private keys (short of somehow scanning the object space for variable names like "private_key")
But yes, in theory I would love for any *-sensitive code to go through a thorough audit after every dependency update before a version gets released in the wild. I'm guessing that next-to-nobody has the resources for that kind of an effort though.
If they hadn't done such a scummy, corporate spin job on this, I'd be inclined to agree with you - but they did.
What they did was to allow komodo to, vigilante style, outright steal the coins, and hold them hostage until you entertain them with proof of ownership/identity and whatever other ridiculous requirements they come up with.
But it's npm and kleptocurrency. Nothing of value lost.
Love that!
If I `npm init` and then `npm install express` as suggested, the result is 50 packages installed containing 21707 lines of javascript.
edit: trying to show what people are steered towards when looking up how to do this themselves.
https://twitter.com/dylanbeattie/status/1098204272653676544?...
"Before you even open a text editor, your project contains 1.5 million lines of code. Most of it contributed by volunteers and enthusiasts. No formal review or release process."
Take any other language with a web framework---Ruby with Rails, Java with Play/Spring, any web server which exposes a CGI interface, Python with Django/flask/whatever---and you will find a colossal amount of framework code behind the simple "hello, world!" demo.
Web apps are complex things, especially when the framework has to account for common cross-cutting concerns and patterns elegantly and succinctly. The fact that the JS ecosystem with NPM exposes that complexity to the user directly with an (albeit massive) node_modules installation isn't a strike against node per se: other languages have this code too, they just don't stash it all in the same folder as your project.
If you want to criticize the JS ecosystem, doing it on the basis of sheer SLOC downloaded by an `npm install` isn't too great.
We can talk about the quality of these packages, or about code review standards, or about the unique challenges auditing code from many sources for bugs and security problems. But talking about how the framework code gets installed seems counterproductive.
So yes, before you even open a text editor, you have eslint, jest, typescript etc all set up. Duh-doy, that's a lot of lines of code.
Sigh…
This comment is a bit meta, but I actually really like this "second chance" feature of hacker news. Mods could in theory "boost" such stories by making them sticky, but I like this kind of hands-off, no-meddling approach :)
> After being notified by our internal security tooling of this threat
you should never update a dependency unless you run a security check.
this isn't npm's fault.
Crazy idea - maybe you shouldn't be writing financial software if you're not going to audit the code that you're using
It should be possible to scan JS modules to determine either the fixed set of dependencies, the fixed set of system dependencies (ex: require('fs')), or whether it's not determinate (code contains "eval" or otherwise invokes require(...) with a non-constant). Including a package that has fs or net access would automatically include that taint as well.
The idea for this is that a malicious package would be more easily noticed as something like leftpad(...) suddenly requiring net access should be flagged. It's not a panacea as something malicious that already has fs or net access could do something new, but it would help add some sanity checks and give a smaller set of packages and versions to manually review.
I think deno (Node.js creator's new project using Typescript + Golang) has a similar idea but built in to the individual packages themselves where they need to explicitly include permissions for fs, net, etc. That'd be a great idea but getting to that point from the current (and growing) Node.js/NPM world is going to take a while.
A isolation/capability solution is the only one that could work, leftpad shouldn't have access to anything but basic CPU compute.
[1] https://github.com/entropic-dev/entropic