The App Store is the same as npm except it’s harder, but not impossible, to get malware on it. And with npm it’s a lot easier to do this without significant repercussions, or even rapid detection.
This is great! With some small automation for maintainers, we could help with the "Don't install anything that is less than 7/14/X days old" by just pinning dependencies and regularly publishing updates just for dependency tree updates.
For most projects the dep tree is small, and this level of maintenance is low. People always hate on me for saying it, but I think this level of effort would be worth it to protect the users of my modules.
I actually think there is room here for an upgrade "helper" or wrapper (yeah I know, I'm gonna try to solve npm package problems by making a new npm package!)
Something that can allow you to delay most updates by a set amount of time, only allow installing or updating dependencies based on a personal whitelist of contributors or authors (requiring some manual review perhaps of diffs from code that isn't from your approved contributors list), and a few other things.
It won't stop everything, and a determined attacker will still get things through, but it could at least make it a bit harder.
It doesn't scale. If a significant portion of the user-base starts doing this, what will happen is that security issues will be discovered 7/14/X days after they are introduced, not immediately.
Because the fundamental problem is still there: nobody is reviewing the code.
When I start a project I get whatever `npm install` returns. I freeze it. And then I only change it if one of a few things happens:
1. npm warns me of security issues
2. A new feature exists that I need
3. I've profiled and am seeking performance improvements
I used to just push everything forward on a regular basis in fear that I'd fall too far behind but I stopped doing that at some point and discovered that it doesn't really manifest as a problem. Chasing bugs and features usually keeps relevant packages current. Also the discipline not to depend on too many top-level dependencies.
...until you discover the feature you wanted is two major versions after yours, and you’ll spend days updating to the major version because you did not care to update. (eg reacr-router)
That one-time cost is at least an order of magnitude greater than the sum of the many-time costs, in my own experience. And that’s before you run into tangential problems like increased time to respond to security issues.
Further, I would bet a large sum of money that the overwhelming majority of vulnerabilities discovered and patched in open-source libraries are never officially reported and never have a CVE or OSVDB entry; they are simply patched in situ with no announcement or fanfare.
Got to be at least partially offset by the likelihood of new code/major rewrites introducing vulns. Kinda curious how the numbers work out on this. Best to install patch updates but hold off on major versions by default?
This is what I generally advocate. Pin to a major version and ruthlessly update. Major versions are fine to delay until an opportune time, but never to the point where you're even close to being more than one major version behind.
Client: "What do you mean it will cost X money and Y time to fix this trivial bug/feature in our App? And whats this babbling about having to upgrade at least a dozen frameworks? I just want this simple thing changed, not have my entire App rebuild. Last time you did that everything else broke and it took forever to get all the new bugs sorted."
Or worse: find out you need a security fix and that a large dependency chain is more than one major version behind. The issue is publically known and easy to scan for. The clock is ticking...
That would be your own fault for using immature libraries that don't offer stable versions with security and bug fixes. Unfortunately that's most of the npm library and a growing problem in other ecosystems.
I commented with a realistic and common use case in mind. We can talk idealistic / paranoid scenarios too, but that's just not how majority of developers work, so it's a bit of a moot point.
Have we sunk that low that the idea of using mature, stable and supported libraries is idealistic and paranoid now? It used to just be called responsible.
> but that's just not how majority of developers work, so it's a bit of a moot point
I don't think identifying the root cause of an issue is ever a moot point.
I think that's good reason to do some research on the main packages your rely on.
I've been burned by react-router more than once, so for my last few projects I went for a much older, stabler package (I think it was page.js). I didn't impact my project much, but saved me from dealing with the react-router hell of tight coupling with react combined with a history of ridiculous non-backwards-compatible updates.
npm doesn't auto update major versions by default anyway. So you will likely need to spend the same amount of work but during unscheduled time, updating code for features you didn't even need yet. It can be very disruptive to work in progress.
I've been burned twice in the last month from patch upgrades to packages. (And yes, I do lock my dependencies.)
The first time was upgrading pstree.remy from 1.1.0 to 1.1.2 to remove the malicious flatmap-stream. Totally incompatible upgrade. Nodemon would just crash, every single time.
The second was in an infrequent general update including a an upgrade of url-polyfill from 1.0.13 to 1.0.14. It changed new URL('https://example.com').host from 'example.com' to incorrect 'example.com:443'. This was used by angular-jwt for whitelisting URLs to send auth creds to, and wound up breaking access for all polyfilled browsers (i.e. just IE) in environments that used the HTTPS's default port (i.e. just production).
Somehow the npm ecosystem has (1) the package manager that most slavishly adheres to semver, while (2) having the community that least closely follows it.
One cool thing that yarn provides (npm might as well, but I'm not sure) is the ability to replace a transitive package with a specific version against what it wants, or even replace it with your own. I'm on mobile but I believe the term for it is "resolutions" and you put it in your package.json
I've used it before to replace deep dependencies with vulnerabilities and bugs with fixed versions or in one case my own implementation.
This is, alas, a result of a community that consists of hundreds of disconnected individuals who are each isolated working on their own little modules and don't need to talk to each other outside of NPM's interface. I doubt this would be such a problem if NPM encouraged larger libraries, or at least distributions of libraries that are managed by teams larger than 1.
Distributions would be a convenient patch which also feeds into our love for reinventing the wheel: an NPM package that depends on a bunch of other NPM packages, does a major update every six months, and provides a single (manually aggregated) changelog via a mailing list. Heck, you could probably monetize it with the changelog.
I'm not sure why people have chosen such small modules. Maybe ease of publishing means you can maintain 15 Node packages easier that 15 Maven artifacts.
I think NPM should store packages in predicable paths and not move them around. That way you can have your dependencies in source control and review all dependency updates. This worked great for me before npm started doing tree shaking. You can lock down the packages to semver minor and you will only get minor updates. Detecting something like an encrypted blob is pretty easy when you review a commit.
Hi specialist, thanks for your comment, I would be interested in working more on it and your comment about "hierarchical system of trust" would be awesome to work on!
None of this solves the referenced social engineering attack, except possibly "run everything in containers".
> Run everything in containers
This only works if you can split up your app into micro-services that can be adequately sandboxed, and even then only if the malicious dependency is in a locked-down service. But for example if your service needs to be able to make outgoing network requests to arbitrary hosts then you can't prevent a malicious dependency from sending your private data to its own server.
> Don't install anything that is less than 7/14/X days old
If everyone followed this advice, then the advice won't change anything. This advice only helps you if other people ignore it, at which it's just "let the suckers try the new versions first". And even then it won't solve the referenced social engineering attack, because that attack was targeting one specific user, and it doesn't matter how many other users test the dependency.
> Sign all packages published with a private key and Complain if the package you dependent on have changed signing key
This doesn't help either. The malicious package was published by an authorized maintainer of the project. Presumably if the package was signed with a private key, that private key would have been given to the maintainer to use in deploying (or alternatively the maintainer would have simply asked the original author to deploy the new version).
> Install only packages from authors you have marked as trusted
The malicious package was published under the name of a trusted author already.
> Only install packages that have been signed by 2/3/X other people
This isn't particularly scalable. It's hard enough as it is to get other people to even review the source of your open source library; asking them to stake their reputation on signing it too would be an even higher bar. Very few packages would end up cross-signed like this.
The advice not to install anything too new helps if there is a group of people beating up in new things specifically to find flaws. They are not "suckers", but "suck(urity resear)kers". :)
Hi eridius, thank you for your awesome comment, I really appreciate it!
> But for example if your service needs to be able to make outgoing network requests to arbitrary hosts then you can't prevent a malicious dependency from sending your private data to its own server.
Ideally you would know exactly which hosts you would want to talk to, but I know and understand that is a huge amount of work, but I know companies that are by default locking down everything and only opening up for things that are asked for by the app.
> If everyone followed this advice, then the advice won't change anything. This advice only helps you if other people ignore it, at which it's just "let the suckers try the new versions first". And even then it won't solve the referenced social engineering attack, because that attack was targeting one specific user, and it doesn't matter how many other users test the dependency.
That you can install something that is only 1 hour old without explicitly saying yes to that, exposes you to something where even outsiders wouldn't have time to even contact npmjs.com to take down the package.
And alpha and beta software is quite common, it is just now common in the node.js/npm sphere.
> This doesn't help either. The malicious package was published by an authorized maintainer of the project. Presumably if the package was signed with a private key, that private key would have been given to the maintainer to use in deploying (or alternatively the maintainer would have simply asked the original author to deploy the new version).
I trust Dominictarr and you would not give others your private key, so I would know that it was not signed by dominic, but by a new author, and that could be raised as a issue. The same thing happends with SSH, first time you connect to a server it generates a host key, if the ip of the server suddenly change, you have to explicitly regenerate the host again to be allowed to connect to it again. It does that so you don't connect to random servers if they were hacked.
> The malicious package was published under the name of a trusted author already.
The publishing rights were moved, but the new author was not a trusted author. The npm publishing model is not based on trust but based on permissions.
> This isn't particularly scalable. It's hard enough as it is to get other people to even review the source of your open source library; asking them to stake their reputation on signing it too would be an even higher bar. Very few packages would end up cross-signed like this.
Yes, but you could put a price on people verifying packages for you, you could make an incentive for people to verify. This is like reviews on Foursquare and all these things, were people make their buying decisions based on reviews.
1) We didn’t discover the hack for over 60 days. Recommending 7/14/X does not address this. What X is sufficient to ensure the community inspects new code? X=60 didn’t cut it.
2) Private signing keys are only private from us, the general public, not necessarily from maintainers, and may be sold and transferred with ease. If I were buying a codebase, I’d require the signing private keys for continuity reasons alone. (CAB EV rules permit this!)
Hi floatingatoll, thanks for the comment, I really appreciate it! It is valid reasons!
1) Yes, absolutely, it is quite interesting that there went so much time by, but that you can install npm packages without noticing that are less than > 1 hour old exposes you to risks that even people noticing can't prevent you from being exposed to.
2) That it could be sold, is a big problem that you can't solve, but if we discovered that it had been sold in this case, we cold then discard all packages that the author had also published, so a author would essentially put all his reputation at risk by just selling/giving away one package.
If I chose to sell my key, I’d simply generate a new one and start using it. If you then chose not to trust my old key retroactively, you would be unable to do your job without rewriting my code. Reputation has no material relevance today sufficient to compel a solution, sadly.
Yes, I would be unable to do my job, and I would have to consider the project security higher than features then. If I knew a project had been compromised, the next step should always be to move away from it or fix the issue.
The solution would maybe not be to delete all the code, but just not accept any updates since it is not sure the key belongs to the rightful owner.
45 comments
[ 3.0 ms ] story [ 69.3 ms ] threadEven the App Store doesn’t necessarily keep you safe; it’s pretty decent most of the time but malicious software sometimes slips through.
But that I can publish a virus to npm and you can download it immediately and that I can't do that with the AppStore is a big difference.
For most projects the dep tree is small, and this level of maintenance is low. People always hate on me for saying it, but I think this level of effort would be worth it to protect the users of my modules.
Something that can allow you to delay most updates by a set amount of time, only allow installing or updating dependencies based on a personal whitelist of contributors or authors (requiring some manual review perhaps of diffs from code that isn't from your approved contributors list), and a few other things.
It won't stop everything, and a determined attacker will still get things through, but it could at least make it a bit harder.
Because the fundamental problem is still there: nobody is reviewing the code.
Changing maintainers behaviors can maybe be difficult, so starting from a normal user could maybe be easier?
1. npm warns me of security issues
2. A new feature exists that I need
3. I've profiled and am seeking performance improvements
I used to just push everything forward on a regular basis in fear that I'd fall too far behind but I stopped doing that at some point and discovered that it doesn't really manifest as a problem. Chasing bugs and features usually keeps relevant packages current. Also the discipline not to depend on too many top-level dependencies.
Further, I would bet a large sum of money that the overwhelming majority of vulnerabilities discovered and patched in open-source libraries are never officially reported and never have a CVE or OSVDB entry; they are simply patched in situ with no announcement or fanfare.
if upgrading is that bad can always (usually) patch it yourself
Have we sunk that low that the idea of using mature, stable and supported libraries is idealistic and paranoid now? It used to just be called responsible.
> but that's just not how majority of developers work, so it's a bit of a moot point
I don't think identifying the root cause of an issue is ever a moot point.
I've been burned by react-router more than once, so for my last few projects I went for a much older, stabler package (I think it was page.js). I didn't impact my project much, but saved me from dealing with the react-router hell of tight coupling with react combined with a history of ridiculous non-backwards-compatible updates.
The first time was upgrading pstree.remy from 1.1.0 to 1.1.2 to remove the malicious flatmap-stream. Totally incompatible upgrade. Nodemon would just crash, every single time.
The second was in an infrequent general update including a an upgrade of url-polyfill from 1.0.13 to 1.0.14. It changed new URL('https://example.com').host from 'example.com' to incorrect 'example.com:443'. This was used by angular-jwt for whitelisting URLs to send auth creds to, and wound up breaking access for all polyfilled browsers (i.e. just IE) in environments that used the HTTPS's default port (i.e. just production).
Somehow the npm ecosystem has (1) the package manager that most slavishly adheres to semver, while (2) having the community that least closely follows it.
I've used it before to replace deep dependencies with vulnerabilities and bugs with fixed versions or in one case my own implementation.
Distributions would be a convenient patch which also feeds into our love for reinventing the wheel: an NPM package that depends on a bunch of other NPM packages, does a major update every six months, and provides a single (manually aggregated) changelog via a mailing list. Heck, you could probably monetize it with the changelog.
This is the correct answer.
I'm okay with using aliases (links) which resolve to preferred semver. Much like nvm does for nodejs runtimes.
Especially with respect to self-hosted repos, eg Artifactory. Enabling a hierarchical system of trust.
> Run everything in containers
This only works if you can split up your app into micro-services that can be adequately sandboxed, and even then only if the malicious dependency is in a locked-down service. But for example if your service needs to be able to make outgoing network requests to arbitrary hosts then you can't prevent a malicious dependency from sending your private data to its own server.
> Don't install anything that is less than 7/14/X days old
If everyone followed this advice, then the advice won't change anything. This advice only helps you if other people ignore it, at which it's just "let the suckers try the new versions first". And even then it won't solve the referenced social engineering attack, because that attack was targeting one specific user, and it doesn't matter how many other users test the dependency.
> Sign all packages published with a private key and Complain if the package you dependent on have changed signing key
This doesn't help either. The malicious package was published by an authorized maintainer of the project. Presumably if the package was signed with a private key, that private key would have been given to the maintainer to use in deploying (or alternatively the maintainer would have simply asked the original author to deploy the new version).
> Install only packages from authors you have marked as trusted
The malicious package was published under the name of a trusted author already.
> Only install packages that have been signed by 2/3/X other people
This isn't particularly scalable. It's hard enough as it is to get other people to even review the source of your open source library; asking them to stake their reputation on signing it too would be an even higher bar. Very few packages would end up cross-signed like this.
> But for example if your service needs to be able to make outgoing network requests to arbitrary hosts then you can't prevent a malicious dependency from sending your private data to its own server.
Ideally you would know exactly which hosts you would want to talk to, but I know and understand that is a huge amount of work, but I know companies that are by default locking down everything and only opening up for things that are asked for by the app.
> If everyone followed this advice, then the advice won't change anything. This advice only helps you if other people ignore it, at which it's just "let the suckers try the new versions first". And even then it won't solve the referenced social engineering attack, because that attack was targeting one specific user, and it doesn't matter how many other users test the dependency.
That you can install something that is only 1 hour old without explicitly saying yes to that, exposes you to something where even outsiders wouldn't have time to even contact npmjs.com to take down the package.
And alpha and beta software is quite common, it is just now common in the node.js/npm sphere.
> This doesn't help either. The malicious package was published by an authorized maintainer of the project. Presumably if the package was signed with a private key, that private key would have been given to the maintainer to use in deploying (or alternatively the maintainer would have simply asked the original author to deploy the new version).
I trust Dominictarr and you would not give others your private key, so I would know that it was not signed by dominic, but by a new author, and that could be raised as a issue. The same thing happends with SSH, first time you connect to a server it generates a host key, if the ip of the server suddenly change, you have to explicitly regenerate the host again to be allowed to connect to it again. It does that so you don't connect to random servers if they were hacked.
> The malicious package was published under the name of a trusted author already.
The publishing rights were moved, but the new author was not a trusted author. The npm publishing model is not based on trust but based on permissions.
> This isn't particularly scalable. It's hard enough as it is to get other people to even review the source of your open source library; asking them to stake their reputation on signing it too would be an even higher bar. Very few packages would end up cross-signed like this.
Yes, but you could put a price on people verifying packages for you, you could make an incentive for people to verify. This is like reviews on Foursquare and all these things, were people make their buying decisions based on reviews.
Many OSS devs have to handle several projects AND a job/freelancing at the same time. More money, less work for others, more time for OSS.
1) We didn’t discover the hack for over 60 days. Recommending 7/14/X does not address this. What X is sufficient to ensure the community inspects new code? X=60 didn’t cut it.
2) Private signing keys are only private from us, the general public, not necessarily from maintainers, and may be sold and transferred with ease. If I were buying a codebase, I’d require the signing private keys for continuity reasons alone. (CAB EV rules permit this!)
1) Yes, absolutely, it is quite interesting that there went so much time by, but that you can install npm packages without noticing that are less than > 1 hour old exposes you to risks that even people noticing can't prevent you from being exposed to.
2) That it could be sold, is a big problem that you can't solve, but if we discovered that it had been sold in this case, we cold then discard all packages that the author had also published, so a author would essentially put all his reputation at risk by just selling/giving away one package.
The solution would maybe not be to delete all the code, but just not accept any updates since it is not sure the key belongs to the rightful owner.