Ultimately, I hope we end up getting (in some future) some form of "cross language" global network of versions & vulnerabilities signaling, with e.g. alerts when you install something, or at runtime, and push notifications for all "subscribers" (outside of a paid product).
OWASP [1] maintains npm, maven (and nuget, I believe) plugins for detecting use of packages with known vulnerabilities, and a vulnerability db, of course.
For those who haven't yet come across it, bundler-audit is a very useful tool to pick up any old and/or insecure gems in your application. It should be part of your CI pipeline, or at a minimum, run locally every so often.
Is there also a bundler-meta-audit the somewhere, so I can audit bundler-audit and it's dependencies? If not, maybe we should start a kickstarter for it?
This is very timely: I was trying to start up an old Rails app today and when I ran the bundle install command, I got a warning that there were no sources for this gem... now I'm glad that this failed to install!
Not this kind of audits. Is there any major language for which all third party modules are audited for security before they are released? I would be surprised.
gems don't, NPM doesn't, PyPI doesn't, NuGet doesn't.... you get the picture.
Basically all apps using package repo's (i.e. all of them) are relying on massive piles of unaudited 3rd party code with usually no idea of provenance.
> We assume that the attacker has obtained the credentials to publish the malicious RubyGems package from one of the two maintainers, but this has not been officially confirmed.
Didn't this _also_ require a commit to the relevant Git repo? Or is it possible to upload a tarball directly to Rubygems without it being backed by a git repo?
> The backdoor was wisely hidden in the 3.2.0.3 version that was only published to RubyGems and no source of the malicious version existed on the GitHub repository and allowed remote attackers to dynamically execute code on servers hosting the vulnerable versions.
I really don't know what we can do as developers to prevent this. What else could be done? Is there any package manager system that isn't vulnerable to an individual being compromised? Having an audit-tool is only useful after you've been compromised and the vulnerability is discovered. Some kind of vulnerability scanner in all pulled in code might be one idea, but much like the cat-mouse battle with AV's, I don't think would be very successful.
Ideas (shoot them down):
- Somehow tie keys to location.
- Require two-step verification for all commits for packages with over 1000 downloads. (Assuming their phone isn't compromised if the keys are).
> Is there any package manager system that isn't vulnerable to an individual being compromised?
One that comes closer is vcpkg [1], which is the Microsoft-run package manager for native C++ packages. It works by having the list of packages and recipes to build them (which it calls "ports") stored in a single repository. All requests to update them are submitted as pull requests and must be approved by Microsoft to enter the main repository.
I'm not necessarily saying whether this is a good or bad idea. Certainly it is a very different situation: only libraries that are already popular are considered, often the ports are submitted/maintained by different people than the library authors, and writing a port to build a library on multiple platforms may require significantly more work than just pointing at an existing GitHub repo. I just point to it as an alternative idea that has had some success.
That's how Homebrew[1] works as well. Any external file pulled during installation requires a matching hash in the main homebrew repo to prevent tampering after publishing.
Lots of things could be done, it's whether people are willing to accept the tradeoffs, which they haven't been (in general) up until now. Unfortunately, modern software development has heavily embedded the idea of using large numebers of un-reviewed open source libraries, changing that assumption would be very difficult.
A non-exhaustive list of things that could be done...
- Improve repository security (e.g. stronger authentication requirements). Problem here is most of the repositories are non-profit and this costs money. Also there's a risk that if you increase friction for developers, they'll go elsewhere.
- Require library signing. This has some potential benefits (may even have stopped this attack) but again increased friction for developers and management overhead for repositories.
- Curation of libraries. Actually pay some people to review the code in the libraries. This doesn't scale easily when you consider the volume that places like NPM have.
If a library doesn't need any permissions, and it's not granted any, then attempts to e.g. access the filesystem or network will throw exceptions. The JVM can do this, but it's not well documented how to use it.
> This doesn't scale easily when you consider the volume that places like NPM have.
Found the blanket statement for npm.
How about instead of reviewing 1 library of 10k lines that does everything, you review 10 focused modules for a total of 1k lines?
Also modules like left-pad don’t change that often so you probably have to review only changes in the top modules.
We need to stop this mentality that dependency count should be low. Look at total LOC and you’ll probably find that it’s not as bad as people like to paint it.
A high dependency count does increase your vulnerability surface area. All it takes is one weak or reused password on an account without 2FA enabled for a malicious package to be uploaded. You're more vulnerable to this when you depend on more individual publishers to get it right.
I propose the radical idea of drastically reducing the number of transitive dependencies we rely on. I don't think that it's okay that I need hundreds of megabytes of unreviewed libraries for running relatively simple programs.
When it comes to verifying this, we need to have independent rebuilders that build these packages reproducibly, and can attest these rebuilds in a secure manner. I have been working with NYU Secure System Labs to try provide this to debian packages and we currently have a PoC for such a system.
Sure there are ways to do it, but every piece of software needs to be aligned. What I was trying to say is that even if you have the tools you probably cannot "just use the tools".
You will likely have to create patches for those piece of softwares that create a different artifact every time are built.
There are a few other challenges but in my humble opinion, it is worthwhile to strive to write your own software with "reproducible builts" in mind so everybody can verify if something was altered.
We are well aware of the challenges. Read a bit into the documentation and the efforts that has gone into achieving reproducible builds on different distributions.
This is hard on distributions as we are suppose to support all the ecosystems. But for rubygems, pypi, go, rust, npm and so on this is a easier problem to solve if they want to. But even with all these challenges, Debian is at 94% reproducible packages, and Arch is edging closer to 80%.
Introducing non-determinism into your build process is something that should be removed, and all ecosystems should be aligned in this effort.
Debian had mandatory use of OpenPGP by developers for twenty years, although I don't think it's linked to strict package signing.
Of course, this adds a big deliberate barrier to participation - newcomers have to get their key signed by an existing Debian developers. There are real security/usability tradeoffs, and at the moment people seem to have default-chosen "anyone can add packages really easily", resulting in a Cambrian explosion of packages.
Keys are somewhat useless in this scenario. Developers are hacked all the time and not all of them pass phrase their keys. And if someone is going through the effort to backflip a popular package via a developer they’d probably keylig the pass phrase too.
> Require two-step verification for all commits for packages with over 1000 downloads.
Just require 2fa everywhere. We're way past the point where password-only is enough if you distribute software.
Package distribution platforms could also ensure that your package comes from a tagged/signed commit in a selected repo. Reproducible builds could help here too. Finally package signing itself and possibly WoT for authors. (Where either the author has to trust the dependencies or you have to explicitly trust all of them)
This is going to practically stop remote attacks with stolen credential. But there are a few other attacks that it will not stop. For example the NPM event-stream kind of attacks.
There are some things to do, but not if you want to trust random third parties. In which case nothing will save you, because not only you have to trust them, you also have to trust they are not compromised and have impenetrable security.
So, not trusting third parties is the only way and with that language level capability based security can help. Although this is pretty far from common programming languages where any function can access any file, environment variable, global variable, constant, socket, etc.
The only way to completely prevent to get malicious code in your products is writing all code by yourself including the interpreter/compiler and maybe the operating system too. Machines should be off the Internet of course and security should be there 24/7. And of course you need to have a very high bar and a paranoid process to select the people you hire.
But now you will move at the pace of a bacteria on a snail. But since the company will probably introduce vulnerability bugs in the code they write, it is probably not a very good idea.
We are using open source and free packages to save time and move fast. Security and speed/usability have tradeoffs.
The paranoid will do everything in their knowledge to protect themselves but they will move slowly. The naive will not do anything and sooner than later will get burnt.
Assuming you trust your current modules and you can trust that they will not be altered, you can lock down the dependencies. But now monitor and updating a library when a bug or vulnerability is found in a modules become another burden.
Those actors that have the budget, patience and the motivation will always find a way in.
I sometimes feel quite sad about the real situation but there is not such a thing as perfect security.
That said some of the suggestions I see here will dramatically improve the situation with a relatively small effort. 2FA and reproducible builds for example.
1) Signing should always happen. Publish time + hashes of external resources should be taken into account to prevent meddling with things after the fact.
2- a much harder solution) Don't use so many libraries.
In this way I think it's borderline impossible. Of course this vuln targeted `bootstrap-sass`.. it's basically a default gem for new rails installations since bootstrap is so commonly used.
Actually, rails 6.0 not only installs dependencies for you upon generating a new app, but it downloads yarn and installs 602 dependencies too.
I really like the idea of a cryptographically provable signature for a set of coordinates in space and time, but not sure how such a thing is possible.
Another one in the line of inevitable supply chain attacks on software libraries.
This isn't going to go away, it's just a question of if/when the damage caused will get bad enough that people start to accept some of the tradeoffs that would be required to improve matters...
Can you elaborate on what do you think would be a good solution to this problem? In my opinion, having big central sources of libraries makes it easier for security scanning services to operate. Companies hosting libraries locally (e.g. with Sinopia, Nexus, Artifactory) have to proactively monitor them, but most commonly will let them frozen at a specific version and miss all security patches.
Really the only "solution" is to ensure that you have some way to establish trust in all the software libraries you use.
For commercially provided libraries that could be a contract specifying security requirements, with some specific measures of how to do that.
For Open source the only real option I can see is to curate your own package repositories and get a level of review that you're comfortable with. Definitely fixing versions and insisting on review before upgrade would help.
The problem with automated scanning is that they can't really find backdoors or just generally insecure code, they can find known vulnerabilities. They could use static analysis to find insecure code, but my experience of SAST tooling is that it takes a lot of manual effort to tune, it's not a pure automated scan option.
That's not to say that automated scanning provides nothing, but that it has limitations.
sure but what's the alternative? Unlike mail servers where there's tonnes of good commercial solutions you can use, I'm not aware of many good options for curated software library repo's
There's tools you can use to add automated scanning to your repo's and pin versions of packages, but that's doesn't really feel like curation to me...
companies like Sourceclear seemed to me to be going down this line but that doesn't seem to be the drive of their offerings any moe.
In the end it all comes down to reviewing code you have to trust. It's just that it's too much for most code, so there has to be something that makes potentially insecure code explicit and dealt with on use and the rest of the code usable without having to trust it.
Reviewing code mitigates a lot the risk but it does not solve everything (ex: vulnerabilities hidden in plain sight).
It also implies that you are compiling everything from source code or you trust the distribution channel of the binaries.
And you do it every time there is an update that you want to use. And you have a system that promptly alert you of vulnerabilities in libraries you use.
Unless you have a handful of modules that are already self contained, it is not practical for most teams.
Anyone maintaining a library with a large audience of consumers should be mandated to use 2FA/MFA, I can only assume from the information here that they only had a set of credentials for authing :-(
- installs yarn & 602 yarn dependencies (without being asked)
The modern programming industry is a joke. We call ourselves "engineers" without any certification or oaths of responsibility, and basically ignore security beyond user credentials at best.
Someone please tell me `yarn list | wc` showing 2,800 transient packages is incorrect.
And it is worse that what you just found out. Did you think about all the software and dependencies that are installed just to get to the point where you can install Rails 6 (ex: Linux)?
What about your coworkers? Are you sure that you can trust all of them and nobody will not sneak anything in?
If you are too paranoid on who and what you can trust, you may end up tossing all your electronic devices and go living in a desert island.
Those attack vectors you mentioned are already known, and we essentially have to live with them. When it comes to "reflecting on trusting trust", we already know that it's basically impossible.
However, this does not mean we should by default be installing heaps of packages upon generating new packages. This is practically begging for a security exploit, particularly the JS packages.
I am not sure why the npm packages in yarn should be more exploitable than Linux.
- Linux is more ubiquitous than Node.js. If somebody exploits Linux, they get at least one order of magnitude more machines. So there is a higher motivation.
- Funny code in C is harder to spot than in Javascript. Furthermore ...
- A binary is harder to inspect than any module written in Javascript (even if minified).
- The code in the Linux default installation has at least 2 orders of magnitudes of the code that yarn installs.
I was very conservative in my estimations.
Considering these kind of attacks, Linux seems more likely to be already exploited by a few organizations in multiple ways.
We have been trusting too much. As more code gets written and many more people come to learn to program, the number of supply chain attacks will increase. And at the same time, as the security know-how is easier to be accessed on the Internet, we will get smarter exploiters capable of hiding their wrong doing.
Of course, the fifth is `@rails/webpacker` which has `webpack`, `babel`, and friends as dependencies. And the node ecosystem is what it is, but I think `yarn list | wc` is going to double count shared dependencies, so 2800 is high.
TL;DR - I think new rails is about as minimal a footprint as possible...
Do people in mission critical systems (like life threatening) write code from scratch all the time without using any external dependencies? I'm sure they don't reinvent encryption.
I'd like to know how they audit their system and the threshold of saying it is adequately secure.
60 comments
[ 5.0 ms ] story [ 118 ms ] thread- https://github.com/twbs/bootstrap-sass/issues/1195
- https://github.com/rubygems/rubygems.org/issues/1941
I hope others copy this. This is truly scary.
https://github.com/rubygems/rubygems/issues/2290
https://reproducible-builds.org/docs/source-date-epoch/
[1]: https://www.owasp.org/
https://github.com/rubysec/bundler-audit
Is there also a bundler-meta-audit the somewhere, so I can audit bundler-audit and it's dependencies? If not, maybe we should start a kickstarter for it?
Basically all apps using package repo's (i.e. all of them) are relying on massive piles of unaudited 3rd party code with usually no idea of provenance.
Didn't this _also_ require a commit to the relevant Git repo? Or is it possible to upload a tarball directly to Rubygems without it being backed by a git repo?
> The backdoor was wisely hidden in the 3.2.0.3 version that was only published to RubyGems and no source of the malicious version existed on the GitHub repository and allowed remote attackers to dynamically execute code on servers hosting the vulnerable versions.
Ideas (shoot them down): - Somehow tie keys to location. - Require two-step verification for all commits for packages with over 1000 downloads. (Assuming their phone isn't compromised if the keys are).
One that comes closer is vcpkg [1], which is the Microsoft-run package manager for native C++ packages. It works by having the list of packages and recipes to build them (which it calls "ports") stored in a single repository. All requests to update them are submitted as pull requests and must be approved by Microsoft to enter the main repository.
I'm not necessarily saying whether this is a good or bad idea. Certainly it is a very different situation: only libraries that are already popular are considered, often the ports are submitted/maintained by different people than the library authors, and writing a port to build a library on multiple platforms may require significantly more work than just pointing at an existing GitHub repo. I just point to it as an alternative idea that has had some success.
[1] https://brew.sh/
A non-exhaustive list of things that could be done...
- Improve repository security (e.g. stronger authentication requirements). Problem here is most of the repositories are non-profit and this costs money. Also there's a risk that if you increase friction for developers, they'll go elsewhere.
- Require library signing. This has some potential benefits (may even have stopped this attack) but again increased friction for developers and management overhead for repositories.
- Curation of libraries. Actually pay some people to review the code in the libraries. This doesn't scale easily when you consider the volume that places like NPM have.
- Sandbox code modules
If a library doesn't need any permissions, and it's not granted any, then attempts to e.g. access the filesystem or network will throw exceptions. The JVM can do this, but it's not well documented how to use it.
Found the blanket statement for npm.
How about instead of reviewing 1 library of 10k lines that does everything, you review 10 focused modules for a total of 1k lines?
Also modules like left-pad don’t change that often so you probably have to review only changes in the top modules.
We need to stop this mentality that dependency count should be low. Look at total LOC and you’ll probably find that it’s not as bad as people like to paint it.
https://reproducible-builds.org/docs/buy-in/
When it comes to verifying this, we need to have independent rebuilders that build these packages reproducibly, and can attest these rebuilds in a secure manner. I have been working with NYU Secure System Labs to try provide this to debian packages and we currently have a PoC for such a system.
https://reproducible-builds.org/docs/sharing-certifications/
https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-pari...
But I think this goes in the right direction.
You will likely have to create patches for those piece of softwares that create a different artifact every time are built.
There are a few other challenges but in my humble opinion, it is worthwhile to strive to write your own software with "reproducible builts" in mind so everybody can verify if something was altered.
This is hard on distributions as we are suppose to support all the ecosystems. But for rubygems, pypi, go, rust, npm and so on this is a easier problem to solve if they want to. But even with all these challenges, Debian is at 94% reproducible packages, and Arch is edging closer to 80%.
Introducing non-determinism into your build process is something that should be removed, and all ecosystems should be aligned in this effort.
Weekly blog posts are written about the current efforts. https://reproducible-builds.org/blog/
Of course, this adds a big deliberate barrier to participation - newcomers have to get their key signed by an existing Debian developers. There are real security/usability tradeoffs, and at the moment people seem to have default-chosen "anyone can add packages really easily", resulting in a Cambrian explosion of packages.
Just require 2fa everywhere. We're way past the point where password-only is enough if you distribute software.
Package distribution platforms could also ensure that your package comes from a tagged/signed commit in a selected repo. Reproducible builds could help here too. Finally package signing itself and possibly WoT for authors. (Where either the author has to trust the dependencies or you have to explicitly trust all of them)
So, not trusting third parties is the only way and with that language level capability based security can help. Although this is pretty far from common programming languages where any function can access any file, environment variable, global variable, constant, socket, etc.
But now you will move at the pace of a bacteria on a snail. But since the company will probably introduce vulnerability bugs in the code they write, it is probably not a very good idea.
We are using open source and free packages to save time and move fast. Security and speed/usability have tradeoffs.
The paranoid will do everything in their knowledge to protect themselves but they will move slowly. The naive will not do anything and sooner than later will get burnt.
Assuming you trust your current modules and you can trust that they will not be altered, you can lock down the dependencies. But now monitor and updating a library when a bug or vulnerability is found in a modules become another burden.
Those actors that have the budget, patience and the motivation will always find a way in.
I sometimes feel quite sad about the real situation but there is not such a thing as perfect security.
That said some of the suggestions I see here will dramatically improve the situation with a relatively small effort. 2FA and reproducible builds for example.
1) Signing should always happen. Publish time + hashes of external resources should be taken into account to prevent meddling with things after the fact.
2- a much harder solution) Don't use so many libraries.
In this way I think it's borderline impossible. Of course this vuln targeted `bootstrap-sass`.. it's basically a default gem for new rails installations since bootstrap is so commonly used.
Actually, rails 6.0 not only installs dependencies for you upon generating a new app, but it downloads yarn and installs 602 dependencies too.
It's a miracle we do anything as programmers.
This isn't going to go away, it's just a question of if/when the damage caused will get bad enough that people start to accept some of the tradeoffs that would be required to improve matters...
For commercially provided libraries that could be a contract specifying security requirements, with some specific measures of how to do that.
For Open source the only real option I can see is to curate your own package repositories and get a level of review that you're comfortable with. Definitely fixing versions and insisting on review before upgrade would help.
The problem with automated scanning is that they can't really find backdoors or just generally insecure code, they can find known vulnerabilities. They could use static analysis to find insecure code, but my experience of SAST tooling is that it takes a lot of manual effort to tune, it's not a pure automated scan option.
That's not to say that automated scanning provides nothing, but that it has limitations.
This is a non-starter. This is even worse than "just run your own mail server".
There's tools you can use to add automated scanning to your repo's and pin versions of packages, but that's doesn't really feel like curation to me...
companies like Sourceclear seemed to me to be going down this line but that doesn't seem to be the drive of their offerings any moe.
It also implies that you are compiling everything from source code or you trust the distribution channel of the binaries.
And you do it every time there is an update that you want to use. And you have a system that promptly alert you of vulnerabilities in libraries you use.
Unless you have a handful of modules that are already self contained, it is not practical for most teams.
Rails 6.0, when generating a new app:
- installs 102 ruby gems (without being asked)
- installs yarn & 602 yarn dependencies (without being asked)
The modern programming industry is a joke. We call ourselves "engineers" without any certification or oaths of responsibility, and basically ignore security beyond user credentials at best.
Someone please tell me `yarn list | wc` showing 2,800 transient packages is incorrect.
[0]: https://gist.github.com/azah/c219844f95243cdfdb3b352ad3dee2f...
And it is worse that what you just found out. Did you think about all the software and dependencies that are installed just to get to the point where you can install Rails 6 (ex: Linux)?
What about your coworkers? Are you sure that you can trust all of them and nobody will not sneak anything in?
If you are too paranoid on who and what you can trust, you may end up tossing all your electronic devices and go living in a desert island.
However, this does not mean we should by default be installing heaps of packages upon generating new packages. This is practically begging for a security exploit, particularly the JS packages.
[0]: https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...
- Linux is more ubiquitous than Node.js. If somebody exploits Linux, they get at least one order of magnitude more machines. So there is a higher motivation.
- Funny code in C is harder to spot than in Javascript. Furthermore ...
- A binary is harder to inspect than any module written in Javascript (even if minified).
- The code in the Linux default installation has at least 2 orders of magnitudes of the code that yarn installs.
I was very conservative in my estimations.
Considering these kind of attacks, Linux seems more likely to be already exploited by a few organizations in multiple ways.
We have been trusting too much. As more code gets written and many more people come to learn to program, the number of supply chain attacks will increase. And at the same time, as the security know-how is easier to be accessed on the Internet, we will get smarter exploiters capable of hiding their wrong doing.
Just doing some eyeballing:
- These look like the 62 gems in the production group: https://gist.github.com/gkemmey/6d6fcd381596b1b355d16e1e3120...
- Of those 62, how many are first party (by rails or other trusted vendors)? Looks like the vast majority...
- These look like the other 16 (dev / test groups): https://gist.github.com/gkemmey/6d6fcd381596b1b355d16e1e3120...
- On the JavaScript side, rails adds 5 top-level dependencies. 4 of those 5 have one external dependency between them: https://gist.github.com/gkemmey/6d6fcd381596b1b355d16e1e3120...
Of course, the fifth is `@rails/webpacker` which has `webpack`, `babel`, and friends as dependencies. And the node ecosystem is what it is, but I think `yarn list | wc` is going to double count shared dependencies, so 2800 is high.
TL;DR - I think new rails is about as minimal a footprint as possible...
I'd like to know how they audit their system and the threshold of saying it is adequately secure.