It won't be just companies. It'll be developers, sysops, etc who npm install a bazillion of packages, because the core language and libaries are not enough. Those people have keys, credentials and access to the internal networks.
Yep a big part of npm’s problems are actually just flaws with JS. Comparing with Java, it’s insane how many dependencies you have to manage, and that batteries are not included (esp weird when you consider front end apps involve downloading the code!).
I’m cackling at how great this is. This is what happens when you trust the internet forever and just scarf down any old thing at build time. Of course it’ll get exploited! That’s what evil people do.
There's more coming.... tons of github integrations ask for blanket access to your account vs Oauth, (https://github.com/marketplace). Tons of github users give that access, the access_tokens are only a password type breach away. If you have these access_tokens you can edit the repos they are for all you want.
Yeah, but at least with PATs (not sure about other token types), you can't scope them to a particular repo, so whenever you need to allow something to even see a private repo or write to a public repo, the token you supply to allow that can do that for all repos and that alone is potentially really destructive. I am not sure if there is a good reason for why PATs can't be scoped to a repository, because if they were allowed to be, it would do a lot for security I think.
We also have multiple orgs, but we hit the requirement that you only have one bot account. It would be super nice if GitHub allowed much tighter scoping for PATs.
I don’t know how many GitHub orgs the Linux Foundation has, but... hundreds? Having one bit account with wide permissions is a non starter
Most integrations just ask for blanket all permissions. They do this because it means they can give you a list of repos and let you choose which ones to integrate their service into with no work on your part except "click yes to give us permission to do everything for you and ... we'll do everything for you"
Apparently the Oauth scopes are much worse than GitHubs apps. Only GitHub apps allow read only access to the “metadata” by default whereas Oauth apps get access to the code, deploy keys, etc with no way to limit that access per repo.
At the moment, there's a story about github1s.com on the front page of HN and people are asking how to give it access to their company private repos [1][2]. Scary.
I wish GitHub would create a proper auth design. I won’t grant blanket permissions to tokens because there’s too much risk of something going wrong.
It seems dumb that they don’t have per repo tokens. I think the issue is with their licensing as if they made proper tokens users could abuse it by giving tokens to their friends. But this should be detectable in a friendly (please don’t do that) way.
I want to be able to give read-only access to private repos.
I want to be able to give fine grained function level and repo level access.
If I’m an admin on multiple repos, I want to be able to issue a token for just a single repo so I can give that to a CI job without worrying if every single repo I admin is at risk.
They allow ssh keys with some similar functionality, but ssh keys can’t be used as much as tokens.
I’ve been waiting for a story about how some third party app granted access to my whole org gets taken over and wreaks havoc. Eventually this will probably be the attack that alters real packages instead of these name overloading packages.
> It seems dumb that they don’t have per repo tokens.
Technically you can create one new GitHub account per repo and generate a token for that... But that is highly annoying :)
They need to support IAM / RBAC style policies and tie every authn+z method to those policies, but my guess is they have different auth methods strung all throughout their codebase so implementing it will take a few years. Then of course they have to make it "user friendly" as we all know how painful IAM can be...
Comically, that’s why my GitHub recommended. Of course that’s a nightmare for a user to manage, violates our sso requirement, and GitHub charges per user.
I'm more amazed by the fact that they got bounties because the attack wouldn't be (easily) possible without insider knowledge on which dependencies their internal build system used
> To test this hypothesis, Birsan began hunting for names of private internal packages that he could find in manifest files on GitHub repositories or in CDNs of prominent companies but did not exist in a public open-source repository.
If I'm not mistaken insider knowledge wasn't necessary.
There are a lot of expensive things you can outsource. Responsibility isn't among those.
Free software / open source propels engineering as you can share and leverage the results of collective efforts. However, at no point did the concept come with inherent guarantees about concerns such as security.
esr defined 19 points for "good" open source software development in his seminal essay "The Cathedral and the Bazaar". I feel some of those are sometimes easily thrown out of the window for the sake of "efficiency" or "cost-effectiveness".
This issue resonates with bullet point 17 in particular:
> A security system is only as secure as its secret. Beware of pseudo-secrets.
I think this issue has less to do with package managers, and a lot with companies rushing into the convenience of public code platforms such as Github without properly vetting whether or not they might be inadvertently leaking internal information through packaging manifests.
Offtopic, but I found nowhere to actually ask this question. Does anybody know if ESR is still alive? His blog [1] has not been updated in months--and looking at his post dates, this seems really out of character--, he hasn't posted anything on twitter, or his usual channels.
>>There are a lot of expensive things you can outsource. Responsibility isn't among those.
That is not true at all, the industry both Development and even more so in Operations has been outsourcing responsibility for a long time, they is why we have support contracts, SLA's and other very expensive services we pay many many times more than the cost of hardware for...
To outsource responsibility... Network down -- Call Cisco... Storage Down Call EMC or Nimble... etc
A support contract allows you to hold a sub-contractor accountable. But that's the extent of what an SLA does. What it doesn't do is diminish your responsibility towards anyone who relies on the services you provide yourself. These are distinct things.
Put more succinctly, if the network's down: that's still very much your problem. Especially if you over-promised 100% availability to the end users of your services. Your end users do not care about Cisco, EMC or Nimble. They don't have contracts with any of those. They have a contract with you and they can and will hold you accountable if you don't deliver on what you sold them.
I guess this is where we need to define our anology
for a sysadmin the customer is "the employer" and they do not really have a contract with the sysadmin, rather the employer has contract with Cisco, or Nimble, etc. the sysadmin has "outsourced" his/her responsibility in that context.
For example instead of rolling your own storage device using linux, or freenas or something else, you buy an expensive 3rd party solution with expensive support contracts to outsource their responsibility. If it goes down "I have a support ticket open with vendor" instead of "I am attempting to merge the lastest kernel patch I have downloaded from github"
That is the source of the phrase "No one ever got fired for buying Cisco" or insert name of other large vendor. They do not get fired for it because they have outsourced their responsibility
That's a fair point. And it's a good point. There's a difference in types of contracts and the relationships they represent. An employee/employer relationship is distinct from a customer/vendor relationship.
An employee/employer relationship is defined by a few key properties. As an employee, you sell your time and your expertise to your employer, and you agree to submit to the authority of your employer in exchange for a salary. The extent of your responsibility - and this is absolutely key - is captured in your contract.
It also means that many things simply aren't your responsibility to begin with, even though you deal with them on a day-to-day basis.
As a systems administrator you, quite likely, won't get fired for failing Cisco gear or services because you're not the one who ultimately signs off on the contract with Cisco on behalf of your employer. Responsibility has always resided with the executives who cover and sanction your actions and decisions.
An executive, though, usually won't get fired over failing Cisco gear/services itself, but they will get fired over any ensuing loss of revenue, damage to brand/image, litigation over exposed liabilities,...
A great example of this is President Harry S. Truman who famously had a sign on his desk stating "The buck stops here".
As for the systems administrator, your role is to actively engage in day-to-day operations. You're basically hired "to keep the lights on". Whether the proverbial "light" was procured from Cisco or handcrafted in-house is inconsequential to your employer as far as your individual role as an employee is concerned.
Not sure if serious, but I will point out this is significantly different. If I'm installing an application like homebrew or the Rust toolchain then I am explicitly giving them the right to code execution. It doesn't much matter whether they get it through the script on their website or the binaries downloaded from that website.
Random libraries, possibly pulled in by a dependency of a dependency of a dependency... not so much.
Honestly, it’s one of the things that makes me nervous about running Linux on all of my computers. At least with Windows (and probably OSX), my updates come from a single vendor who has strict internal code audits and security requirements. With Linux (I’m using Pop), my updates come from a package manager with a crapload of packages, each maintained by a different team / group with no central policy. There’s no way the small team at Pop can review and audit all of the things in the apt package system, and there have to be plenty of maintainers of popular packages who get sweet offers to sell out.
Anyway. I’m sticking with Pop / Linux. But it does make me nervous!
At least with windows, the drivers aren't checked that much and accordingly, have had some serious issues.
I'd guess distros are generally better off in that respect, but kernel space & user space aren't that different nowadays, when caring about your own security
I'm very happy to finally have a real world example to motivate all the folks that eye-rolled me every time I've raised it in the past. It just resonates better, especially with less technical leadership folks.
Helps in this specific case, but will not eliminate the broader issue. The broader issue is how can you trust 3rd party code to not do anything harmful, and it’s not like we can even perfectly trust our own fingers in that regard.
But the broad issue will never be resolved. The entire foundation of society is built around some trust. What has proven to work is that we trust actors who are identifiable and have something to lose. This goes with everything from your bank, to government, to name brands, to your local restaurant.
Nix package managers/repositories have a level of scrutiny to get into, and highly dedicated people in charge of. Random github repos (or npm packages) are extremely low effort/risk to set up.
Of course the former can be abused, but the incentives are at least in its favor to likely be more trustworthy. And we have to make assumptions of trust everytime we sit down in our chair or turn on our computer, or plug in a space heater. We will never get around trust, but there are differences in levels of trust and trustworthiness.
>Nix package managers/repositories have a level of scrutiny to get into, and highly dedicated people in charge of. Random github repos (or npm packages) are extremely low effort/risk to set up.
That's not really true though. Nix doesn't support signed sources, there are no signatures in the package repository and in theory "John Doe" with no information can add packages and send pull-requests.
In practise nixpks is just a well moderated user repository and the level of scrutiny is less then the enterprise distros can offer.
That is insane that any company allowed this to happen.
""That said, we consider the root cause of this issue to be a design flaw (rather than a bug) in package managers that can be addressed only through reconfiguration," a Microsoft spokesperson said in the email."
No, npm has scopes for a reason, why would that not fix this issue?
Isn't it considered best practice to be secure by default? Wasn't that big fiasco with MongoDB? Why should PyPI, RubyGems, or npm be any different? I'm sure there is some reason but I'd expect them to all pull private repos before public.
Maybe the bug wasn't explained correctly but if it prefers public over private that seems like a bug.
OTOH, it certainly is an issue that if you forget and happen to test some code without being configured to have the private package server as your default then you'd get public repos.
Maybe instead of named packages companies should be using private URLs for packages. That way you always get what you ask for?
npm does not have any 'private package' functionality at all, instead you point it at a different registry server (using eg. Verdaccio or Artifactory) which then serves local packages and proxies public packages if they don't exist locally - or at least that's what they're supposed to do.
Artifactory apparently didn't, and served up whichever was the highest version of public vs. private. Which is stupid.
But the bottom line is that when using npm, the exact package selection policy is determined by whatever registry implementation you're talking to, and so it's the registry implementation which should prioritize private packages by default.
This is exactly what Verdaccio does, and has been doing since forever. It frankly kind of boggles my mind that other private registry implementations don't.
Just "prioritizing" doesn't fix it, you have to limit scoped packages to be provided by a single (trusted, internal) repo. Otherwise, what do you do when internal offers v1.2.3 but external says it has v1.99.99?
Probably, it's more fun to play with syscall filtering in containers or with fuzzers than to review side-channels or educating coworkers. Therefore, security theater.
I'm flabbergasted by how silly this is. Bump the version and the package manager chooses yours online vs. the private one. Amazing. How silly and how expensive is this going to be as this blatant security issue is going ripple on for the next months to come.
This is why explicit pins are a good idea. Whenever you finish a project you should set the explicit versions in the lock and then tag it. The problem is with dependencies of your dependencies, but if they are public, then by their nature they won’t be using private packages that can be hijacked.
Even public packages have been hijacked. Pin all your dependencies (transitive included) and then use automation (e.g. dependabot) to update the pinned versions as needed.
I don't think it actually works this way for NPM specifically, if you're using scoped packages correctly. I believe you can associate a scope with one (private) repo and it will not fall back on the public repo, or choose newer / higher-numbered versions on the public repo over a version from the private one.
Apt supports TLS via apt-transport-https (as you are probably already aware) but I don't think it's default in either Debian nor (X)Ubuntu derivatives. I'd like to know why TBH.
The packages themselves are signed though, so I guess the risk is now on server authenticity as opposed to package integrity.
To mitigate this kind of supply chain attacks for python, we have created following tool [1], that will check python packages on Artifactory instance you specify and create packages with the same name on the PyPi.
Specifying `index-url` is not a solution since `pip` will always choose a package with higher version regardless of the repository. Moreover, in case of same version, it will prioritize PyPI. This was discussed in following issues [0], [1].
Yes, if you have packages on the artifactory the `index-url` is always a way to go. However, if you forget to specify `no-index`, you might not get what you wanted, see [1] for how packages are found. And it's easy to make such mistake when using local resources (you forget to set proxy or internal DNS, new developer is not familiar with the setup and does plain `pip install`, internal server is temporarily unreachable).
>It just pollutes PyPi and a nuisance to others.
I agree, but so are the packages that are no longer maintained. You also reserve pakcage name if you decide to opensource it. Furthermore, by creating package you are leaking metadata about your organization, i.e. some functionality can be inferred from package names.
And sure you can train and try to enforce security awareness, but your people need to be right 100% of the time, while attackers need them to make only one mistake. Similar with namesquatting of the popular packages.
The thing that just happened is like a catastrophic chain-reaction collision in space. Now we will have to use guids for everything. Nothing has meaning.
The article mentions that RubyGems is vulnerable to this, and that Shopify in particular downloaded and ran a gem named "shopify-cloud", but I'm curious as to how this is possible given a "normal" bundler pure-lockfile setup, or more generally the source-block directives I've seen in most Gemfiles.
How would Bundler ever try and download the `appraisal` gem from RubyGems?
The Gemfile section is more explicable. While newer Gemfiles look like this:
source "http://our.own.gem.repo.com/the/path/to/it" do
gem 'gemfromourrepo'
end
# or
gem 'gemfromourrepo', source: "http://our.own.gem.repo.com/the/path/to/it"
Older Gemfiles apparently looked like the following:
Which seems obviously vulnerable to the dependency confusion issue mentioned.
So is the understanding that Shopify's CI systems were running `bundle upgrade` or another non-lockfile operation? (possibly as a greenkeeper-like cron job?) Or is `--pure-lockfile` itself more subtly vulerable?
Someone will eventually update deps, not necessarily CI. But now that devs machine is compromised. The attacker probably only has a small window of time after it gets in, but it should be long enough to exfiltrate dot-files and the source code of whatever it gets included in. Now they have ssh keys (mine are on a yubikey), and the GitHub url. They can further push malicious code into the repo.
I would hope most SSH keys are password-encrypted if not protected by a hardware token like yours, but I agree that the "unscoped-source" Gemfile syntax is a huge vulnerable hole, and a bad one. I'm just confused about how what seems like a pretty uncommon operation led to such an immediate response and code execution from Shopify.
(I also don't think it's true that the attacker has a "small window of time"—as soon as they get a single RCE, it's over, if they're running on a normal dev machine then they can daemonize into the background, add persistence, and snoop events over time. CI systems are obviously less vulnerable to this by nature.)
Note that at least on Linux, deploying a keylogger to exfiltrate the password for a SSH key is not hard either. Even if a keylogger is somehow not possible, you can probably still replace key binaries with patched versions or change desktop shortcuts to launch modified programs.
It’s a small window because it’s going to take about 20-30 mins for the dev to figure out why tests failed, locate the bogus dependency and shut down their computer, notify secops, revoke keys (if they even think of that), etc.
If you know your computer was compromised. Shut down and reinstall from a backup, you don’t try and clean it.
Edit: I’m assuming the attacker would be replacing a dependency with an empty repo since they don’t know the actual source code. If they know the interface the dependency is supposed to provide, it could spread across the entire organization before anyone noticed.
> I would hope most SSH keys are password-encrypted
TBH, these are probably weak passwords for convenience.
I don't think this is correct—I read the earlier section of the docs ("Block Form Of Source, Git, Path, Group And Platforms") as saying that the block form is equivalent to the "source explicitly attached to the gem", the first priority item in your link
However, this section is concerning:
> The presence of a source block in a Gemfile also makes that source available as a possible global source for any other gems which do not specify explicit sources. Thus, when defining source blocks, it is recommended that you also ensure all other gems in the Gemfile are using explicit sources, either via source blocks or :source directives on individual gems.
Yikes! This is yet another easy footgun for people to reintroduce this issue
I know that node has `package-lock.json` and `yarn.lock`, which include integrity checks. Are these checks decorative only? How could npm have been affected by this issue?
IIRC you need to use npm ci to ensure that package-lock.json is used. That said, when developing locally you are going to use npm install or npm update and update the package.json and package-lock.json files accordingly. I could be entirely off target here since I'm writing purely from memory. But there seems to be a few different ways one could trigger a pull from the malicious repo and end up with it inside the package-lock.json file
I feel stupid for using npm for so long without knowing about this command. In hindsight it's obvious that install was updating the lock file each time, so why should it have been any different on the ci server?
No that's not correct, in fact this comment and the two sibling comments are both wrong.
Quouting from NPMs documentation[0] for npm install
> This command installs a package, and any packages that it depends on. If the package has a package-lock or shrinkwrap file, the installation of dependencies will be driven by that, with an npm-shrinkwrap.json taking precedence if both files exist. See package-lock.json and npm
Consider an example where in package.json you have `"react": "^16.11"` and this has been resolved and fixed in package-lock.json as 16.12 at a previous point in time. Running npm install will not cause NPM to install 16.14 even though it matches the pattern specified in package.json, instead 16.12 will be installed because that's what package-lock.json says.
What npm install does do, is detect if you've made changes in package.json and only then does it re-resolve the dependency. In the above example, if you changed `"react": "^16.11"` to `"react": "^16.14"` in package.json and then ran npm install the package-lock.json would be changed and version 16.14 would be installed.
The final build would probably have failed (on a build server using CI). But when developing locally I think package.json wins over the lock file (? at least often the lockfile is updated after doing an npm install here).
So this probably wouldn't show up on the final build distributed and deployed somewhere. But it did manage to run arbitrary code on developers' machines of those companies.
You can easily misconfigure npm by using the `npm i`/`npm install` command on CI/CD instead of `npm ci`. `npm install` does not take any lockfiles into account and only uses the package.json and upgrades any package/dependency that is not pinned.
The real solution is to design and build software components that can be finished, so they can be ruthlessly vetted - rather than the endless churn of updates.
Not sure why parent is being down-voted as I believe this is an important point. In my opinion this would be applying the unix philosophy of having small tools that does one thing and does it well to code libraries.
Because as long as the underlying hardware and technology overall keeps progressing there isn't much practicality in "finishing" software.
Sure you could just "finish" Linux at 5.0 and then introduce e.g. io_uring via Linux-with-io_uring 1.0 instead of adding it to Linux 5.1. Same goes for all the libraries that add support for io_uring.
Yes, you could "finish" some software on the feature level, but you would still need to maintain it if you want to add support for new platforms, etc., or it will become obsolete sooner or later. In the case of still maintaining libraries, this would solve nothing in the context of this attack vector.
But this is exactly the philosophy in the NPM ecosystem where things like left-pad are rife. And NPM is generally considered a dumpster fire precisely because you need 8,000 deps for relatively "simple" projects like a basic create-react-app project.
Gamers often complain how they become free QA testers if they buy a game in the first few months after release as most games are full of bugs (hi Bethesda!) but it is way worse in things like JavaScript libraries etc. It's as if finished have become a foreign word to most developers. Look at the resent story about Linux stable kernels that have had more than 255 minor releases and think how much of a shit show it would have been if they added features too like most developers do. The excellent small stable tools of Unix should have taught us something.
> Look at the resent story about Linux stable kernels that have had more than 255 minor releases and think how much of a shit show it would have been if they added features too like most developers do.
At least some distribution kernels do new feature backports - mostly to support new hardware on LTS versions (like e.g. Ubuntu did for the Raspberry Pi, see https://github.com/raspberrypi/linux/issues/3464).
As you see with chrome extensions and android barcode apps, this is not a solution. Developers change or for whatever reasons can change their mind and ship bad things.
How does it work in practice, though? For example, create-react-app in NPM has a bajillion deps. Do I trust 8,000 keys? Which ones are OK?
I get that you could in principle namespace things (at least for package managers that support this) and insist on a small set of company-internal signing keys for those namespaces. But managing all that isn't easy and what about for package ecosystems that don't really have namespaces (e.g. PyPI, NuGet)?
> After spending an hour on taking down these packages, Ingram stressed that uploading illicit packages on PyPI puts an undue burden on the volunteers who maintain PyPI.
I'm surprised the reverse fully-qualified domain name (FQDN) model used by Java isn't more widely adopted. If you want to upload artifacts to the main repository (Maven Central) you first need to show ownership of a particular domain. For example, via a DNS TXT record (example [1]). Would make these kind of attacks a lot more difficult.
"Years of maturity" or, just thinking about the problem for a bit.
How long did it take npm to have scoped packages. Sure, let me create a "paypal" project, they only need one js project no?
If Java suffers from excessive bureaucracy, the newer package developers/repos suffer from too much eagerness to ship something without thinking
Not to mention dependency and version craziness. If you want your software to be repeatable you need to be specific with the versions and code you're taking.
It drives me crazy that "official" sounding package names like yaml are seemingly given basically first-come first serve, with no oversight. Publish anything you want, but call it Mark's awesome yaml library, or companyName-yaml or something like that so that people are aware that's not an officially supported project
> What would you imagine that oversight looking like, who decides who gets the name `yaml`, and how do they verify it, and who pays for that time?
Just use name spaces. foo.com/yaml instead of yaml. NPM way of doing things is/was just insane, with no regard for trust or security. No wonder NPM corp then went into the business of selling namespaces, AKA selling trust...
Unless it's a part of the standard library included with the language, nobody gets it. There has to be some designation before the name. It's not only node, python also does things like that
I don't understand how a designation in front of the name solves anything. The designation is basically just a name itself, you've just made it a two-part name, and a requirement that all names have two parts. ok, so?
As someone who worked with java for more than a decade before touching the js world, the degree to which npm has been hacked together without any of study prior art is extremely irritating. If you must build something from scratch at least invent some new problems instead of just re-discovering solved ones.
The very existence of package-lock grinds my gears and that's before it starts flip flopping because someone mistook URLs for URIs. Of course that only exists because ranged dependencies are a terrible idea, and that's before anybody even mentions things like namespaces or classifiers.
No maven wasn't perfection, and it could be (and has been) improved on - but npm doesn't even get into spitting distance.
At this point I really wish we'd just go with a proper cryptography model, with a discovery overlay to provide names.
What I want as a developer is to establish my trust relationship to developers of libraries I depend on.
`npm install <somepackage>` should first check a record of signing keys in my source code repo, then check a user-level record of signing keys I've trusted before, and then - and only then - add a tentative trust relationship if this is brand new.
`npm release` or whatever (npm is just an example - every system could benefit from this) - would then actually give me the list of new trust relationships needed, so I can go and do some validation that these are the packages I think they are.
> If only people creating new package managers would bother to spend an hour or two learning prior art.
No, no... We should move fast and break things. We can implement this in a week because the old dinosaurs are too close minded to implement these things.
Today, the hardware is cheap and network is reliable. No need for any safeguards or security features.
Despite being very interested in Rust, this is the first I've heard of crev. It's a very cool project.
That said, it's interesting to me that several people are trying to get the project to drop the Web of Trust, and focus on code reviews. I'm the exact opposite - the code reviews are an interesting, experimental approach, but I'm interested in the project because of the cryptographic Web of Trust. Any use of dev-originated code signing in a package ecosystem is great. For this reason, I'd love for this to get major pickup from Rust, and beyond.
Finally, I am a bit wary because the project is starting to look moribund. It's important for projects like these to know that the maintainer is in it for the long haul, even if there's initially very little adoption. When the project founder writes that they're in a "fight for survival", it makes me think they may abandon the project if it doesn't get significant adoption.
Packages are typically considered immutable once published. If I have a particular package e.g. "FooLib.Bar v 1.2.3" then this zip file should _always_ contain the same bits. If I need to change those bits, e.g. to fix a bug then I need to ship e.g. "FooLib.Bar v 1.2.4"
Also packages aren't always small. So it makes sense to cache a copy locally. On dev machine "package cache" and in an org's "internal feed" and only check upstream if it's not there.
So I shouldn't need to go to the source url to get it. Ideally, I just ask "who has "FooLib.Bar v 1.2.3" for me?"
It also means that tampering can be detected with a hash.
But the "check upstream" model is now vulnerable to fake new versions.
Using a FQDN is less likely to be unknowingly hijacked when it's a domain they control and use daily.
URL references also contain the version number, typically an immutable Git tag reference. They also benefit from just needing to download the source code that's referenced and not the entire package. With Deno you can also optionally host versioned 3rd Party packages on their deno.land/x CDN.
URL references are also cached locally on first use and can be used offline thereafter.
For better or for worse, many projects auto-update their dependencies these days.
They do this to address the shortfalls of modern conventions like small packages, continuous release cycles and dependencies nested several layers deep.
So if you were using the internal package FooLib.Bar v 1.2.3 and an attacker posts FooLib.Bar v 1.2.4 to a global repository, anyone using auto-updating will update to it.
I don't disagree, but both of these (fully or partly automated updates, and attackers) are fairly recent developments to the model.
Of course, mitigation is needed. Supply chain attacks are a hot topic after SolarWinds.
But identifying a package version solely by a url doesn't seem like the right abstraction to me. IMHO, the metadata is more structured: Name (text), version (SemVer) and also maybe now fields to verify and mitigate these attacks: content hash, source feed, etc.
Even if I run an internal feed that transparently proxies and caches the public one, as well as hosting my company's internal artefacts, the rules now might need to be different between packages?
for e.g. between Newtonsoft.Json (new versions always originate on the public feed, never locally) and "SDCo.GeneralUtils" (new versions always originate on the local feed, never upstream)
Using a URL isn't what makes Go's dependency management that good. It's just convenience that the import is a URL.
The key thing with Go is that all dependencies have a checksum (go.sum file) and that should be committed to the repo.
So even if the domain gets hijacked and a malicious package is served up, then the checksum will fail and it will refuse to build.
People should be using internal module proxies anyway for Go. You can just store the module files in a directory, a git repo or a web service and serve up an internal cache.
Javas FQDN model is actually pretty bad in practice. Domain names change quite often (I've seen many packages with a dead FQDN), and relying on the TXT record is going to be a security nightmare even worse than the username/password required by npm (since domains expire).
I believe the TXT record validation is only an additional measure, eg to prevent a random developer from registering/uploading a package like org.apache.http2. Surely other authentication methods are used in practice.
I find it hard to believe any high profile organization would allow their domains to expire, or else they would also lose e-mail and websites, right?
You only need to validate the domain once using a TXT record. And then you use another authentication mechanism such as a username/password combination.
> Javas FQDN model is actually pretty bad in practice
Right, that's why we see this kind of attack all the time on Maven Central, but never on npm... oh, wait?! NO! The kind of simple attacks you see routinely on npm (typo squatting, ownership transfers to malicious authors, now this) just doesn't happen on Maven Central at all.
> Right, that's why we see this kind of attack all the time on Maven Central, but never on npm...
Oh yes, the differences is necessarily explained by Maven design being better, and absolutely not because there are two orders of magnitude difference in usage between these two systems…
There are supply chain attacks in Maven Central too[1], but it's not gonna make the front page of HN…
You may notice that in the linked article, only the artifact id has been spoofed. In maven you need to declare both groupId and artifactId for your dependency (and a fixed version, a range is generally considered a bad practice).
To be noted, it makes this kind of attack more difficult, but not impossibile.
Especially the mix public/private artifacts. I guess it will force a lot of companies to at least lock their groupId on maven central, if they never bothered to do so.
apples and oranges, the name conflict was perfectly disambiguated by the use of the mandatory group identifier.
npm design was so bad that you could at the beginning upload over an existing version of your package name and break dependencies retroactively even to people that pinned versions.
if you want to try some good old whataboutism, at least try to be in the same ballpark.
Nobody in this thread argues that npm is not bad (it is), the current topic is: “is maven's design[1] better” and there is little evidence on this front.
Maven was (yes, I'm using the past on purpose) not a panacea that later system failed to equal: it has the usability of an IRS form and never gained as much popularity in the Java world than npm in the JavaScript one for that reason. In 2014, last time I did Java for work, the main security feature against supply chain attack was: “we are getting .jar files individually and not using maven because it's a fucking mess”
[1]: not implementation, which is what's make npm arguably a pile of shit
Maven is ubiquitous in the Java world and the de-facto package/dependency management system out there. Has been since the mid-2000's and as of 2018 when I last did Java development (Scala really), it is still widely in use. Getting jar files manually would have me running from whatever company that was doing that. Let me guess, they wrote all their code in Notepad because IDE's are a "fucking mess" too right?
You vastly underestimate the level of bureaucracy that can exist in the biggest Java users of this planet (namely banks and public administrations): in these organization (at least a few years ago, the Solarwind attack shows it may not be the case anymore) every single dependency you want to use must be justified, and then is audited by a dedicated team, which ends up handing you the validated .jar.
It was a common development practice in these entities (I was working as a contractor, for different customers), most of them have been using computer programs at there core long before the internet went mainstream.
That bureaucracy makes sense to me. You're _shipping_ those dependencies. When something misbehaves or allows an exploit it doesn't matter if it happened in a dependency or not -- the dependency isn't the one who's accountable to customers and the regulatory authorities.
It sounds like the processes in use to do this may have been pretty crappy in the organizations you've been with, but it also sounds like it would take less time than implementing a dependency's functionality from scratch in most cases you'd want to pull in a dep.
So you worked with customers using some particularly strict vetting protocols. That's a far cry from claiming Maven never reached the popularity in the Java world that NPM has in the Javascript world -- Maven is the dependency manager in the Java world. The entities you worked for are the exception.
Another thing that strikes me as odd in your comparison: those customers you worked with wouldn't have used javascript+NPM either, since it has all of the problems of Maven and external deps and then some! So what exactly are we comparing then?
This "bureacucracy" is very necessary if security is at all a concern. Solarwinds is hot to talk about right now but it has always been the case that having a build download code willy-nilly is a recipe for getting attacked.
In any security conscious organization the only way to pull dependencies is from a local trusted repository. And the only way they get placed there is through a review process.
Maven is/was huge in the Java world. For years it was pretty much the only way to resolve dependencies, until people got fed up with its many idiosyncrasies.
> “we are getting .jar files individually and not using maven because it's a fucking mess”
That seems odd and a bizarre edge case. Nobody worked like that with Java projects, and I bet nobody does today either.
Another example: SBT was supposed to be the "savior" in the Scala world and... I still have nightmares about SBT from a couple of years ago. Maybe it's finally become usable and intelligible in recent years.
I am confusing what now? I'm the one arguing Maven is huge and that the parent post I'm replying to is mistaken. I never mentioned Gradle, that was a sibling comment.
I might have misinterpreted you the first time around. I thought you were saying that people have moved on from Maven, when in fact Maven is still the defacto repository for open source Java projects. Sorry if I did.
Sounds like you had some poor experiences with people who didn't know what they were doing.
The proper way to audit your dependencies is to run an in-house Maven repository server. Just like you would for npm, or any package repository really.
So you just spin up Sonatype Nexus, proxy the repositories you trust and disallow releases from being overwritten. That way you're certain the jar you're using today, is the exact same as the one you'll be using years from now.
Alternatively, if you have a vendor who ships their jars manually, you can just vet them, then upload them to your in-house Maven repository and have your developers pull them in as usual.
We do this. I had to work on a greenfield project and it used a ton of libs that weren’t in our repo. It was so annoying to have a list of repos to add to the in-house list, then discover things didn’t work, so now we need these. It literally added weeks of man-hours to the project, per day.
> It literally added weeks of man-hours to the project, per day.
Can you explain what exactly you mean here, because the way I read it realistic time estimates for the projects grew by weeks pr day which probably means
- I misread
- you wrote something you didn't mean
Anyways it sounds like something was way off and I have worked on some projects with Maven and other systems.
Due to the amount of bureaucracy and vetting required, each package needed four people to touch the task, it took about an hour each person (four man-hours). Then you throw in their dependencies and it grew to about 50 man-hours per top-level dependency.
I agree on "not maven fault" but I don't find that much bureaucracy insane, for one changing dependencies on a mature java project doesn't happen that often, and for another knowing licensing, possible patent violation and a scan against a known vulnerabilities database is not a bad thing to do and it's normal for it to take some time as it passes hand between different people, after all you don't want devs working on licensing and you don't want to waste legals just to run a package trough vuln scans software.
beside, companies that care usually also have a database of previously cleared packages, so one can reduce one own work/delays by picking from the approved deps list.
50 hours pr top level dependency not once or twice or ten times but every time does however sound like something "cache-like" is missing on the human level?
true I was reasoning more on the 4 hour per package, which is in line. but you're right for sure the unique dependency graph can't be averaging that high
Yes, vetting everything takes a LOT of time. Regardless of the language, package manager, etc.
We (small shop) don't vet the code of all of our dependencies, since we simply don't have the manpower. But we do run nexus to have the guarantee of version pinning. So something that is fine today will be fine in 5 years.
> it has the usability of an IRS form and never gained as much popularity in the Java world than npm in the JavaScript one for that reason.
I thought IRS forms were hard?
I have more than a decade of experience collected from work with CDs (Delphi), downloaded libraries (Delphi, Java, PHP), Ant (Java), Maven (Java), PEAR (PHP), Composer (PHP), Nuget (.Net), NPM/Yarn (Javascript/TypeScript) and Gradle (Java/Kotlin).
Two of these have been somewhat easy to work with for as long as I used them: Maven and Yarn. I hear NPM us usable now, but it absolutely wasn't good early on.
> “we are getting .jar files individually and not using maven because it's a [...] mess”
It seems obvious from your writing that you either worked in a place that was really serious about security or had no clue. Both could result in this conclusion, but based on your writing my bet is on the latter, i.e. they were clueless.
I have concluded that it was a mix of those two: the people you worked for were trying really hard to be really serious about security and failing to automate it.
That issue isn't even remotely similar, it's just someone uploading new packages and some people choose to use those instead of the official ones, god knows why. It didn't get pulled in automatically for existing projects.
Also, it's cute how you think maven is used orders of magnitude less.
Sure, the build systems probably won't CONSTANTLY be redownloading all the modules like NPM does, instead they keep a cache, but come on.
log5j.online is on sale for $5 / month. What's the expected ongoing cost of a package? If it's $0, then it's literally infinite. At $0.01 / month it's merely 500x more expensive. The real cost is somewhere in-bewteen.
Normally when people squat they squat like 20+ variations of the name. So that would start to add up to hundred of dollars.
Also, having the domain doesn't make it available on Maven Central. You need to apply to have your domain become a registered groupID on it. This is a manual review process. They validate your domain through TXT verification to make sure the requester to create the group is the domain owner. Then they look to make sure the library is packaged to it. And finally there's a check that the groupID isn't too similar to any existing ones in name, especially to popular ones.
This generally takes 3 to 7 days to get approved.
Once you have a groupID you can release many libraries under it, you don't have to go through that process again.
Now from the user side, things are simpler too, because every lib has a groupID ownership and the lib name. Similar to how on GitHub you have owner/repo.
So it's much easier for me as a user not to confuse org.apache/Log4J with org.malware/Log4J
And like I said, even if someone owned the domain apoche.org they most likely wouldn't get approved to register org.apoche on Maven, because the name is too similar.
It still isn't fool proof admittedly. But it seems much harder to manipulate. And especially if you're a careful user, much easier to trust the source. As long as you got the groupID correct, it's signed and validated. And you can be sure that what you found on apache.org is going to be org.apache on Maven.
Finally, even if the domain changes hands, it doesn't matter. You won't be given access to the Maven repo. Access is given to the Maven user account who registered the group. All you need is to own the domain when you create your groupID. Now if someone transfers their Maven user/pass to a malicious users or become malicious themselves you're still at risk.
Also, I believe there is an appeal, again manually reviewed, like in case you believe your account was stolen, where if you can prove that you own the source repo and/or domain and all they might reinstate you.
But also artifacts are signed, so if your account gets stolen, the thief would need to steal your signature too so it can publish malicious artifacts to the Maven repo.
You have a username/password to Maven Central and you also have a private key to it.
But in order to be granted a groupID (think of it as an account), you need to prove at the time of account creation that you own the domain that matches the groupID (think account name).
So if you try to register com.foo on Maven Central, at that time you need to own foo.com, otherwise you'll be rejected.
If you do own it at that time, well your account is approved and now you have a username/password to it and a private key you need to use to sign artifacts when you publish them.
If your domain expires and is later bought by someone else, that doesn't make them the new owner of your Maven Central groupID.
Except this wasn't a problem with npm but rather with private registry implementations, and a setup with npm + Verdaccio is apparently actually one of the few configurations that isn't vulnerable to this problem.
Not that I didn't expect someone to immediately take the opportunity to complain about npm, of course, despite it having nothing to do with the problem at hand... as has become tradition in tech circles.
This post seems like a good time to note that by default, there's no direct way to verify that what you are downloading from dockerhub is the exact same thing that exists on dockerhub [1].
Discovered after seeing a comment on HN about a bill of materials for software, i.e., a list of "approved hashes" to ensure one can audit exactly what software is being installed, which in turn led me to this issue.
I remember when we used to sign binaries and packages and nobody checked the pgp files anyways. We could have something similar better today, just need to be automated enough.
I think image signing support (or at least was) is not as good as it can be. It would be nice if more images were signed by publishers and verification performed by default.
Even then, that only gives you a stronger indication that the image hasn't been altered since it was signed by the image author at any point after it being signed. However it is not a guarantee that the source produced the binary content. It's also not a guarantee that the image author knew what they were signing - though this is a different issue.
Debian has a reproducible builds initiative[1] so people can compile packages themselves and them match byte for byte what Debian built. Not sure how far they've got with that.
Approximately 25,000 of just over 30,000 source packages are now reproducible builds - generating over 80,000 binary packages. See the graphic on the page you linked to:
You can enable client enforcement of Docker Content Trust [1] so that all images pulled via tag must be signed. Whether people are actually signing their images is a different question that I don't know the answer to.
I'll try to answer this from a JS-specific perspective. As someone previously mentioned - you do get hash checks if you're using `npm ci` in your CI/CD setup. You get the resolution path as well. Which is all you need to reproducibly resolve dependencies, *if* you have set up npm correctly in your pipeline. It would be unlikely to be exposed to this particular attack, at least not automatically in your deployment pipelines.
However this is still very, very dangerous, because of day-to-day engineering, really. Any engineer doing a simple `npm install` can inadvertently bring in and execute malicious code from their machine. From there on out it would be somewhat trivial to gain further access to the same network the code war run from.
I never understood why these package repositories don't include some (opt-in?) integrity checking option using digital signatures. If I download code that executes on my machine there should be at least the option to establish some level of trust. We have been doing that with linux distro package managers for decades. Seems like common sense to me.
They largely do in various forms. Both npm and yarn, by default, record hashes of the dependencies you're using and check them when redownloading.
I think the issue tends to be more that there's just so many packages (often nested 10+ deep) and it's best practice to keep them as up to date as possible.
When it's fairly typical for a JS project to have thousands of dependencies, there isn't really any practical way to both stay up to date and carefully review everything you pull in.
I think the only viable solution for companies taking this issue really seriously is to keep their numbers of dependencies down and avoid having significant deep/indirect dependencies.
Edit: as an example, in my company's Node stack (for 10 services) - there's >900 dependencies. In our React stack (for 2 sites), more than 1600.
Contrary to what you might think, these are actually pretty small, lightweight systems. So really whatever you might have thought was the worst-case scenario on numbers of deps, the reality is more like 10x that in the modern JS ecosystem.
In many ways, the vast number of tiny dependencies are one of the strongest points of the JS ecosystem. But it doesn't come without caveats.
I was thinking the same thing. Surely PayPal's packages should be signed by a certificate only PayPal has, and they would want to verify that before using their packages?
If the signature reqt is attached to the package metadata, the new package just removed it. If it's part of their custom build system, what signs third party packages? Would it just sign the new one anyway, because how does it know which ones should have PayPal internal signing? Or are you proposing manual controls? shudder
Channels and priorities embedded in the package tools are a better approach, combined with something like Artifactory. Some channels might require packages are signed, and possibly monotonically versioned.
I've found that it is easy for discipline to slide on manual controls. It starts off rigourous and tails off into being done infrequently (which makes it a big job) and perfunctorily. This will save you from things that hit the bleeding edge, the idiots who pull from latest on a prod instance, but leaves you exposed to the patched bugs, with increasingly good exploits.
Diff inspection will catch some obviously bad things, but it will rarely catch anything clever. So it would be down to luck, if you had merged in this patch before it was spotted/announced. Unless you have something to separate the namespaces? Check for conflicts? I guess CI might work, hoping your CI machines are sandboxed.
The package integrity would be fine in this case. The packages downloaded from PyPI would be legitimately signed by PyPI, and the internal packages would be signed by the local package server. The issue is not knowing which source to use for each package, and you'd have the same issue with not knowing which certificate to use to check them.
It surprises me a bit the way they refer to in-house dependencies purely by version number. When we have internal dependencies in e.g. package.json, it's always referred to by an explicit url and git ref.
411 comments
[ 5.3 ms ] story [ 328 ms ] threadGitHub is terrible.
I don’t know how many GitHub orgs the Linux Foundation has, but... hundreds? Having one bit account with wide permissions is a non starter
https://docs.github.com/en/developers/apps/scopes-for-oauth-...
https://docs.github.com/en/rest/reference/permissions-requir...
[1] https://news.ycombinator.com/item?id=26087017
[2] https://news.ycombinator.com/item?id=26086789
It seems dumb that they don’t have per repo tokens. I think the issue is with their licensing as if they made proper tokens users could abuse it by giving tokens to their friends. But this should be detectable in a friendly (please don’t do that) way.
I want to be able to give read-only access to private repos.
I want to be able to give fine grained function level and repo level access.
If I’m an admin on multiple repos, I want to be able to issue a token for just a single repo so I can give that to a CI job without worrying if every single repo I admin is at risk.
They allow ssh keys with some similar functionality, but ssh keys can’t be used as much as tokens.
I’ve been waiting for a story about how some third party app granted access to my whole org gets taken over and wreaks havoc. Eventually this will probably be the attack that alters real packages instead of these name overloading packages.
Technically you can create one new GitHub account per repo and generate a token for that... But that is highly annoying :)
They need to support IAM / RBAC style policies and tie every authn+z method to those policies, but my guess is they have different auth methods strung all throughout their codebase so implementing it will take a few years. Then of course they have to make it "user friendly" as we all know how painful IAM can be...
If I'm not mistaken insider knowledge wasn't necessary.
Remember how much was temporarily broken in the leftPad event? Imagine if all that had been silently back-doored instead?
Free software / open source propels engineering as you can share and leverage the results of collective efforts. However, at no point did the concept come with inherent guarantees about concerns such as security.
esr defined 19 points for "good" open source software development in his seminal essay "The Cathedral and the Bazaar". I feel some of those are sometimes easily thrown out of the window for the sake of "efficiency" or "cost-effectiveness".
This issue resonates with bullet point 17 in particular:
> A security system is only as secure as its secret. Beware of pseudo-secrets.
I think this issue has less to do with package managers, and a lot with companies rushing into the convenience of public code platforms such as Github without properly vetting whether or not they might be inadvertently leaking internal information through packaging manifests.
https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar
[1] http://esr.ibiblio.org/
[0]: https://lists.nongnu.org/archive/html/gpsd-dev/2021-01/msg00...
Scroll down to contribution activity. Forked a repo, created a pull request, etc, on Feb 1st 2021.
That is not true at all, the industry both Development and even more so in Operations has been outsourcing responsibility for a long time, they is why we have support contracts, SLA's and other very expensive services we pay many many times more than the cost of hardware for...
To outsource responsibility... Network down -- Call Cisco... Storage Down Call EMC or Nimble... etc
A support contract allows you to hold a sub-contractor accountable. But that's the extent of what an SLA does. What it doesn't do is diminish your responsibility towards anyone who relies on the services you provide yourself. These are distinct things.
Put more succinctly, if the network's down: that's still very much your problem. Especially if you over-promised 100% availability to the end users of your services. Your end users do not care about Cisco, EMC or Nimble. They don't have contracts with any of those. They have a contract with you and they can and will hold you accountable if you don't deliver on what you sold them.
for a sysadmin the customer is "the employer" and they do not really have a contract with the sysadmin, rather the employer has contract with Cisco, or Nimble, etc. the sysadmin has "outsourced" his/her responsibility in that context.
For example instead of rolling your own storage device using linux, or freenas or something else, you buy an expensive 3rd party solution with expensive support contracts to outsource their responsibility. If it goes down "I have a support ticket open with vendor" instead of "I am attempting to merge the lastest kernel patch I have downloaded from github"
That is the source of the phrase "No one ever got fired for buying Cisco" or insert name of other large vendor. They do not get fired for it because they have outsourced their responsibility
An employee/employer relationship is defined by a few key properties. As an employee, you sell your time and your expertise to your employer, and you agree to submit to the authority of your employer in exchange for a salary. The extent of your responsibility - and this is absolutely key - is captured in your contract.
It also means that many things simply aren't your responsibility to begin with, even though you deal with them on a day-to-day basis.
As a systems administrator you, quite likely, won't get fired for failing Cisco gear or services because you're not the one who ultimately signs off on the contract with Cisco on behalf of your employer. Responsibility has always resided with the executives who cover and sanction your actions and decisions.
An executive, though, usually won't get fired over failing Cisco gear/services itself, but they will get fired over any ensuing loss of revenue, damage to brand/image, litigation over exposed liabilities,...
A great example of this is President Harry S. Truman who famously had a sign on his desk stating "The buck stops here".
https://en.wikipedia.org/wiki/Buck_passing
As for the systems administrator, your role is to actively engage in day-to-day operations. You're basically hired "to keep the lights on". Whether the proverbial "light" was procured from Cisco or handcrafted in-house is inconsequential to your employer as far as your individual role as an employee is concerned.
what could possibly go wrong?
Random libraries, possibly pulled in by a dependency of a dependency of a dependency... not so much.
They offer PGP signatures too!
curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin <additional install-script args>
See https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-in...
There are other examples I've seen from time to time.
Anyway. I’m sticking with Pop / Linux. But it does make me nervous!
I'd guess distros are generally better off in that respect, but kernel space & user space aren't that different nowadays, when caring about your own security
I'm very happy to finally have a real world example to motivate all the folks that eye-rolled me every time I've raised it in the past. It just resonates better, especially with less technical leadership folks.
Nix package managers/repositories have a level of scrutiny to get into, and highly dedicated people in charge of. Random github repos (or npm packages) are extremely low effort/risk to set up.
Of course the former can be abused, but the incentives are at least in its favor to likely be more trustworthy. And we have to make assumptions of trust everytime we sit down in our chair or turn on our computer, or plug in a space heater. We will never get around trust, but there are differences in levels of trust and trustworthiness.
That's not really true though. Nix doesn't support signed sources, there are no signatures in the package repository and in theory "John Doe" with no information can add packages and send pull-requests.
In practise nixpks is just a well moderated user repository and the level of scrutiny is less then the enterprise distros can offer.
https://discourse.nixos.org/t/trust-model-for-nixpkgs/9450
https://github.com/NixOS/nixpkgs/issues/20836
has if the nix peeps are reading the code nix is wget'ing
Your code is what it depends on.
""That said, we consider the root cause of this issue to be a design flaw (rather than a bug) in package managers that can be addressed only through reconfiguration," a Microsoft spokesperson said in the email."
No, npm has scopes for a reason, why would that not fix this issue?
Maybe the bug wasn't explained correctly but if it prefers public over private that seems like a bug.
OTOH, it certainly is an issue that if you forget and happen to test some code without being configured to have the private package server as your default then you'd get public repos.
Maybe instead of named packages companies should be using private URLs for packages. That way you always get what you ask for?
Artifactory apparently didn't, and served up whichever was the highest version of public vs. private. Which is stupid.
But the bottom line is that when using npm, the exact package selection policy is determined by whatever registry implementation you're talking to, and so it's the registry implementation which should prioritize private packages by default.
But that's just NPM, it's an issue in all of the mentioned package managers.
The packages themselves are signed though, so I guess the risk is now on server authenticity as opposed to package integrity.
Here is an explanation of the process under 'The current scheme for package signature checks' at the following url
https://www.debian.org/doc/manuals/securing-debian-manual/de....
[1] https://github.com/pan-net-security/artifactory-pypi-scanner
You have always been able to specify the `index-url` when installing packages using pip. This can also be added to `requirements.txt` files as well.
[0]: https://github.com/pypa/pip/issues/5045
[1]: https://github.com/pypa/pip/issues/8606
>It just pollutes PyPi and a nuisance to others. I agree, but so are the packages that are no longer maintained. You also reserve pakcage name if you decide to opensource it. Furthermore, by creating package you are leaking metadata about your organization, i.e. some functionality can be inferred from package names.
And sure you can train and try to enforce security awareness, but your people need to be right 100% of the time, while attackers need them to make only one mistake. Similar with namesquatting of the popular packages.
https://pip.pypa.io/en/stable/reference/pip_install/#finding...
That is, given a Gemfile.lock like, e.g.
How would Bundler ever try and download the `appraisal` gem from RubyGems?The Gemfile section is more explicable. While newer Gemfiles look like this:
Older Gemfiles apparently looked like the following: Which seems obviously vulnerable to the dependency confusion issue mentioned.So is the understanding that Shopify's CI systems were running `bundle upgrade` or another non-lockfile operation? (possibly as a greenkeeper-like cron job?) Or is `--pure-lockfile` itself more subtly vulerable?
(I also don't think it's true that the attacker has a "small window of time"—as soon as they get a single RCE, it's over, if they're running on a normal dev machine then they can daemonize into the background, add persistence, and snoop events over time. CI systems are obviously less vulnerable to this by nature.)
If you know your computer was compromised. Shut down and reinstall from a backup, you don’t try and clean it.
Edit: I’m assuming the attacker would be replacing a dependency with an empty repo since they don’t know the actual source code. If they know the interface the dependency is supposed to provide, it could spread across the entire organization before anyone noticed.
> I would hope most SSH keys are password-encrypted
TBH, these are probably weak passwords for convenience.
However, this section is concerning:
> The presence of a source block in a Gemfile also makes that source available as a possible global source for any other gems which do not specify explicit sources. Thus, when defining source blocks, it is recommended that you also ensure all other gems in the Gemfile are using explicit sources, either via source blocks or :source directives on individual gems.
Yikes! This is yet another easy footgun for people to reintroduce this issue
Quouting from NPMs documentation[0] for npm install
> This command installs a package, and any packages that it depends on. If the package has a package-lock or shrinkwrap file, the installation of dependencies will be driven by that, with an npm-shrinkwrap.json taking precedence if both files exist. See package-lock.json and npm
Consider an example where in package.json you have `"react": "^16.11"` and this has been resolved and fixed in package-lock.json as 16.12 at a previous point in time. Running npm install will not cause NPM to install 16.14 even though it matches the pattern specified in package.json, instead 16.12 will be installed because that's what package-lock.json says.
What npm install does do, is detect if you've made changes in package.json and only then does it re-resolve the dependency. In the above example, if you changed `"react": "^16.11"` to `"react": "^16.14"` in package.json and then ran npm install the package-lock.json would be changed and version 16.14 would be installed.
Bundler and CocoaPods also work this way.
So this probably wouldn't show up on the final build distributed and deployed somewhere. But it did manage to run arbitrary code on developers' machines of those companies.
Sure you could just "finish" Linux at 5.0 and then introduce e.g. io_uring via Linux-with-io_uring 1.0 instead of adding it to Linux 5.1. Same goes for all the libraries that add support for io_uring.
Yes, you could "finish" some software on the feature level, but you would still need to maintain it if you want to add support for new platforms, etc., or it will become obsolete sooner or later. In the case of still maintaining libraries, this would solve nothing in the context of this attack vector.
At least some distribution kernels do new feature backports - mostly to support new hardware on LTS versions (like e.g. Ubuntu did for the Raspberry Pi, see https://github.com/raspberrypi/linux/issues/3464).
This is not as amendable to CI, but that's the point.
I get that you could in principle namespace things (at least for package managers that support this) and insist on a small set of company-internal signing keys for those namespaces. But managing all that isn't easy and what about for package ecosystems that don't really have namespaces (e.g. PyPI, NuGet)?
You can at least trust more 8000 developers whose keys are centrally signed, than 8000 packages thrown into signing CI tooling by who knows.
Fixed versions for as many things as you can (including OS images, apt packages, Docker images, etc) lead to changes in your CI under your control.
Sure, you have to upgrade manually or by a script. But isn't plain build stability worth it? Not even talking about security.
So you wouldn't get a random version even considered.
Version shadowing and overriding is a totally different concern of course.
I dunno, feels like fair game to me
[1] https://issues.sonatype.org/browse/OSSRH-61509
How long did it take npm to have scoped packages. Sure, let me create a "paypal" project, they only need one js project no?
If Java suffers from excessive bureaucracy, the newer package developers/repos suffer from too much eagerness to ship something without thinking
Not to mention dependency and version craziness. If you want your software to be repeatable you need to be specific with the versions and code you're taking.
Just use name spaces. foo.com/yaml instead of yaml. NPM way of doing things is/was just insane, with no regard for trust or security. No wonder NPM corp then went into the business of selling namespaces, AKA selling trust...
The very existence of package-lock grinds my gears and that's before it starts flip flopping because someone mistook URLs for URIs. Of course that only exists because ranged dependencies are a terrible idea, and that's before anybody even mentions things like namespaces or classifiers.
No maven wasn't perfection, and it could be (and has been) improved on - but npm doesn't even get into spitting distance.
What I want as a developer is to establish my trust relationship to developers of libraries I depend on.
`npm install <somepackage>` should first check a record of signing keys in my source code repo, then check a user-level record of signing keys I've trusted before, and then - and only then - add a tentative trust relationship if this is brand new.
`npm release` or whatever (npm is just an example - every system could benefit from this) - would then actually give me the list of new trust relationships needed, so I can go and do some validation that these are the packages I think they are.
https://central.sonatype.org/pages/requirements.html#sign-fi...
If only people creating new package managers would bother to spend an hour or two learning prior art.
With npm, you can only add "trust" to npm itself LOL: https://docs.npmjs.com/about-pgp-signatures-for-packages-in-...
What a joke.
No, no... We should move fast and break things. We can implement this in a week because the old dinosaurs are too close minded to implement these things.
Today, the hardware is cheap and network is reliable. No need for any safeguards or security features.
/s
That said, it's interesting to me that several people are trying to get the project to drop the Web of Trust, and focus on code reviews. I'm the exact opposite - the code reviews are an interesting, experimental approach, but I'm interested in the project because of the cryptographic Web of Trust. Any use of dev-originated code signing in a package ecosystem is great. For this reason, I'd love for this to get major pickup from Rust, and beyond.
Finally, I am a bit wary because the project is starting to look moribund. It's important for projects like these to know that the maintainer is in it for the long haul, even if there's initially very little adoption. When the project founder writes that they're in a "fight for survival", it makes me think they may abandon the project if it doesn't get significant adoption.
Packages are typically considered immutable once published. If I have a particular package e.g. "FooLib.Bar v 1.2.3" then this zip file should _always_ contain the same bits. If I need to change those bits, e.g. to fix a bug then I need to ship e.g. "FooLib.Bar v 1.2.4"
Also packages aren't always small. So it makes sense to cache a copy locally. On dev machine "package cache" and in an org's "internal feed" and only check upstream if it's not there.
So I shouldn't need to go to the source url to get it. Ideally, I just ask "who has "FooLib.Bar v 1.2.3" for me?"
It also means that tampering can be detected with a hash.
But the "check upstream" model is now vulnerable to fake new versions.
URL references also contain the version number, typically an immutable Git tag reference. They also benefit from just needing to download the source code that's referenced and not the entire package. With Deno you can also optionally host versioned 3rd Party packages on their deno.land/x CDN.
URL references are also cached locally on first use and can be used offline thereafter.
They do this to address the shortfalls of modern conventions like small packages, continuous release cycles and dependencies nested several layers deep.
So if you were using the internal package FooLib.Bar v 1.2.3 and an attacker posts FooLib.Bar v 1.2.4 to a global repository, anyone using auto-updating will update to it.
Of course, mitigation is needed. Supply chain attacks are a hot topic after SolarWinds.
But identifying a package version solely by a url doesn't seem like the right abstraction to me. IMHO, the metadata is more structured: Name (text), version (SemVer) and also maybe now fields to verify and mitigate these attacks: content hash, source feed, etc.
Even if I run an internal feed that transparently proxies and caches the public one, as well as hosting my company's internal artefacts, the rules now might need to be different between packages?
for e.g. between Newtonsoft.Json (new versions always originate on the public feed, never locally) and "SDCo.GeneralUtils" (new versions always originate on the local feed, never upstream)
The key thing with Go is that all dependencies have a checksum (go.sum file) and that should be committed to the repo.
So even if the domain gets hijacked and a malicious package is served up, then the checksum will fail and it will refuse to build.
People should be using internal module proxies anyway for Go. You can just store the module files in a directory, a git repo or a web service and serve up an internal cache.
I find it hard to believe any high profile organization would allow their domains to expire, or else they would also lose e-mail and websites, right?
Off the top of my head I remember both Microsoft (hotmail.co.uk) and Foursquare forgetting to renew their domains.
Right, that's why we see this kind of attack all the time on Maven Central, but never on npm... oh, wait?! NO! The kind of simple attacks you see routinely on npm (typo squatting, ownership transfers to malicious authors, now this) just doesn't happen on Maven Central at all.
Oh yes, the differences is necessarily explained by Maven design being better, and absolutely not because there are two orders of magnitude difference in usage between these two systems…
There are supply chain attacks in Maven Central too[1], but it's not gonna make the front page of HN…
[1] a quick ddg search, just last month https://securityboulevard.com/2021/01/sonatype-stops-softwar...
To be noted, it makes this kind of attack more difficult, but not impossibile.
Especially the mix public/private artifacts. I guess it will force a lot of companies to at least lock their groupId on maven central, if they never bothered to do so.
npm design was so bad that you could at the beginning upload over an existing version of your package name and break dependencies retroactively even to people that pinned versions.
if you want to try some good old whataboutism, at least try to be in the same ballpark.
Maven was (yes, I'm using the past on purpose) not a panacea that later system failed to equal: it has the usability of an IRS form and never gained as much popularity in the Java world than npm in the JavaScript one for that reason. In 2014, last time I did Java for work, the main security feature against supply chain attack was: “we are getting .jar files individually and not using maven because it's a fucking mess”
[1]: not implementation, which is what's make npm arguably a pile of shit
It was a common development practice in these entities (I was working as a contractor, for different customers), most of them have been using computer programs at there core long before the internet went mainstream.
It sounds like the processes in use to do this may have been pretty crappy in the organizations you've been with, but it also sounds like it would take less time than implementing a dependency's functionality from scratch in most cases you'd want to pull in a dep.
Another thing that strikes me as odd in your comparison: those customers you worked with wouldn't have used javascript+NPM either, since it has all of the problems of Maven and external deps and then some! So what exactly are we comparing then?
In any security conscious organization the only way to pull dependencies is from a local trusted repository. And the only way they get placed there is through a review process.
> “we are getting .jar files individually and not using maven because it's a fucking mess”
That seems odd and a bizarre edge case. Nobody worked like that with Java projects, and I bet nobody does today either.
Many of us still prefer Maven.
Having to deal with other people's artistic Gradle files works wonders when it comes to understanding the goodness of standards.
Another example: SBT was supposed to be the "savior" in the Scala world and... I still have nightmares about SBT from a couple of years ago. Maybe it's finally become usable and intelligible in recent years.
Gradle for example still uses Maven for its dependency artifact repository. So Maven is still the standard for Java.
The proper way to audit your dependencies is to run an in-house Maven repository server. Just like you would for npm, or any package repository really.
So you just spin up Sonatype Nexus, proxy the repositories you trust and disallow releases from being overwritten. That way you're certain the jar you're using today, is the exact same as the one you'll be using years from now.
Alternatively, if you have a vendor who ships their jars manually, you can just vet them, then upload them to your in-house Maven repository and have your developers pull them in as usual.
Can you explain what exactly you mean here, because the way I read it realistic time estimates for the projects grew by weeks pr day which probably means
- I misread
- you wrote something you didn't mean
Anyways it sounds like something was way off and I have worked on some projects with Maven and other systems.
Because otherwise the workload should shrink fast after one has vetted the dependecies for a couple of packages?
beside, companies that care usually also have a database of previously cleared packages, so one can reduce one own work/delays by picking from the approved deps list.
We (small shop) don't vet the code of all of our dependencies, since we simply don't have the manpower. But we do run nexus to have the guarantee of version pinning. So something that is fine today will be fine in 5 years.
I thought IRS forms were hard?
I have more than a decade of experience collected from work with CDs (Delphi), downloaded libraries (Delphi, Java, PHP), Ant (Java), Maven (Java), PEAR (PHP), Composer (PHP), Nuget (.Net), NPM/Yarn (Javascript/TypeScript) and Gradle (Java/Kotlin).
Two of these have been somewhat easy to work with for as long as I used them: Maven and Yarn. I hear NPM us usable now, but it absolutely wasn't good early on.
> “we are getting .jar files individually and not using maven because it's a [...] mess”
It seems obvious from your writing that you either worked in a place that was really serious about security or had no clue. Both could result in this conclusion, but based on your writing my bet is on the latter, i.e. they were clueless.
Edit:
Reading your comment here: https://news.ycombinator.com/item?id=26091480
I have concluded that it was a mix of those two: the people you worked for were trying really hard to be really serious about security and failing to automate it.
Also, it's cute how you think maven is used orders of magnitude less.
Sure, the build systems probably won't CONSTANTLY be redownloading all the modules like NPM does, instead they keep a cache, but come on.
Also, having the domain doesn't make it available on Maven Central. You need to apply to have your domain become a registered groupID on it. This is a manual review process. They validate your domain through TXT verification to make sure the requester to create the group is the domain owner. Then they look to make sure the library is packaged to it. And finally there's a check that the groupID isn't too similar to any existing ones in name, especially to popular ones.
This generally takes 3 to 7 days to get approved.
Once you have a groupID you can release many libraries under it, you don't have to go through that process again.
Now from the user side, things are simpler too, because every lib has a groupID ownership and the lib name. Similar to how on GitHub you have owner/repo.
So it's much easier for me as a user not to confuse org.apache/Log4J with org.malware/Log4J
And like I said, even if someone owned the domain apoche.org they most likely wouldn't get approved to register org.apoche on Maven, because the name is too similar.
It still isn't fool proof admittedly. But it seems much harder to manipulate. And especially if you're a careful user, much easier to trust the source. As long as you got the groupID correct, it's signed and validated. And you can be sure that what you found on apache.org is going to be org.apache on Maven.
Finally, even if the domain changes hands, it doesn't matter. You won't be given access to the Maven repo. Access is given to the Maven user account who registered the group. All you need is to own the domain when you create your groupID. Now if someone transfers their Maven user/pass to a malicious users or become malicious themselves you're still at risk.
Also, I believe there is an appeal, again manually reviewed, like in case you believe your account was stolen, where if you can prove that you own the source repo and/or domain and all they might reinstate you.
But also artifacts are signed, so if your account gets stolen, the thief would need to steal your signature too so it can publish malicious artifacts to the Maven repo.
You have a username/password to Maven Central and you also have a private key to it.
But in order to be granted a groupID (think of it as an account), you need to prove at the time of account creation that you own the domain that matches the groupID (think account name).
So if you try to register com.foo on Maven Central, at that time you need to own foo.com, otherwise you'll be rejected.
If you do own it at that time, well your account is approved and now you have a username/password to it and a private key you need to use to sign artifacts when you publish them.
If your domain expires and is later bought by someone else, that doesn't make them the new owner of your Maven Central groupID.
https://naildrivin5.com/blog/2019/07/10/the-frightening-stat...
https://techbeacon.com/security/check-your-dependencies-gith...
https://thenewstack.io/npm-password-resets-show-developers-n...
Not that I didn't expect someone to immediately take the opportunity to complain about npm, of course, despite it having nothing to do with the problem at hand... as has become tradition in tech circles.
Discovered after seeing a comment on HN about a bill of materials for software, i.e., a list of "approved hashes" to ensure one can audit exactly what software is being installed, which in turn led me to this issue.
[1] - https://github.com/docker/hub-feedback/issues/1925
Even then, that only gives you a stronger indication that the image hasn't been altered since it was signed by the image author at any point after it being signed. However it is not a guarantee that the source produced the binary content. It's also not a guarantee that the image author knew what they were signing - though this is a different issue.
Debian has a reproducible builds initiative[1] so people can compile packages themselves and them match byte for byte what Debian built. Not sure how far they've got with that.
https://wiki.debian.org/ReproducibleBuilds
https://tests.reproducible-builds.org/debian/unstable/amd64/...
[1] - https://docs.docker.com/engine/security/trust/#client-enforc...
Very useful for your own images that you publish in your registry though.
However this is still very, very dangerous, because of day-to-day engineering, really. Any engineer doing a simple `npm install` can inadvertently bring in and execute malicious code from their machine. From there on out it would be somewhat trivial to gain further access to the same network the code war run from.
https://pip.pypa.io/en/stable/reference/pip_hash/
https://pip.pypa.io/en/stable/reference/pip_install/#hash-ch...
I think the issue tends to be more that there's just so many packages (often nested 10+ deep) and it's best practice to keep them as up to date as possible.
When it's fairly typical for a JS project to have thousands of dependencies, there isn't really any practical way to both stay up to date and carefully review everything you pull in.
I think the only viable solution for companies taking this issue really seriously is to keep their numbers of dependencies down and avoid having significant deep/indirect dependencies.
Edit: as an example, in my company's Node stack (for 10 services) - there's >900 dependencies. In our React stack (for 2 sites), more than 1600.
Contrary to what you might think, these are actually pretty small, lightweight systems. So really whatever you might have thought was the worst-case scenario on numbers of deps, the reality is more like 10x that in the modern JS ecosystem.
In many ways, the vast number of tiny dependencies are one of the strongest points of the JS ecosystem. But it doesn't come without caveats.
Channels and priorities embedded in the package tools are a better approach, combined with something like Artifactory. Some channels might require packages are signed, and possibly monotonically versioned.
That's what we do. Dependencies are committed to our repository, so changes show up as diffs which are looked through then.
Then again we don't use libraries for padding strings and such...
Diff inspection will catch some obviously bad things, but it will rarely catch anything clever. So it would be down to luck, if you had merged in this patch before it was spotted/announced. Unless you have something to separate the namespaces? Check for conflicts? I guess CI might work, hoping your CI machines are sandboxed.