I'd love to see a infographic style explanation website of all the different package managers and their tradeoffs, written to a package-maintainer level of detail.
It's such an interesting and important field but largely treated as an annoyance, dismissed by the very people who should be learning about it.
Simply having a table of all the fundamental concepts involved and how each package manager handles them would be really interesting. Or even just a list of the fundamental concepts useful for building your mental model of how they work!
Malicious packages and typosquatting are very different from what's being talked about here. The problem in this case is a clash of policies: NPM historically allowed people to delete their packages after they published them, and then after left-pad they realized that was a terrible idea from an ecosystem reliability perspective so they made it so that you can't unpublish packages that are listed as dependencies of other packages, and so someone took this to the logical conclusion and a made a package that was dependent on all other packages.
NPM wants to have its cake and eat it too, which is the problem here. The solution is just to say that if you publish a package to NPM you give it the perpetual right to distribute it as-is, and then remove the ability for users to delete their packages at-will.
When authors want to remove a package, realistically there are two main reasons: (a) they don’t need it anymore or (b) there’s a serious issue with it in general, like deprecation or unfitness. Why not just (a) hide it in ui or (b) flag it, explain the reason and give everyone enough time to migrate away.
So many issues arise from the fact that our UIs can’t do simple things on lists like folders (e.g. archive) and tags/notes.
> NPM wants to have its cake and eat it too, which is the problem here. The solution is just to say that if you publish a package to NPM you give it the perpetual right to distribute it as-is, and then remove the ability for users to delete their packages at-will.
Yes, exactly!
Kind of unrelated, but I think it is important to remember
that the left-pad was also ENTIRELY npm team's fault.
You can't just take away a namespace from someone
just because some startup like kik comes knocking.
Toyota does not have a right to my domain dot tld slash toyota
The correct answer would have been npm to tell kik to pound sand.
Your linked article does not show that PyPI has been attacked in this way (I don't think this is really an attack, but putting that aside for a second).
The way PyPI does deletions is through "yanking," which is really a soft-delete mechanism. Yanking prevents usage of a particular package version for new resolutions, but allows fully-resolved versions to be downloaded. As a result, PyPI doesn't have the specific behavior that caused this (forbidding people from removing their packages because removal means hard-deletion).
Ironically, this happened because of a mitigation put in place to prevent the leftpad disaster from happening again. To recap: the developer of leftpad removed his package, which broke packages that depended on it. To prevent this from happening again NPM decided that you can't unpublish a package that has dependents, which I think is a sensible policy. The result of this is that you can prevent anyone's package from being unpublished by depending on it. This 'everything' package simply took this concept to its extreme conclusion.
Personally, I think if you're not sure you want a package to be available publicly in perpetuity, you should maybe consider publishing it to a private registry. Even if NPM did allow developers to unpublish any package, there's nothing stopping me from downloading a tarball of your package as soon as you publish it, and hosting a mirror of it somewhere else.
> Even if NPM did allow developers to unpublish any package, there's nothing stopping me from downloading a tarball of your package as soon as you publish it, and hosting a mirror of it somewhere else.
As far as I can tell, NPM does not enforce any limitations on the license of published packages; I would not assume you can legally do this unless you have verified the package has been published under a license which permits such actions and which you are certain you understand.
Weird downvotes! From the source: "Your Content belongs to you. You decide whether and how to license it. But at a minimum, you license npm to provide Your Content to users of npm Services when you share Your Content. That special license allows npm to copy, publish, and analyze Your Content, and to share its analyses with others."[0]
I'm not a lawyer, but in my best judgment I see no reason to assume an NPM package can continue being used after it is taken down -- you need to actually evaluate the license it's published under to determine that. If I'm mistaken, I'd love to see a source establishing the contrary.
Just to be clear: I'm not planning on running an NPM mirror. My point is that if you're not sure you want your code to be publicly available, you should perhaps consider not publishing it to a public registry in the first place. Everything published to the internet can be downloaded and archived. If I publish a blog post I assume it might be cached by a search engine, or snapshotted by archive.today or the wayback machine.
Oh, I agree you shouldn't publish if you aren't sure your code should be public! On the flip side, a package manager could easily address this by requiring code to be published under a limited selection of licenses that actually ensconce those rights, rather than treating it like the wild west as NPM chooses to do.
Sure, that's true, but good luck suing a Chinese NPM mirror or a Russian NPM mirror.
When you upload something to the internet it's there forever. The only way to ensure something is not on the internet is if you don't post it (and even then...). People are destined to learn this lesson over and over again.
It works both ways, if you want something gone, it will remain forever. If you spend hours searching for something and finally finding a post on a forum from 2007 where someone goes "this fixed it for me: <link>", that link will never work and won't be archived by archive.org.
Maybe something like this should put up big siren noises that it has so many dependencies. Can -anything- have that many dependencies? seems like a check they could build in and not let it go into the npm packages without a review.
It's a combination of low barrier to entry and dogmatic adherence to "not reinventing anything" resulting in quantity over quality and massively deep dependency graphs.
So the “prank”, as called in the article, used the same flaw that allowed someone to do it last January. His apology appears deleted along with the repo.
Was there a legitimate purpose or is he just a troll?
What is the correct mitigation for this? Packages can have large dependency trees, so outside of putting an arbitrarily high upper limit, what is the fix?
I think you named at least one reasonable fix: an upper limit.
And it's not really arbitrary. You can count the dependency tree of every package and get the P99, exclude joke packages and get the max of the rest. Set the limit around there with an error message to open a bug if you need more, grandfathering legit old large packages with some kind of admin-level exception.
The number of people actually needing more will be incredibly tiny. Service those folks individually to avoid pain.
It's still arbitrary because P99 is an arbitrary percentage. Why not P97 or P90? Or P99.9? And if you have an arbitrary cutoff but make exceptions for _some_ people who are over that cutoff, you're opening yourself up to two things you don't want: more work for yourself to review the special cases, and more complaining from people. Better to have a rule apply to everyone equally.
The person made thousands of modules that look like `@everything/sub-chunk-{id}`, each depending on N modules. Those sub-chunk modules, together, depend on all NPM modules. Whatever upper limit you come up with, anyone can do this by setting N to your upper limit.
The problem here isn't that you can create modules that depend on other modules. If there's a problem at all, it's that the NPM's yank policy could be improved.
It would affect the everything package, but not the issue since the issue is caused by npm's policy + simply creating thousands packages with, say, 5 dependencies that cover all of the module space.
That they rolled all of those submodules into one big everything package is just a gimmick, not a source of the problem.
> Was there a legitimate purpose or is he just a troll?
Call attention to a real issue that should be solved by the ecosystem so that people cannot do it in the future? In my opinion it is not useful or even sane to expect people to just _not_ do annoying things if you develop systems that allow people to do things that annoy you with them. Design and build systems that assume people will be trolls, and make it so people cannot so easily troll you with them. Failing that, there's nobody to point the finger at besides the person or people who made it possible.
I’m just curious about the intent, as the article implies it was a deliberate prank. That would definitely have professional repercussions. We can laud him for drawing more attention to a major problem, but many (perhaps most?) potential employers may not see it as a positive.
What's your intent by assuming what these hypothetical employers would think? They stated in the article that the intent was to raise awareness, not to troll. People are just assuming they're lying about their intent, but I don't see much point in trying to uncover a grand conspiracy here, fix the problem with npm.
There isn't really any conceivable reason other than for lulz is there, though? That's the whole point. Nobody NEEDS this, so the Node ecosystem needs to fix this in such a way where creating an everything package isn't even possible. The only attention this person deserves is whatever amount of attention is required to get people that work on NPM to work on a fix. I'm sorry for all the people that feel the need to act attacked by this, but frankly if nobody shines a light and causes some pain by being a lawful-evil prankster, the people working on Node will never bother to fix it. As is evidenced by the fact that it is possible.
To be fair, they thought that the idea of an "everything" package was funny and claimed that they only understood the full consequences after experiencing them. Never attribute to malice...
I'll say it again: why does each language need its own package manager? Why not generalize it to delivering files from various online sources, with a thin layer of glue for module loading that is language-specific.
What you're advocating is essentially the way Javascript originally worked, with the web itself serving as the "package manager." but one day someone decided Javascript needed to be a serious business language for serious business, and serious business languages use compilers and toolchains and have a package manager. And so here we are.
You don't have a runtime dependency if you just download the js to your server and write some script tags. You do have to do things manually and locally but you'd be surprised how robust that can be. Just vendoring scripts and not automating builds would solve like 99% of modern JS problems.
The benefit of CDNs used to be that you could cache a library for use on multiple domains but I don't think that works anymore.
What’s in js that makes it different from any other language? Any other language would likely bend into <what you mean by that> form with enough users coming to it from the “web” direction. It’s just a law of big numbers.
Compared to other high-level languages, nothing, really. I actually really like JS for its intended purpose (or just as a way to learn programming).
But I also think it’s been permanently tainted due to a constantly shifting ecosystem that can’t settle down, and a user community that is constantly chasing new shiny things.
JS is a messy language. But I don't think that this is really fair.
Dependency management is a disaster in a ton of ecosystems. It is often a different set of disasters than what npm has, but it isn't like this is a solved problem everywhere else and it is just the js ecosystem that sucks. NPM is made marginally worse by the poor standard library in JS but I don't think the problem is fundamentally attached to the language in any meaningful way.
And the push to use JS for backends was not "we want this to be a serious language" but was instead "well, we are already using JS for frontends so it'd be nice from a practical standpoint for hiring and training to be able to use the same language and ecosystem for backends too."
I don't want anything in my Rust dependency tree depending on node packages.
By being hermetically sealed, the community can make progress on testing, repeatability, and lots of things that would become incredibly difficult if you introduced anything else into the mix.
Moreover, each language gets to innovate in its own way. Cargo is fantastic and fits the needs of Rust quite well. Custom builds, cross compiling, codegen, ...
> I don't want anything in my Rust dependency tree depending on node packages.
This is not what they’re arguing for.
> Custom builds, cross compiling, codegen, ...
That’s great, none of that is “package management”.
The point is, every language has a way to specify dependencies, find the correct versions, and download them recursively. Could that piece be made generic?
Some languages do all kinds of arbitrary things during package installation so the lowest common denominator is "the package manager can execute arbitrary code that does anything" unless languages change their designs to something more restrictive.
I think things have improved, but you didn't used to be able to build a dependency tree without installing all packages and iirc there was a dependency resolution case where pip would download every version counting backwards until one happened to install
I'm afraid I didn't hear you last time (did I miss aught?)
Perhaps we should standardise "languages" too: all compiled programming do be done in N-- and all interpreted programming in Piffle. Thou shalt not touch machine code and that unless thou be a priest
... no of course not!
A "standardised" package manager is close to the least of our worries in IT land. Today I updated a .deb based server software on an Ubuntu box and used a built in update mechanism to update its client software - this is a MinIO instance. I'm not worried that it will fall to bits.
I hear ya but in practice that doesn't work well. For instance Ubuntu has (or had) .dpkg distributions for many popular Python and Perl libraries, installable via apt. We quickly learned never to use them because they were always out of date and often packaged strangely.
Debian (and therefore Ubuntu) Python packages are largely intended for system Python, that is, the version of Python that ships with the distro. If you install a package from a Deb-like repo, its dependencies will be present, and it will work* as intended.
The downside is, as you found, they're usually out of date. The upside is that if you get a Python program added to Stable, you can rest easy knowing that end users won't have to battle dependency hell to get it running.
*: This does not apply if you've made a FrankenDebian by combining Sid and Stable, or other such things. You're on your own, good luck.
Because the job of a package manager is complex and humanity hasn't figured out a universally satisfactory solution.
Research and development are ongoing, and the state of the art is at a level of complexity that I'd compare to a doctoral project or something, requiring one to throw away tons of the abstractions they know and learn a lot from scratch (nix).
I think about this one very often. I think it can even be generalized: "complex system problem seems intractable due to complexity... I know, I can simplify some bits of it by adding complexity to it".
I know way too many coders and firms taking a "throw stuff to the wall and see what sticks" approach to software. The phrase "just solve the problem and move on" comes to mind. These places are often in firefighting mode and cannot connect the dots for some reason.
The world is simply too complicated for that, there is no way you would get the maintainers of rust, npm, deno, haskell, etc to all agree to one "package system to rule them all".
Delivering files is the easy part, integrating with the language tooling/buildsystem is the hard bit. Seemingly simple concepts like a dependency don’t translate neatly across languages.
Bazel is also 1.4 million lines of code which proves the point of how complex of an endeavor this is. Bazel has a mixed reputation due to that complexity and the overhead that comes with it, and is definitely not just a simple drop in replacement for a language's existing tooling.
Bazel does a lot more than just download packages though. It can also track dependencies down to individual file level and isolate build steps to assure they don’t have side effects. This is largely why it is so big and need to be language aware. Add remote build cache and lots of other stuff.
A general package manager that only sees packages as directory blobs would not need to be that complicated. In fact I don’t think npm has this deep knowledge about JavaScript either, it just put stuff inside node_modules and calls it a day.
That said, it would be great if more projects adopted Bazel, its biggest problem today is adoption, forcing you to convert any dependencies to make it Bazel compatible.
Why is it allowed at all to delete packages? The article states that’s it’s not an issue in Maven since you can’t unpublish anything, only deploy a newer version.
I like the approach that Rust takes with Cargo and crates.io
You can mark a package you’ve published as “yanked”, which will make it so that if someone tries to depend on that version in the future they “can’t” (cargo will tell you it’s yanked). But if it’s in your Cargo.lock file since before, cargo will still fetch it and use it.
This way you can mark a broken version of your published package as bad so that fewer people will use that version. While not preventing anyone who was already using it from continuing to use it.
I don't think it was ever a link to an explanation. It's just a link to the Github advisories page with a search query for 'no-one-left-behind' attached: www.npmjs.com/advisories?search=no-one-left-behind. It's the equivalent of linking to a Google search.
Given the number of bots uploading malicious packages, I wouldn't be surprised if this is mostly all automated, and there's no manpower available to dedicate to descriptions of each shady package uploaded to the world's largest package repository.
If this were a vulnerability in the infrastructure of NPM itself, then I would definitely expect a full writeup with a working link.
Okay, well... then it is broken in THREE ways. First, the link no longer leads to the search, as the redirect seems to strip the parameters. Second, searching the github for "no-one-left-behind" gives no results. Third, this _is_ a vulnerability of the NPM ecosystem, as shown by the fact that this happened _AGAIN_ this week.
> First, the link no longer leads to the search, as the redirect seems to strip the parameters.
Github acquired NPM, and now they're trying to integrate NPM with all of Github's products, so NPM advisories now link to the Github advisories page, which is probably why the search query is being stripped. This is definitely not ideal, but I would probably characterize this as corporate ineptitude. Classic case of a company acquiring a smaller company and not properly handling the integration.
> Second, searching the github for "no-one-left-behind" gives no results.
Like I said, there are bots publishing malicious packages faster than human reviewers can even describe them, so I don't expect a full write up for every prank package. Are you really upset that you had to Google the package instead of following a link in a package's description? Really? Would you instantly mistrust any organization with a link that lead to a 404 page?
> Third, this _is_ a vulnerability of the NPM ecosystem, as shown by the fact that this happened _AGAIN_ this week.
I would hardly characterize this as a vulnerability of the NPM ecosystem. I'm not even sure if this would warrant its own CVE. You act as if the package was installing bitcoin miners on developer machines. Like I said, if this were a vulnerability in the infrastructure of NPM itself (say for instance it managed to hack the servers hosting NPM), then I would definitely expect a full write up. All it's doing is preventing people from unpublishing packages. It was honestly a curtesy of NPM to even take it down. If it was up to me I might make the whole registry immutable.
Another commenter said it best: "In any case this is a mild inconvience at most. Anyone who gets upset over this needs to go touch grass."[1]
Fool me once, shame on you. Fool me twice, shame on me.
I get that this royally inconvenienced a ton of people, but come on: the fact that this can happen in the first place, along with the fact that the expectation for most users is that it can't, is a clear failing on NPM's part.
I don't know what the solution is - maybe packages should be permanent after 24 hours instead of after they're depended upon, maybe there's a different solution - but it's disheartening to see the vitriol with which folks went after PatrickJS in the screenshotted comments in TFA.
That’s what you get after making someone’s life harder. I’m all for positiveness, but what did PatrickJS expect? It’s understandable when someone has a monetary incentive, but this is basically a (light) hooliganism. Protecting from all sorts of jokes makes life harder too. Corner cases are hard. Why can’t we just respect each other and have these few nice things that are left.
In the next episode we’ll be unable to publish more than three packages a day, have to request dependency quotas, investigate ip blocking issues, solve captcha for every basic operation and try to weed through support rounds to unlock the org that was locked due to suspicious activity.
That said, props for them not hiding in a shadow watching things happen.
> I’m all for positiveness, but what did PatrickJS expect? It’s understandable when someone has a monetary incentive, but this is basically a (light) hooliganism.
You're making the assumption that they were aware of the complications their experiment would cause beforehand. The fact that they weren't "hiding in the shadows" indicates they had no ill intent. Also if they didn't do this, someone else eventually would (possible with ill intent).
In any case this is a mild inconvience at most. Anyone who gets upset over this needs to go touch grass.
The article tries to make it sound more serious than it actually is. So you can't unpublish packages on NPM now. So what? Why are you unpublishing it in the first place? Just leave it alone and slap a deprecation notice on it.
Because I'm human and sometimes I make idiotic mistakes and I don't want that embarrassing stuff out there where everyone can see. Same reason I sometimes force push to my repositories: I realized I made a mistake immediately after publishing and would like to correct it as quickly as possible.
Some time ago I ran head first into an npm bug when I tried to symlink the README file which resulted in the package getting published without a README file because apparently someone somewhere is "opinionated" about symlinks.
That was embarrassing and annoying. So I unpublished it. Then I discovered I couldn't replace the bad package, I had to create a new version of the package. So I did and then I discovered they had slapped me with a stupid 24 hour count down on top of all that during which I could not actually push the corrected version. I hate this thing so much, you have no idea.
OTOH if you’ve published sensitive info the cat is out of the bag. Deleting the version is not the correct response, rotating creds is the only viable answer.
I’m sure there are bots watching new versions posted to NPM, scanning them for things like AWS access keys or crypto wallet creds. I’d bet that if you publish a package with AWS creds you will have crypto miners launched on your account within the hour if not within 5 minutes (probably all automated).
As someone who frequently chases dead links and tries to piece together things from archives: Good. We need things to be more permanent. Storage is cheap, history is priceless.
The ironic thing is after all this GitHub accused us of harassment and deleted the GitHub organization (that contained numerous instances of us being harassed)
I haven’t read TFA, but judging from the domain it missed the whole point. We weren’t intending to break anything, we just wanted to install everything. Combined storage size of package manifests were 82mb
I wonder what other pathological packages could be devised. Like, a package that only includes packages that are not included anywhere else. It could be called “barber”.
I've said it before and I'll say it again (and this applies to other scripting language package managers too, e.g. composer for php): while "this public package was unpublished" is probably something a package repo shouldn't support, the actual solution is pretty simple.
Vendor your dependencies.
By which I mean: commit the dependencies you rely on, to your VCS. It's really that simple.
Yes, this probably requires some different usage or default behaviour from package managers to be "efficient" (i.e. you probably don't need to include all the tests, docs etc for each dependency in your repo).
If you're following "best practices" for every library package manager I've seen (to commit their respective "lock" file, thereby fixing the used version of each dependency for subsequent installs), the end result is the same in terms of what version of dependency is available when the source is checked out - but now you have (a) zero risk of dependencies being pulled; (b) the ability to do comprehensive change reviews when versions are bumped; (c) likely faster updates for anyone pulling changes subsequent to a dependency change; (d) zero risk of 'install dependencies' related issues at deployment time.
This only works to a point. When you install dependencies with a tool like npm install, you're installing the dependencies meant for your computer platform. You can't just copy binaries between computers and expect them to work. Even with the same OS you could be missing shared libraries.
Can you give an example of this phenomenon, e.g. a package that one would install using `npm` which includes binaries? I wasn't even aware this was a thing people do.
I'm most disappointed that not a single person has asked the most pertinent question: what's the total size of node_modules after successfully installing the everything package?
No. If foo@1.2.3 is already published, you can not overwrite foo@1.2.3 - you'd publish foo@any.other.version (typically something at least ^1.2.4) and mark foo@1.2.3 as deprecated (which won't actually make it unavailable like an unpublish would).
NPM is a very good package manager registry. It's just extremely popular because javascript is extremely popular and the entry barrier is tiny compared to other languages and registries.
These issues would pop up in any package manager, no matter the language, you just need enough users and usecases.
No, NPM has never been thought out well. Many of the improvements in npm over the years were all either (1) due to accidents that many other package managers like NuGet already prevent by default or (2) features that were already implemented in Yarn 1.x.
108 comments
[ 3.2 ms ] story [ 190 ms ] threadIt's such an interesting and important field but largely treated as an annoyance, dismissed by the very people who should be learning about it.
Simply having a table of all the fundamental concepts involved and how each package manager handles them would be really interesting. Or even just a list of the fundamental concepts useful for building your mental model of how they work!
Doesn't happen to literally any other programming language (or maybe I just haven't paid attention to others)
As has rust. https://blog.rust-lang.org/2022/05/10/malicious-crate-rustde...
NPM wants to have its cake and eat it too, which is the problem here. The solution is just to say that if you publish a package to NPM you give it the perpetual right to distribute it as-is, and then remove the ability for users to delete their packages at-will.
So many issues arise from the fact that our UIs can’t do simple things on lists like folders (e.g. archive) and tags/notes.
Yes, exactly!
Kind of unrelated, but I think it is important to remember that the left-pad was also ENTIRELY npm team's fault. You can't just take away a namespace from someone just because some startup like kik comes knocking.
Toyota does not have a right to my domain dot tld slash toyota The correct answer would have been npm to tell kik to pound sand.
npm has never fixed this grave error.
https://blog.npmjs.org/post/141577284765/kik-left-pad-and-np...
> We stand by our package name dispute resolution policy, and the decision to which it led us.
npm deserves to die.
The way PyPI does deletions is through "yanking," which is really a soft-delete mechanism. Yanking prevents usage of a particular package version for new resolutions, but allows fully-resolved versions to be downloaded. As a result, PyPI doesn't have the specific behavior that caused this (forbidding people from removing their packages because removal means hard-deletion).
[1]: https://hackage.haskell.org/package/acme-everything
Personally, I think if you're not sure you want a package to be available publicly in perpetuity, you should maybe consider publishing it to a private registry. Even if NPM did allow developers to unpublish any package, there's nothing stopping me from downloading a tarball of your package as soon as you publish it, and hosting a mirror of it somewhere else.
As far as I can tell, NPM does not enforce any limitations on the license of published packages; I would not assume you can legally do this unless you have verified the package has been published under a license which permits such actions and which you are certain you understand.
I'm not a lawyer, but in my best judgment I see no reason to assume an NPM package can continue being used after it is taken down -- you need to actually evaluate the license it's published under to determine that. If I'm mistaken, I'd love to see a source establishing the contrary.
[0]: https://docs.npmjs.com/policies/open-source-terms
I once saw a package on NPM that had a commercial license. It even had a link to its source on GH.
And a note that if you used the package you needed to pay up.
When you upload something to the internet it's there forever. The only way to ensure something is not on the internet is if you don't post it (and even then...). People are destined to learn this lesson over and over again.
I don't see these as serious contenders anymore.
Was there a legitimate purpose or is he just a troll?
How crazy that there’s not like a limit or some kind of basic check to prevent the upload or something?
Honestly at this point it seems to be a failure on NPM’s part. People should really direct their anger to the maintainers at this point
And it's not really arbitrary. You can count the dependency tree of every package and get the P99, exclude joke packages and get the max of the rest. Set the limit around there with an error message to open a bug if you need more, grandfathering legit old large packages with some kind of admin-level exception.
The number of people actually needing more will be incredibly tiny. Service those folks individually to avoid pain.
When it's a staffing limit the question becomes less arbitrary.
The person made thousands of modules that look like `@everything/sub-chunk-{id}`, each depending on N modules. Those sub-chunk modules, together, depend on all NPM modules. Whatever upper limit you come up with, anyone can do this by setting N to your upper limit.
The problem here isn't that you can create modules that depend on other modules. If there's a problem at all, it's that the NPM's yank policy could be improved.
And I agree about yank! I just think you could implement multiple reasonable solutions here
That they rolled all of those submodules into one big everything package is just a gimmick, not a source of the problem.
Call attention to a real issue that should be solved by the ecosystem so that people cannot do it in the future? In my opinion it is not useful or even sane to expect people to just _not_ do annoying things if you develop systems that allow people to do things that annoy you with them. Design and build systems that assume people will be trolls, and make it so people cannot so easily troll you with them. Failing that, there's nobody to point the finger at besides the person or people who made it possible.
The pranksters "thought it would be funny" (one is quoted as saying in the article).
The benefit of CDNs used to be that you could cache a library for use on multiple domains but I don't think that works anymore.
These don't seem bad for dependency-free libraries, but they won't warn you about incompatible package dependencies.
SMACK
Somewhere in an alternate universe, there's a better timeline.
But I also think it’s been permanently tainted due to a constantly shifting ecosystem that can’t settle down, and a user community that is constantly chasing new shiny things.
Dependency management is a disaster in a ton of ecosystems. It is often a different set of disasters than what npm has, but it isn't like this is a solved problem everywhere else and it is just the js ecosystem that sucks. NPM is made marginally worse by the poor standard library in JS but I don't think the problem is fundamentally attached to the language in any meaningful way.
And the push to use JS for backends was not "we want this to be a serious language" but was instead "well, we are already using JS for frontends so it'd be nice from a practical standpoint for hiring and training to be able to use the same language and ecosystem for backends too."
By being hermetically sealed, the community can make progress on testing, repeatability, and lots of things that would become incredibly difficult if you introduced anything else into the mix.
Moreover, each language gets to innovate in its own way. Cargo is fantastic and fits the needs of Rust quite well. Custom builds, cross compiling, codegen, ...
This is not what they’re arguing for.
> Custom builds, cross compiling, codegen, ...
That’s great, none of that is “package management”.
The point is, every language has a way to specify dependencies, find the correct versions, and download them recursively. Could that piece be made generic?
Tight coupling matters.
> Could that piece be made generic?
Semver has different meanings to different languages based on compiler versions.
Some languages want 100% build reproducibility and hermeticity, and some languages will never get there.
Some languages permit yanking.
Some languages allow binary distributions.
...
These decisions go very deep to the fundamental ethos of the languages themselves. I don't think we'd find any agreement.
Python is an example of a language that can (and historically has) done arbitrary things in package installation https://stackoverflow.com/a/43350802/2751619
I think things have improved, but you didn't used to be able to build a dependency tree without installing all packages and iirc there was a dependency resolution case where pip would download every version counting backwards until one happened to install
Perhaps we should standardise "languages" too: all compiled programming do be done in N-- and all interpreted programming in Piffle. Thou shalt not touch machine code and that unless thou be a priest
... no of course not!
A "standardised" package manager is close to the least of our worries in IT land. Today I updated a .deb based server software on an Ubuntu box and used a built in update mechanism to update its client software - this is a MinIO instance. I'm not worried that it will fall to bits.
Anyone with a Windows box with a worrying number of Visual C++ runtimes might like to remove the lot and install the latest from here: https://learn.microsoft.com/en-us/cpp/windows/latest-support...
The downside is, as you found, they're usually out of date. The upside is that if you get a Python program added to Stable, you can rest easy knowing that end users won't have to battle dependency hell to get it running.
*: This does not apply if you've made a FrankenDebian by combining Sid and Stable, or other such things. You're on your own, good luck.
Because the job of a package manager is complex and humanity hasn't figured out a universally satisfactory solution.
Research and development are ongoing, and the state of the art is at a level of complexity that I'd compare to a doctoral project or something, requiring one to throw away tons of the abstractions they know and learn a lot from scratch (nix).
I know way too many coders and firms taking a "throw stuff to the wall and see what sticks" approach to software. The phrase "just solve the problem and move on" comes to mind. These places are often in firefighting mode and cannot connect the dots for some reason.
I guess the real answer is because it's a lot of work to maintain packages and versioning and there is space for multiple solutions.
A general package manager that only sees packages as directory blobs would not need to be that complicated. In fact I don’t think npm has this deep knowledge about JavaScript either, it just put stuff inside node_modules and calls it a day.
That said, it would be great if more projects adopted Bazel, its biggest problem today is adoption, forcing you to convert any dependencies to make it Bazel compatible.
You can mark a package you’ve published as “yanked”, which will make it so that if someone tries to depend on that version in the future they “can’t” (cargo will tell you it’s yanked). But if it’s in your Cargo.lock file since before, cargo will still fetch it and use it.
This way you can mark a broken version of your published package as bad so that fewer people will use that version. While not preventing anyone who was already using it from continuing to use it.
https://learn.microsoft.com/en-us/nuget/nuget-org/policies/d...
1: https://github.com/npm/feedback/discussions/858
2: https://www.npmjs.com/package/no-one-left-behind
Given the number of bots uploading malicious packages, I wouldn't be surprised if this is mostly all automated, and there's no manpower available to dedicate to descriptions of each shady package uploaded to the world's largest package repository.
If this were a vulnerability in the infrastructure of NPM itself, then I would definitely expect a full writeup with a working link.
Github acquired NPM, and now they're trying to integrate NPM with all of Github's products, so NPM advisories now link to the Github advisories page, which is probably why the search query is being stripped. This is definitely not ideal, but I would probably characterize this as corporate ineptitude. Classic case of a company acquiring a smaller company and not properly handling the integration.
> Second, searching the github for "no-one-left-behind" gives no results.
Like I said, there are bots publishing malicious packages faster than human reviewers can even describe them, so I don't expect a full write up for every prank package. Are you really upset that you had to Google the package instead of following a link in a package's description? Really? Would you instantly mistrust any organization with a link that lead to a 404 page?
> Third, this _is_ a vulnerability of the NPM ecosystem, as shown by the fact that this happened _AGAIN_ this week.
I would hardly characterize this as a vulnerability of the NPM ecosystem. I'm not even sure if this would warrant its own CVE. You act as if the package was installing bitcoin miners on developer machines. Like I said, if this were a vulnerability in the infrastructure of NPM itself (say for instance it managed to hack the servers hosting NPM), then I would definitely expect a full write up. All it's doing is preventing people from unpublishing packages. It was honestly a curtesy of NPM to even take it down. If it was up to me I might make the whole registry immutable.
Another commenter said it best: "In any case this is a mild inconvience at most. Anyone who gets upset over this needs to go touch grass."[1]
[1] https://news.ycombinator.com/item?id=38883067
I get that this royally inconvenienced a ton of people, but come on: the fact that this can happen in the first place, along with the fact that the expectation for most users is that it can't, is a clear failing on NPM's part.
I don't know what the solution is - maybe packages should be permanent after 24 hours instead of after they're depended upon, maybe there's a different solution - but it's disheartening to see the vitriol with which folks went after PatrickJS in the screenshotted comments in TFA.
In the next episode we’ll be unable to publish more than three packages a day, have to request dependency quotas, investigate ip blocking issues, solve captcha for every basic operation and try to weed through support rounds to unlock the org that was locked due to suspicious activity.
That said, props for them not hiding in a shadow watching things happen.
You're making the assumption that they were aware of the complications their experiment would cause beforehand. The fact that they weren't "hiding in the shadows" indicates they had no ill intent. Also if they didn't do this, someone else eventually would (possible with ill intent).
In any case this is a mild inconvience at most. Anyone who gets upset over this needs to go touch grass.
Some time ago I ran head first into an npm bug when I tried to symlink the README file which resulted in the package getting published without a README file because apparently someone somewhere is "opinionated" about symlinks.
https://github.com/npm/cli/issues/6746
That was embarrassing and annoying. So I unpublished it. Then I discovered I couldn't replace the bad package, I had to create a new version of the package. So I did and then I discovered they had slapped me with a stupid 24 hour count down on top of all that during which I could not actually push the corrected version. I hate this thing so much, you have no idea.
This doesn't work if you've accidentally published sensitive information or files. Though I agree that deleting packages is generally not necessary.
I’m sure there are bots watching new versions posted to NPM, scanning them for things like AWS access keys or crypto wallet creds. I’d bet that if you publish a package with AWS creds you will have crypto miners launched on your account within the hour if not within 5 minutes (probably all automated).
I haven’t read TFA, but judging from the domain it missed the whole point. We weren’t intending to break anything, we just wanted to install everything. Combined storage size of package manifests were 82mb
Vendor your dependencies.
By which I mean: commit the dependencies you rely on, to your VCS. It's really that simple.
Yes, this probably requires some different usage or default behaviour from package managers to be "efficient" (i.e. you probably don't need to include all the tests, docs etc for each dependency in your repo).
If you're following "best practices" for every library package manager I've seen (to commit their respective "lock" file, thereby fixing the used version of each dependency for subsequent installs), the end result is the same in terms of what version of dependency is available when the source is checked out - but now you have (a) zero risk of dependencies being pulled; (b) the ability to do comprehensive change reviews when versions are bumped; (c) likely faster updates for anyone pulling changes subsequent to a dependency change; (d) zero risk of 'install dependencies' related issues at deployment time.
NPM is a very good package manager registry. It's just extremely popular because javascript is extremely popular and the entry barrier is tiny compared to other languages and registries.
These issues would pop up in any package manager, no matter the language, you just need enough users and usecases.