It looks like this addresses the biggest issues people have with npm's CLI, and it's coming from such huge names: Facebook, Google, and Tilde. Reproducible builds are a _huge_ issue, and this gives you that. Looks great!
One interesting little tidbit I found from diving into the source:
Oh hi Steve! https://registry.yarnpkg.com is just a proxy to the normal npm registry provided by Cloudflare so we can add additional caching and work on network performance (lots of stuff we could layer on top). It should hopefully help those in China as well which would be awesome. But everything you do still lives on the normal npm registry.
hi! they are running a mirror of the npm registry which contains all the package data but none of the permissions/user mgmt. this means that they are still dependent on npm infrastructure.
npm, Inc strongly encourages people to build and use mirrors. for example, cloudflare (where the mirror lives, and where seb used to work) has been running https://npmjs.cf/ for a long time.
That's not 100 % accurate. Arguably symlinks are kinda weird under Windows, but that doesn't mean symlinking node_modules doesn't work (you just need admin privileges).
Just a heads up: someone tried to comment out the Windows choco instructions with HTML comments but the angle brackets were escaped so the instructions are still there (with the escaped brackets).
The recommended installation method is actually to use the package/installer for your environment. Currently there's a Windows installer, Debian package, and RPM package, with Homebrew coming soon. See https://yarnpkg.com/en/docs/install
I hope this becomes the default way to build npm projects... and then npm install becomes yarn :-) Exactly reproducible builds is a godsend to the js community.
> Yarn, a collaboration with Exponent, Google, and Tilde.
They should mention this at the very beginning. Multiple big players investing in this package manager means that we should maybe inspect a little bit more before chanting xkcd.com/927.
927 was also my first thought. But what made me to reconsider was not more than one big name behind it (but it helped), but the fact that they rely on npm backend and did not reinvent everything from scratch.
Basically, yarn is npm client done right reusing the same npm package repo.
This is something I really want to explore more through less-compatible modes. It was the original way yarn worked, but it wasn't compatible enough to be the default mode: https://github.com/yarnpkg/yarn/issues/57
If you do, would love see experiments on that. I do wish NPM would start adding badges or some other meta data to packages that signify that a module has constraints like:
- No native code
- No module state (No multiple singletons, and easily load variants of a module)
- Pure installation, aka no build steps and is fully cacheable via hashing/immutable module patterns
- Is all pure functional
- 100% code coverage
- Etc.
Yeah, that would be much better. But unfortunately the node_modules structure (which yarn is attempting to be fully compatible with) makes that impossible.
The reason is that each package only finds its dependencies relative to its own location. So your second level dependencies cannot vary from project to project unless you do copying.
(Example: AppA and AppB depend on LibX. LibX depends on LibY. Through their deterministic lockfiles, AppA and AppB disagree on which version of LibY to use. There is no way to symlink things together to satisfy that case without copying LibX or altering node's package resolution algorithm.)
npm chooses the first version of a package it encounters to install at the top level. Every other version is installed nested. If you have N uses of version A and one use of version B, but version B is installed first, then you get N copies of the package at version A.
What's the solution for that case though (one which requires no nesting)? I don't think a solution with no nesting exists given the current module resolution algorithm of Node.js, which allows for only a single version of a particular package to exist at a given level.
nesting can't be prevented in that case (at least with the current node require() design). It gets really bad if you have 10 packages that depend on c@1.0.0 and 10 packages that depend on c@2.0.0 -- one of them will install in the root directory and the other 10 will be duplicated. ied stores a single copy of each package@version and uses symlinks which is an improvement. Apparently yarn tried something similar but it breaks some packages (that check where they're located for example).
I could be prevented. Another project does this by using a hash of the project to all all dependencies at one level then symlinks the dependencies from hash to name so it references the correct version.
You can use `npm run env bin-cmd` for that. It's not as convenient as `npm exec bin-cmd` would be, but it might be easier to type than `$(npm bin)/bin-cmd`.
Symlink that all you want, how is it in any way whatsoever thoughtful and elegant? It's not a real solution by any stretch of the imagination. Myriads of pre-existing systems that correctly deal with this scenario existed at the time that NPM was concoted. Ignored. Considering that node is touted as a cross-platform solution, how well does version@3 deal with filesystems that don't support symlinks? I've run into this: not at all.
NPM version@3 could not be further from the ethos of "engineering." It's an outright hack, and it quickly betrays its limitations. It could have been far simpler, far more considerate of pre-existing wisdom.
no, it just installs everything it can into the top level node_modules. npm@2 would do this to a degree as well, it was just less sophisticated about it.
Interested to see the emphasis on security, it's definitely something that could use more focus in this area. On that note, are there any plans to implement TUF (https://theupdateframework.github.io/) or similar into Yarn?
You say "decent performance" and "predictability". What is the basis of this claim? I've heard these all before, but unless you've actually shipped a product using this tool I don't know how you can back this up.
We're not using Yarn at Tilde yet, but I've been using it on Yarn itself and when working on Ember packages, and it's worked well. As far as performance, we have good benchmarks tracking yarn's performance in a whole bunch of different scenarios (https://yarnpkg.com/en/compare)
Ok, didn't realize he worked on it. Bundler is great so that gives me some hope for Yarn. But in general I think its more stable in the long-term for projects to use open standards instead of vendor solutions for browser package management.
I can understand this sentiment in general, but don't think it applies here. npm is also a private company, and this does work with their registry. There currently isn't a vendor-neutral standard to follow.
As for performance, Yarn is about 3-5x faster installing all of the dependencies and devDependencies of react-native. The benchmarks are here: https://yarnpkg.com/en/compare. It's much faster in most scenarios, especially the ones that used to take minutes.
The benchmarks test both the hot cache and cold cache scenarios for npm and Yarn. Except in the scenario when node_modules is already populated and there is no yarn.lock file, which is uncommon if you're using Yarn, Yarn is faster than npm in all of the benchmarked scenarios where there is a cold cache.
This looks awesome. But I have to wonder why create a whole new project rather than fork or upstream these changes to NPM? It doesn't seem like it's doing anything fundamentally different or outside of NPM's scope of responsibility.
Doesn't npm effectively replicate bundler's functionality (poorly) via shrinkwrap? Why not fix that feature rather than replacing it with a separate utility?
The official npm cli is quite old and evolved along with all the different coding styles and architecture choices of the node and JS ecosystems.
It's also full of dependencies on originally purpose-built modules that suffer from the same problems. This compounds various issues, resulting in long-standing bugs like `npm publish` sometimes not actually including all files in the tar bundle it generates[0].
NPM 3 was a partial rewrite but it's still built on the same mountain of code as NPM 2 and 1. npm Inc developers have in the past complained about not being able to address issues for a lack of resources despite having a ton of developers on the team[1].
The official npm client also doubles as an account management CLI to the npm registry. Which apparently npm Inc now considers a design flaw as they recently started creating a standalone tool for managing other aspects of their commercial services[2].
And finally it's important to understand that having a new package manager that is not tied into npm Inc means it's easier to replace the npm registry as the official node module registry in the future. Which would allow the community to get rid of the privileged position npm Inc currently holds in the node distribution (which is otherwise governed by the Node Foundation).
Rewriting the client from scratch actually turned out to be the sanest approach in this situation, I think.
fwiw, only 2 of those humans work on the CLI, (3 if you count their manager, who spends a lot of time managing). other people work in sales, marketing, engineering on the website, engineering the registry, community + docs, etc.
wombat is not a tool for managing our other commercial services! it's a way to manage our webhooks product, one of many products we have, though very certainly the smallest.
yarn depends on the npm registry, which depends on npm, Inc making money to keep it running.
Thanks for clarifying. But that does show that having a company fund two full-time developers to work on the CLI wouldn't have been sufficient. Yarn was mostly built by Seb (403 commits) starting in late January and Konstantin Raev (121 commits) with contributions from several others (<30 commits each). The commit graph[0] actually shows development mostly paused in April and May, putting the total development time closer to half a year.
This most likely includes collaboration with the other people supporting the project (both Seb and Konstantin work for FB) and a lot of architectural planning -- aspects, which likely would have required an even greater investment of time if building on the legacy codebase of the official npm client (not to mention having to negotiate all changes with npm Inc directly).
> wombat is not a tool for managing our other commercial services!
Sorry about the misunderstanding. When wombat came out a lot of people wondered why wombat was a standalone tool when practically every other aspect is part of the npm cli and from what I gathered from npm devs at the time the decision was more of a technical one (different people working on different tools for different things) than a managerial one (different products deserving different tools).
> yarn depends on the npm registry, which depends on npm, Inc making money to keep it running.
Yarn currently uses a downstream mirror of npm as the direct source of truth, with support for private registries being actively worked on to support the private parts of the npm registry and products like npm enterprise or free alternatives like sinopia.
However that is obviously an implementation detail. Decoupling yarn from the npm registry by going through an intermediary allows experimentation (as the npm blog post points out) and ultimately allows avoiding a hard dependency on npm Inc as a company.
If npm Inc were to shut down right now, yarn would still have a read-only mirror. I'm sure Facebook and Google have enough experience solving problems at scale that they could build a public registry alternative if necessary (such as if npm Inc went out of business tomorrow).
I'm not saying yarn will kill npm Inc. But yarn can kill the npm client and replace it with a community-supported alternative (under an open governance model similar to Ember), which in turn makes it a lot easier to fade out the reliance on the public npm registry maintained by npm Inc -- benefiting both the community (by eliminating a single point of commercial failure in the node ecosystem -- exactly like joyent was to the node project) as well as npm Inc (by eliminating a major drag on development resources and cost center).
The only drawback from such a move to npm Inc would be the loss of its privileged position in the ecosystem, the "first mover" advantage of being the place to go if you want private packages (because it's more frictionless than any of the alternatives -- exactly like GitHub in the git hosting space).
The first release of Node.js was in 2009. "Old" is relative.
Nearly all modern JavaScript build tools are built on Node.js and NPM pre-dates most of them.
The only other thing I can think of that has seen as many shifts in JS development practices as NPM is Express (*2009) and that library has far smaller complexity and has arrived at a drastic reduction in features and a far tighter scope.
The NPM client on the other hand was originally cobbled together by Isaac and then grew organically as the NPM registry massively grew over the months and years of node's success.
The NPM client and registry are the epitome of technical debt. Calling it "old" is the most charitable way to phrase this.
This is a huge leap forward for the JavaScript community—probably more than many people will realize right away.
I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once. npm solved this problem elegantly, but still struggles with non-determinism. I had resigned myself to thinking that maybe these were just fundamental tradeoffs in package management that could never be resolved.
Then I had the opportunity to use Cargo, the package manager for Rust. It synthesized what, in my mind, are the best features of npm and Bundler into a single package manager, with terrific speed to boot.
Yarn looks like it will be Cargo for JavaScript. After Cargo improved on many of the great ideas of npm, those ideas are returning to help improve the JavaScript ecosystem. Cross-pollination at its best.
This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.
I'm also happy to see that a significant amount of work has already gone into the governance model. Despite the largest contributors being Facebook employees, it looks like they've really outdone themselves making sure this is a community-run project. A BSD license, no PATENTS file, and an Ember/Rust RFC process[1]; this has all the hallmarks of a community open source project. That's critical for open infrastructure projects like this, and they've nailed it.
I'm very much looking forward to using Yarn in my own projects because it looks like it solves a lot of real problems I encounter every day. Thanks for all the hard work!
Well, it seems like it could be according to the article. What would you still need to use NPM (the cli) for, other than package hosting?
According to http://blog.npmjs.org/post/151660845210/hello-yarn, it seems it doesn't work with private packages yet, which may or may not be an issue for your project. But it seems this is a complete CLI replacement for NPM.
"Could have been implemented as part of the existing client" isn't the same as "Should have been implemented as part of the existing client".
I personally don't know much about either tool (don't do a ton of JS), but it's possible that fixing the existing client without either breaking backward compatibility or making it too complicated (multiple modes of operation) was too difficult or not worth it.
Also, I'm having a really hard time understanding the complaint about a new client. The value is in repository of reusable code, not the client. That you can use different clients with the same repository is a feature, not a bug.
Yeah - the npm client codebase would not be fun to do a big refactor on. And you would need to do a big refactor to make the v3 flat install process deterministic and/or performant.
These issues have been raised a few times, along with the shrinkwrap/reproducability stuff, and it didnt seem like it was a big priority for the core team. Understandably I guess they seem more focussed on the enterprise/private repo side of things and just keeping things running on the client side.
The whole version range stuff got me many times. I went to use fixed versions on my own package.json files, but the deps of my deps could still be dynamic, which is even worse, since they sit deeper in my dependency graph AND there are more indirect deps than direct deps. (~50 direct, >200 indirect)
Also, npm isn't deterministic and it got even worse with v3. Sometimes you get a flat list of libs, if a lib is used with multiple versions, the first usage will get installed flat, the rest in the directory of the parent lib, etc.
The fun of kicking off a CI build after the weekend with no commits and see stuff randomly break because some dependency of a dependency got updated and broke things in a minor version is something I've only experienced in JS - beautiful.
What we do currently is we lock everything to an explicit version - even libraries.
At least it's possible to get deterministic builds if you are willing to do a bit of work carefully / manually updating all of your dependencies at once.
You shouldn't have that happen with Cargo, given that we have a lockfile. Even when you specify version ranges, you're locked to a single, specific version.
If you follow the best practices for using Cargo you don't have a Cargo.lock file for libraries.
This means your library tests will not be deterministic.
Using the Cargo.lock file for libraries does solve this, but then every binary package you build that references your library will have to be specifically tied to the versions in the library.
We do this internally because it's the only way to provide deterministic builds over time.
Over time I suspect it will get harder and harder to keep this rigid web of dependencies working.
One thing we might try is to enforce the rule 'no library will contain any tests'. At first glance this kinda makes my skin crawl, but maybe if we could find a way where every project that used a library could somehow share the tests for that library it could actually work.
git submodules might actually be able to provide these tests for each binary package. If only git submodules would stop corrupting our git repos... :-(
I agree 100%, but the default upgrade strategy for npm --install does not help matters: it's much saner to wildcard patch versions only and lock major.minor to specific versions.
this obviously doesn't fix anything and I think the points in this discussion stand, but I've never understood why the defaults are not more conservative in this regard.
Obviously without until we broke and looked in to it :D
JS dev has been a minefield like that, the entire ecosystem reminds me of C++ except lower barrier to entry means a lot of noise from newbies on all channels (eg. even popular packages can be utter crap so you can't use popularity as a meaningful metric)
In fairness to the language and tools, this seems to be more of a cultural problem than anything.
You can do the same kind of version range tricks in typical Java builds, for example (Maven), but most people hardcode the values to keep builds as deterministic as possible.
For some reason, the JS community seems to prefer just trusting that new versions won't break anything. Its either very brave of them really (or maybe just foolish).
I hate non-reproducible builds and semver-relaxed dep-of-the-dep issues, but, while a broken dep fails the build for lots of people (downside), the upside of this is that very quickly (within hours of a new dep being published) there will be lots angry people complaining about it on GitHub, and a faulty dep will be typically quickly rolled back / superseded with a patch. Otherwise, some bugs might be sitting hidden for a long time.
It's not a cultural problem. People make mistakes. People don't know what a non-breaking change is, especially those not well versed in refactor work.
I don't think Yarn solves any of these problems, tbh. It seems like what we really need is a package manager that tests the api of each package update to make sure nothing in it has broken expectation in accordance with Semver.
> the JS community seems to prefer just trusting that new versions won't break anything. Its either very brave of them really (or maybe just foolish).
Let's not pretend that we aren't all blindly tossing in random libs of dubious quality and origin we find on github into our package.json and hoping for the best anyway. My company talks a mean talk about "best practices", but, my god, the dependencies we use are a true horror show.
Say that I use yarn to depend upon some module X, specifying a fixed version number like the good boy scout that I am. Module X resides on npmjs and in turn depends upon the bleeding edge version of module Y. And then one day module Y goes all Galaxy Note and bursts into flames.
You shouldn't be kicking off CI builds based on unpinned deps (unless you're deliberately doing 'canary testing' etc), because of course that will break. The npm solution for this is to use 'npm shrinkwrap' and you should always have been using this at your site/project level otherwise there was no hope it could work.
It's not that npm devs were naive enough to believe that unpinned deps would be safe for reproducible builds.
However I've heard several people allude over the years that 'npm shrinkwrap' is buggy, and isn't fully reproducible (though never experienced any problems personally). This is the aspect yarn claims to address, along with a supposedly more succinct lockfile format.
Mhm that sounds reasonable.
I haven't gotten into problems with versions yet but I don't really trust npm update, because I am not always sure how it behaves.
From a build tool perspective (we use npm scripts for basically everything [and some webpack]) I am also not missing something particular.
Looking at other comparable solutions (from other languages) I'd say npm does a pretty good job.
NPM is deterministic when there using the same package.json and there is no existing node_modules folder.
And if you want to lock versions for your entire dependency tree, npm shrinkwrap is what you're looking for (It's essentially the same as lockfiles in other development package managers). Though for security reasons I prefer to keep things locked to a major version only (e.g. "^2.0.0"). Shrinkwrapping is useful in this instance too if you need to have predictable builds (and installs as it'll use the shrinkwrap when installing dependencies too if it's a library rather than the main app) but want to ensure your dependencies stay up to date.
It's not perfect by any measure, but there are ways to make it work the way you want.
Cargo definitely allows two dependencies to rely on different versions of a transitive dependency. If those deps expose a type from that dependency in their API, you can very occasionally get weird type errors because the "same" type comes from two different libraries. But otherwise it Just Works(tm).
Cargo does allow multiple versions of transitive dependencies. It tries to unify them as much as possible, but if it can't, it will bring in two different versions.
What it _won't_ let you do is pass a v1.0 type to something that requires a v2.0 type. This will result in a compilation error.
There's been some discussion around increasing Cargo's capacity in this regard (being able to say that a dependency is purely internal vs not), we'll see.
I feel like this is especially problematic when using NPM for your frontend. Now you have to run all your code through deduplication and slow down your build times or end up with huge assets. I wonder if it's really worth the trouble.
This is especially true when you're going to be serving your code over the web. It's very easy when using npm to accidentally end up shipping a lot of duplicate code.
That alone has me super excited about Yarn, before even getting into the performance, security, and 'no invisible dependency' wins.
As a practical matter, the npm ecosystem today relies on duplication, and no new client that made the "highlander rule" (there can be only one) mandatory could succeed.
Yarn does offer a `--flat` option that enforces the highlander rule, and I'm hopeful that the existence of this option will nudge the ecosystem towards an appreciation for fewer semver-major bumps and more ecosystem-wide effort to enable whole apps to work with `--flat` mode.
Just imagine two packages you depend on (a and b) that both have a shared dependency (x). Both start off depending on x version 1.0 but then later a is updated to 2.0 while b isn't. Now you have two packages depending on different versions of the same package and hence the need for duplication. You have a that needs x@2.0 and b that needs x@1.0, so both copies are kept.
Don't upgrade a when it wants a half-baked x. Choose versions of a and b that agree on a known-good version of x. If there aren't any, it's not sane to use a and b together unless x is written very carefully to accommodate data from past and future versions of itself.
I'm arguing for choosing dependency versions that don't require you to break the highlander rule. "a is updated to 2.0" doesn't mean you should start using that version of a right now.
Right but what I'm saying is that two versions of the same library will live quite happily together, because node.js absolutely abhors global scope. The two versions have their own module-level scope to store functions and data.
So go crazy and install every version of lodash! Nothing will break.
It's almost certain that a's version of x and b's version of x have distinct types whose names collide, and I don't know of any Typescript or Flow notation to prevent passing incompatible instances between them (e.g., a wants to call a method that didn't exist in b's version of x), so if anything works it's only by luck.
Node's module system doesn't import dependencies into a global namespace. So the `x` required by `a` and the `x` required by `b` will have separate namespaces and won't conflict, or even know of each others' existence. There are pros and cons to this approach, but it definitely does work. Flow, at least, (and presumably Typescript) understands the module resolution logic and handles this very common situation without issue.
It's not possible to import two different versions of a module in the same module.
If a depends on x v1 and exports this dependency then your application also imports x v1.
If b depends on x v2 and exports this dependency too that means your application is transitively importing both x v1 and x v2 which is not possible.
If a or b only use instances of x internally and do not accept instances of x as parameter or return an instance of x they don't have to export their dependency on x.
If either a or b do not export their dependency on x then there is no problem. Your application will only depend only on x v1 or x v2 directly.
That will work for lodash, because it's just a bag of functions. But anything that creates objects that are passed outside of a single module could have problems.
That seems to be a slightly different situation? As the issue poster described, a single page had invoked two copies of React, both of which assumed ownership of a particular "global" variable. (Not really global, but a particular member of a global.)
I'm not a React expert, but I don't see why that situation would only affect multiple React copies with different versions?
I'm not a large JS developer but with my other experiences managing dependencies, it doesn't always allow this as a possibility. A bug in your system is tracked down to the library b using x@1.0 but the fix is to upgrade to b with x@2.0 however a using x@1.0 doesn't have an upgrade path. Waiting for another company or organization to release with an update is not an option. Our projects have several cases of requiring different versions of the same library -- we try to avoid this using the same logic you suggest but it's not a possibility in all cases so we have to work with it.
It's placing your own release cycle at the whims of your dependencies' release cycles. In the corporate world that would not be a viable solution.
It's not as simple as that. Lodash is a great example of why the highlander rule doesn't work within the npm ecosystem: older versions are depended on by many widely-used packages which are now "complete" or abandoned. Refusing to use any packages which depend on the latest version of Lodash is just not practical.
Would it be possible to create hardlinks or symlinks to a particular package/version pair shared as a dependency between other packages? I know this only works on unix-like OSes but otherwise it could revert to the old behaviour of duplicating the dependency.
I think they're just saying that any new client that tried to not support duplication at all would likely quickly run into a large amount of npm packages/package combinations that just don't work. So within the context of using the npm registry duplication is mandatory.
With tiny modular utilities this is very much necessity - and not a bad idea if the different versions are being used in different contexts.
For instance, when using gemified javascript libraries with bundler it is painful to upgrade to a different version of a javascript library for the user facing site while letting the backend admin interface continue using an older one for compatibility with other dependencies.
That's exactly what Cargo does, but it also takes a further step of making `^` the default operator and strongly discouraging version ranges other than "semver compatible" post-1.0.
Personally I'm not really sure I like it. If I specify an exact revision of something, chances are I really do mean to install that exact revision. I don't see why I need an extra flag for that.
That can cause some serious problems in at least some portion of times. I've dealt with the subtle errors that have been caused by this problem in c++, and don't really know javascript libraries that well so I can't give a more concrete example. But imagine that there are the following libraries:
* LA: handles linear algebra and defines a matrix object.
* A: reads in a csv file and generates a matrix object using LA
* B: takes in a matrix object from LA, and does some operations on it
In this case, if B depends on version 5 of LA and the new version of A depends on version 6 of LA, then there's going to be a problem passing an object that A generated from version 6 and passing it to B which depends on version 5.
The problem does happen in JavaScript. But since its unityped, there is a strategy to deal with it
* Figure out early on (before 1.0) what your base interface will be.
For example, for a promise library, that would be `then` as specified by Promises/A+
* Check if the argument is an instance of the exact same version.
This works well enough if you use `instanceof`, since classes defined in a different copy of the module will have their own class value - a different unique object.
* If instanceof returns true, use the fast path code (no conversion)
* Otherwise, perform a conversion (e.g. "thenable" assimilation) that
only relies on the base interface
Its not easy, but its not always necessary either. Most JS libraries don't need to interoperate with objects from previous versions of themselves.
Would this even work in what I describe? For instance, if mat.normalize() was added in LA-6, and B provides an LA-5 mat, and then A (which has been updated to use the new method) calls mat.normalize() on the LA-5 mat expecting an LA-6 mat but because of duck-typing that method doesn't exist.
However, since A exposes outside methods that take a matrix as an argument, it should not assume anything beyond the core interface and should use LA-6's cast() to convert the matrix.
The problem is partially alleviated when using TypeScript. In that case the inferred type for A demands a structure containing the `normalize` method, which TypeScript will report as incompatible with the passed LA-5 matrix (at compile time). That makes it clearer that `cast` would need to be used.
If an API expects the outside world to hand it an instance of a specific library, all bets are off. Maybe it gets `null` or the `window` object, who knows? But a library can at least declare what dependencies it wants. If you take that away, it ratchets up the uncertainty factor that much more.
Define the matrix class in a seperate library for compatibility purposes if it's so widely used and doesn't change. Maybe some people don't need both the linear algebra and instead only the definition of the matrix object?
Another solution is to provide version overrides and make B depend on version 6.
However if there are differences in the matrix class between different versions of the library then you're forced to write a compatibility layer in any case.
You've got to take the ecosystem into account. There are a lot of very depended-upon modules that are 1 KB of code and bump their major version as a morning habit. Forcing people to reconcile all 8 ways they indirectly depend on the module would drive them nuts, but including 4 copies of the module doesn't hurt much.
> This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.
And a shrewd move by FB: to not announce their new registry on the same day.
> Yarn pulls packages from registry.yarnpkg.com, which allows them to run experiments with the Yarn client. This is a proxy that pulls packages from the official npm registry, much like npmjs.cf.[0]
Time will tell whether they only want to be proxying NPM or will allow direct pushing to their own registry. If they do, JS ecosystem might see another big shift.
That big shift will have to happen first. I don't see them ever making their own registry unless 99.99% of people are using yarn and are having a lot of problems with the current npm registry. While I see a lot of people using yarn, I'm not sure about 99.99% and I think npm's registry itself is pretty good.
So I don't think interests will ever align to create a new registry. Nobody wants to do that. That would have serious consquences for the JS community and would take years to recover, in my opinion.
Why would it be a bad thing to support additional repositories? Personally, I don't like how centralized the JS ecosystem is.
For example, if I refer to 'left-pad', it would default to 'npmjs.org/left-pad'. If the author goes rougue, I think it would be great to enable people to publish and consume 'thirdparty.com/left-pad'
Disclosure: I'm a FB employee with no knowledge of our plans in this regard
1. any left-pad issue has been "eliminated" with the new rules npm (the company) has enforced.
2. you can already have your own version of thirdparty/left-pad by maintaining your own npm repository. i think what your parent post is referring to is facebook saying we're going to make our own public node/javascript package registry and you should publish to our registry.
doing this at the moment does nothing for the community other causing a lot of pain points. ex) now npm authors will have to publish to both registries so developers don't have to dig to find where it was published to, then they also have to hope that someone else didn't register the module name in one of the registries..
there is just too much splintering if facebook decided to become a competing registry rather than just using npm's registry and building on top of it.
I used to find such comments funny until I began working for Big Name Corps myself and I realized how much one's personal philosophy could be inconsistent or sometimes even contradictory to the employer's philosophy. I make money by selling my skills to an employer despite inconsistent philosophies. I think it's like a chef that can cook meat for his/her guests although the chef has decided to refrain from consuming meat himself/herself.
> So I don't think interests will ever align to create a new registry.
Possibly not soon, at least from a technical perspective, but I could definitely see a PR fiasco (security, bias, etc.) causing a loss of confidence in its stewardship.
And I don't think it'd be that big of a disruption if it were to happen; for the hypothetical case of Facebook+Yarn, they're already proxying to NPM, so they could easily continue to do so while also accepting packages that are Yarn-only.
Even if they don't get 99% of users (or any major percentage) I think it would still benefit the community to have an alternative to npm.
Also Facebook has the advantage of not needing to make a business out of it (they already got a pretty good one) so in theory it could be entirely open source and free.
Just to expand on the stricter versioning rules that I mentioned, some things that in my opinion could improve the reliability of a package repository:
- strictly defined version update time intervals, e.g. you can't update your package more than once a week (or have to take some special actions for critical updates, e.g. contact support)
- "delayed" publishing, e.g. when you submit your package it will only be published in 24 hours, until then you can re-submit updates, etc.
- similar to above, but your package wont be published until it was tagged on github (or elsewhere) for a certain amount of time
- published packages can not be removed, but you can disassociate them with your account by marking them as "not maintained" and possibly assign new maintainers for it
- maybe introduce some way for developers to mark new versions as "backwards incompatible" if they do break backwards compatibility
I think there is definitely a "market" for some stricter node package repo.
> I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once
This is due to the way that Ruby includes code, where it's available globally, vs Node where code is scoped to the including module. I'm not sure how Ruby could support multiple versions with changes to the language and/or RubyGems
Why do you think that everybody is inspired from javascript tech? It is probably the other way around. For example there were very good build tools long before npm which are still used today and unmeasurably better (like Maven or Gradle).
actualyl at this point, unless there is a legal request or a serious security issue, we don't allow any unpublishes. we encourage transferring the package to the npm user and deprecating it. yay immutable registry!
If by "social" you mean bad decisions on settling a naming dispute, yes it was a social issue. That issue hasn't really been addressed publicly (the official policy still doesn't specify how disputes are decided by npm when no amicable agreement can be reached -- other than what essentially boils down to "we'll do what we feel is right").
The actual disruption however was solved by npm Inc disabling unpublishing (except for some very specific circumstances) and by creating a dummy account unpublished modules are reassigned to to avoid abuse (whereas previously anyone could simply claim unpublished modules and publish new versions).
I think it's more secure because you can deliver to it ("somehow") a global cache of packages.
For e.g. Facebook, I bet they have some online filesystem like NFS or similar, that hosts the global yarn cache for their continuous integration system. This way, their CI system doesn't call out to the cloud for every build.
Lock file contains hashes of packages, so it works like many "trust on first use" systems (e.g. SSH): if you installed leftpad@0.2.3, you can be sure that futher installations of this same package will be exactly the same — if malicious npm server gives you something else, this should be detected and rejected.
Looks like they use SHA-1 hashes, though. Please, let's get rid of SHA-1 already.
>We decided to zip the entire node_modules folder and upload it to an internal CDN so that both engineers and our continuous integration systems could download and extract the files consistently.
Recently I got annoyed how hard it is to use shrinkwrap in npm and started working on a npm wrapper that would make npm as easy to work with as Ruby Bundler by copying its workflow as closely as possible (https://github.com/szimek/bundlerjs). Thankfully, I don't have to develop it anymore ;) Big thanks to all Yarn developers!
wow already one of the features i'm loving in yarn is that it tells you which package is firing warnings about package incompatability.
warning electron-prebuilt-compile > electron-compilers > jade@1.11.0: Jade has been renamed to pug, please install the latest version of pug instead of jade
in npm, that would have just said the part after "jade@1.11.0" which was really vague and didn't really make you want to "fix" it because which npm module do you have to go into? who knows because npm (the package manager) didn't tell you.
Just experienced the same. Loved it! npm was giving vague errors about dependency's dependency. Now it's so much clearer that which package needs to be updated. Also the speed of installation has reduced a lot !
i seriously want npm to stop warning me of outdated deps 3 levels down my hierarchy. so what if some shitty old unmaintained lib we started relying on a few years back use lodash v3.x? i'm not going to worry about it as long as it works.
"The React Native package.json currently lists just 68 dependencies, but after running npm install the node_modules directory contains 121,358 files."
That, to me, is what is wrong with npm. The problem stems from node.js not coming with "batteries included" so there is a proliferation of tiny libraries that do the most trivial things.
As I understand it, the related problem is the poor support in build tools for dead code elimination which encourages people to publish libraries that contain a single function like left-pad.
> That, to me, is what is wrong with npm. The problem stems from node.js not coming with "batteries included" so there is a proliferation of tiny libraries that do the most trivial things.
This is in no way a fault of node.js but of the whole JS standard. People these days use npm modules to run in a browser and ship stuff bundled together with webpack (or other bundlers), so even if nodejs had a proper stdlib, you'd still need to depend on a polyfill so that your stuff works in a browser environment, too.
And due to the fact that even if a sane stdlib would ever be standardized, it would take YEARS of time until it reaches significant market share (looking at you, Android, Safari and IE), so you'd always have to ship a polyfill.
That's kind of a red herring when it comes to NodeJS. Webpack and the 50 other tools you need to run it are in a completely controlled environment. Most of the build tooling is working around issues with NodeJS, and NodeJS doesn't need to wait for anyone to come up with improved APIs around filesystem interactions and build tooling. This is purely build-time stuff.
To make matters worse, each package gets its own node_modules directory. Do a grep for 'async' in your node_modules. Mine contains ~20 instances of it. Some of those, probably, are the exact same version.
Going further down shit creek, each project I work on has its own node_modules. Which duplicates all of this crap, ad nauseam.
Do I really need React and Babel and jQuery and Webpack installed for every goddamn repo? You're not a package manager if you're not managing packages!! You're a glorified wget.
> To make matters worse, each package gets its own node_modules directory. Do a grep for 'async' in your node_modules. Mine contains ~20 instances of it. Some of those, probably, are the exact same version.
Yarn is currently being powered by a service running at https://registry.yarnpkg.com - at least that's what's referenced in the yarn.lock file that's created.
I see no documentation nor code for the service powering this, nor is there a way to tell the command line tool to use a registry at a different address.
Aren't we just replacing the dependency on npm, Inc.'s registry with a new dependency on Facebook Inc's registry with this?
Yes. The client has a few advantages, but the registry is still closed and while with npm there's an option of running a private registry if you pay them, here, there's no option at all.
that registry is a mirror of npm's registry. the mirror contains all package data but does not include any of the permissions or user management, so this tool still depends on npm infra for both that, and as a replication source.
registry.yarnpkg.com is a proxy, it doesn't contain any package data except for what gets cached by Cloudflare. Just a Cloudflare'd "CNAME registry.yarnpkg.com. registry.npmjs.org."
The 'attempts at scaling npm client' section is incredibly cringy even to read. Those are real complex problems that every one who works on a massive scale will face at some point in time.
I wish some kind of "auto-bundle" feature prevented the creation of multiple hundred MB's worth of tests, readmes and docs on disk when something like single "babel.bundle.js" file would do.
Is it really important that one knows that a tool dependency uses left-pad ? What could be the drawbacks of such bundling?
Rather impressed with this with the few tests I've run. Kudos to the creators and can't wait to see this project progress (and to contribute into!) in the near future.
Yarn is particularly great for front-end web apps because of its flat installation mode.
ES6 module imports and HTML Imports both require that dependencies are imported by URL. This means that the only reliable way to import another module is by relative URL, like:
import * as $ from '../jquery/jquery.js';
This requires that packages are installed flat, as siblings. Yarn is going to enable native JS modules and projects like Polymer to use npm instead of Bower. :)
--flat is the cli option, but there's also "flat": true support in package.json. Used at the top level this forces a flat install, but used in a dependency it means that that package requires a flat install and it throws an error if it's not.
This means that HTML imports and ES6 modules can force flat installs and start moving the front-end package ecosystem in that direction.
You can use webpack or JSPM to resolve directories at the top level. That way you can just import "web/X" even if the web/ directory isn't a sibling to your current file.
It's not very helpful if you are distributing an NPM package, but for people who aren't authoring libraries, its a godsend.
That was basically what I was suggesting. I'm not sure what the other option really is. If you're working within the browser (implied by this being a frontend), then you have to use a build tool since javascript modules/imports haven't landed in any browser yet.
> If you're working within the browser (implied by this being a frontend), then you have to use a build tool since javascript modules/imports haven't landed in any browser yet.
For now, but native JS module loading is right around the corner, and at that point I don't want to have a required tool to load my modules.
Currently I use HTML Imports, which don't require tools because they use relative URLs. It works really well, so I hope to use that pattern with JS modules.
485 comments
[ 0.20 ms ] story [ 284 ms ] threadPractical, usable, deterministic lockfiles are wonderful and the lack of them has been the single biggest pain point in the npm ecosystem.
One interesting little tidbit I found from diving into the source:
It's not mentioned in the post, but looks like they're running their own registry as well...Oh! And open governance: https://github.com/yarnpkg/rfcs !
npm, Inc strongly encourages people to build and use mirrors. for example, cloudflare (where the mirror lives, and where seb used to work) has been running https://npmjs.cf/ for a long time.
Nix and ied (which borrowed from Nix) got these problems pretty much solved.
I don't understand what spoke against these approaches?
I mean okay, nix has its own language, which is probably a turnoff for JS devs, but ied?
(Disclaimer: I'm the author of ied.)
· http://yehudakatz.com/2016/10/11/im-excited-to-work-on-yarn-...
· https://yarnpkg.com/
Chocolatey is coming soon, once they approve the package.
[+] https://github.com/yarnpkg/yarn/issues/599#issuecomment-2529...
https://github.com/samholmes/yarn
They should mention this at the very beginning. Multiple big players investing in this package manager means that we should maybe inspect a little bit more before chanting xkcd.com/927.
Basically, yarn is npm client done right reusing the same npm package repo.
Stop copying stuff. Just make it global and link.
Next step is to make packages immutable and signed.
I'm happy with this step and the fact that facebook will be able to push this.
I'm a huge fan of bundler - it's dependency heaven. I'm also a big fan of the rubygems repo. It does not allow changes in released versions.
Even without symlinks it's a much needed improvement in the javascript ecosystem
The reason is that each package only finds its dependencies relative to its own location. So your second level dependencies cannot vary from project to project unless you do copying.
(Example: AppA and AppB depend on LibX. LibX depends on LibY. Through their deterministic lockfiles, AppA and AppB disagree on which version of LibY to use. There is no way to symlink things together to satisfy that case without copying LibX or altering node's package resolution algorithm.)
Looks like it's new in Node 6. Yarn supports Node LTS.
TLDR:
- open, community governance that will support long-term evolution
- the technical details get a lot right out of the gate (decent performance, predictability, and security)
npm chooses the first version of a package it encounters to install at the top level. Every other version is installed nested. If you have N uses of version A and one use of version B, but version B is installed first, then you get N copies of the package at version A.
https://github.com/alexanderGugel/ied
"$(npm bin)/bin-cmd" is getting really old
"$(npm bin)/bin-cmd" is getting really old
Absolute and utter hogwash.
I don't even need to get into versioning to make this argument but I'll exclude it for clarity's sake. Simple scenario:
What folder structure is created? Symlink that all you want, how is it in any way whatsoever thoughtful and elegant? It's not a real solution by any stretch of the imagination. Myriads of pre-existing systems that correctly deal with this scenario existed at the time that NPM was concoted. Ignored. Considering that node is touted as a cross-platform solution, how well does version@3 deal with filesystems that don't support symlinks? I've run into this: not at all.NPM version@3 could not be further from the ethos of "engineering." It's an outright hack, and it quickly betrays its limitations. It could have been far simpler, far more considerate of pre-existing wisdom.
Just tried using npm3 explicitly and I stand corrected.
http://yehudakatz.com/2016/10/11/im-excited-to-work-on-yarn-...
(I don't work at Tilde so I can't tell you if it's been shipping with yarn, but it would shock me if it wasn't.)
Are there any branches of ember-cli implementing yarn?
It is the same pattern.
The official npm cli is quite old and evolved along with all the different coding styles and architecture choices of the node and JS ecosystems.
It's also full of dependencies on originally purpose-built modules that suffer from the same problems. This compounds various issues, resulting in long-standing bugs like `npm publish` sometimes not actually including all files in the tar bundle it generates[0].
NPM 3 was a partial rewrite but it's still built on the same mountain of code as NPM 2 and 1. npm Inc developers have in the past complained about not being able to address issues for a lack of resources despite having a ton of developers on the team[1].
The official npm client also doubles as an account management CLI to the npm registry. Which apparently npm Inc now considers a design flaw as they recently started creating a standalone tool for managing other aspects of their commercial services[2].
And finally it's important to understand that having a new package manager that is not tied into npm Inc means it's easier to replace the npm registry as the official node module registry in the future. Which would allow the community to get rid of the privileged position npm Inc currently holds in the node distribution (which is otherwise governed by the Node Foundation).
Rewriting the client from scratch actually turned out to be the sanest approach in this situation, I think.
[0]: https://github.com/npm/npm/issues/5082
[1]: https://github.com/npm/humans
[2]: https://npmjs.org/package/wombat
fwiw, only 2 of those humans work on the CLI, (3 if you count their manager, who spends a lot of time managing). other people work in sales, marketing, engineering on the website, engineering the registry, community + docs, etc.
wombat is not a tool for managing our other commercial services! it's a way to manage our webhooks product, one of many products we have, though very certainly the smallest.
yarn depends on the npm registry, which depends on npm, Inc making money to keep it running.
Thanks for clarifying. But that does show that having a company fund two full-time developers to work on the CLI wouldn't have been sufficient. Yarn was mostly built by Seb (403 commits) starting in late January and Konstantin Raev (121 commits) with contributions from several others (<30 commits each). The commit graph[0] actually shows development mostly paused in April and May, putting the total development time closer to half a year.
This most likely includes collaboration with the other people supporting the project (both Seb and Konstantin work for FB) and a lot of architectural planning -- aspects, which likely would have required an even greater investment of time if building on the legacy codebase of the official npm client (not to mention having to negotiate all changes with npm Inc directly).
> wombat is not a tool for managing our other commercial services!
Sorry about the misunderstanding. When wombat came out a lot of people wondered why wombat was a standalone tool when practically every other aspect is part of the npm cli and from what I gathered from npm devs at the time the decision was more of a technical one (different people working on different tools for different things) than a managerial one (different products deserving different tools).
> yarn depends on the npm registry, which depends on npm, Inc making money to keep it running.
Yarn currently uses a downstream mirror of npm as the direct source of truth, with support for private registries being actively worked on to support the private parts of the npm registry and products like npm enterprise or free alternatives like sinopia.
However that is obviously an implementation detail. Decoupling yarn from the npm registry by going through an intermediary allows experimentation (as the npm blog post points out) and ultimately allows avoiding a hard dependency on npm Inc as a company.
If npm Inc were to shut down right now, yarn would still have a read-only mirror. I'm sure Facebook and Google have enough experience solving problems at scale that they could build a public registry alternative if necessary (such as if npm Inc went out of business tomorrow).
I'm not saying yarn will kill npm Inc. But yarn can kill the npm client and replace it with a community-supported alternative (under an open governance model similar to Ember), which in turn makes it a lot easier to fade out the reliance on the public npm registry maintained by npm Inc -- benefiting both the community (by eliminating a single point of commercial failure in the node ecosystem -- exactly like joyent was to the node project) as well as npm Inc (by eliminating a major drag on development resources and cost center).
The only drawback from such a move to npm Inc would be the loss of its privileged position in the ecosystem, the "first mover" advantage of being the place to go if you want private packages (because it's more frictionless than any of the alternatives -- exactly like GitHub in the git hosting space).
[0]: https://github.com/yarnpkg/yarn/graphs/contributors
Nearly all modern JavaScript build tools are built on Node.js and NPM pre-dates most of them.
The only other thing I can think of that has seen as many shifts in JS development practices as NPM is Express (*2009) and that library has far smaller complexity and has arrived at a drastic reduction in features and a far tighter scope.
The NPM client on the other hand was originally cobbled together by Isaac and then grew organically as the NPM registry massively grew over the months and years of node's success.
The NPM client and registry are the epitome of technical debt. Calling it "old" is the most charitable way to phrase this.
I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once. npm solved this problem elegantly, but still struggles with non-determinism. I had resigned myself to thinking that maybe these were just fundamental tradeoffs in package management that could never be resolved.
Then I had the opportunity to use Cargo, the package manager for Rust. It synthesized what, in my mind, are the best features of npm and Bundler into a single package manager, with terrific speed to boot.
Yarn looks like it will be Cargo for JavaScript. After Cargo improved on many of the great ideas of npm, those ideas are returning to help improve the JavaScript ecosystem. Cross-pollination at its best.
This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.
I'm also happy to see that a significant amount of work has already gone into the governance model. Despite the largest contributors being Facebook employees, it looks like they've really outdone themselves making sure this is a community-run project. A BSD license, no PATENTS file, and an Ember/Rust RFC process[1]; this has all the hallmarks of a community open source project. That's critical for open infrastructure projects like this, and they've nailed it.
[1]: https://github.com/yarnpkg/rfcs
I'm very much looking forward to using Yarn in my own projects because it looks like it solves a lot of real problems I encounter every day. Thanks for all the hard work!
But at least it's a move away from NPM. I think the most problems I had with JavaScript develompent in the last 2 years came from NPM.
According to http://blog.npmjs.org/post/151660845210/hello-yarn, it seems it doesn't work with private packages yet, which may or may not be an issue for your project. But it seems this is a complete CLI replacement for NPM.
I personally don't know much about either tool (don't do a ton of JS), but it's possible that fixing the existing client without either breaking backward compatibility or making it too complicated (multiple modes of operation) was too difficult or not worth it.
Also, I'm having a really hard time understanding the complaint about a new client. The value is in repository of reusable code, not the client. That you can use different clients with the same repository is a feature, not a bug.
These issues have been raised a few times, along with the shrinkwrap/reproducability stuff, and it didnt seem like it was a big priority for the core team. Understandably I guess they seem more focussed on the enterprise/private repo side of things and just keeping things running on the client side.
npmjs.com = package repo which Yarn can use.
at least as far as I can tell
Also, npm isn't deterministic and it got even worse with v3. Sometimes you get a flat list of libs, if a lib is used with multiple versions, the first usage will get installed flat, the rest in the directory of the parent lib, etc.
The npm-cli is basically a mess :\",
What we do currently is we lock everything to an explicit version - even libraries.
At least it's possible to get deterministic builds if you are willing to do a bit of work carefully / manually updating all of your dependencies at once.
This means your library tests will not be deterministic.
Using the Cargo.lock file for libraries does solve this, but then every binary package you build that references your library will have to be specifically tied to the versions in the library.
We do this internally because it's the only way to provide deterministic builds over time.
Over time I suspect it will get harder and harder to keep this rigid web of dependencies working.
One thing we might try is to enforce the rule 'no library will contain any tests'. At first glance this kinda makes my skin crawl, but maybe if we could find a way where every project that used a library could somehow share the tests for that library it could actually work.
git submodules might actually be able to provide these tests for each binary package. If only git submodules would stop corrupting our git repos... :-(
Anyways you can put a lockfile with your library, and it shouldn't affect downstream.
this obviously doesn't fix anything and I think the points in this discussion stand, but I've never understood why the defaults are not more conservative in this regard.
JS dev has been a minefield like that, the entire ecosystem reminds me of C++ except lower barrier to entry means a lot of noise from newbies on all channels (eg. even popular packages can be utter crap so you can't use popularity as a meaningful metric)
You can do the same kind of version range tricks in typical Java builds, for example (Maven), but most people hardcode the values to keep builds as deterministic as possible.
For some reason, the JS community seems to prefer just trusting that new versions won't break anything. Its either very brave of them really (or maybe just foolish).
I don't think Yarn solves any of these problems, tbh. It seems like what we really need is a package manager that tests the api of each package update to make sure nothing in it has broken expectation in accordance with Semver.
Let's not pretend that we aren't all blindly tossing in random libs of dubious quality and origin we find on github into our package.json and hoping for the best anyway. My company talks a mean talk about "best practices", but, my god, the dependencies we use are a true horror show.
Say that I use yarn to depend upon some module X, specifying a fixed version number like the good boy scout that I am. Module X resides on npmjs and in turn depends upon the bleeding edge version of module Y. And then one day module Y goes all Galaxy Note and bursts into flames.
Can yarn shield my app from that?
You can do (and are supposed to do) the same with npm's own shrinkwrap, but people claim that it doesn't work as intended.
It's not that npm devs were naive enough to believe that unpinned deps would be safe for reproducible builds.
However I've heard several people allude over the years that 'npm shrinkwrap' is buggy, and isn't fully reproducible (though never experienced any problems personally). This is the aspect yarn claims to address, along with a supposedly more succinct lockfile format.
From a build tool perspective (we use npm scripts for basically everything [and some webpack]) I am also not missing something particular.
Looking at other comparable solutions (from other languages) I'd say npm does a pretty good job.
but it still far from perfect.
And if you want to lock versions for your entire dependency tree, npm shrinkwrap is what you're looking for (It's essentially the same as lockfiles in other development package managers). Though for security reasons I prefer to keep things locked to a major version only (e.g. "^2.0.0"). Shrinkwrapping is useful in this instance too if you need to have predictable builds (and installs as it'll use the shrinkwrap when installing dependencies too if it's a library rather than the main app) but want to ensure your dependencies stay up to date.
It's not perfect by any measure, but there are ways to make it work the way you want.
The problem isn't really fundamental. Bundler makes almost all the right choices already. Its major disadvantage is that it only works for Ruby.
What it _won't_ let you do is pass a v1.0 type to something that requires a v2.0 type. This will result in a compilation error.
There's been some discussion around increasing Cargo's capacity in this regard (being able to say that a dependency is purely internal vs not), we'll see.
That alone has me super excited about Yarn, before even getting into the performance, security, and 'no invisible dependency' wins.
Yarn does offer a `--flat` option that enforces the highlander rule, and I'm hopeful that the existence of this option will nudge the ecosystem towards an appreciation for fewer semver-major bumps and more ecosystem-wide effort to enable whole apps to work with `--flat` mode.
Plz send halp!
So go crazy and install every version of lodash! Nothing will break.
Edit: I haven't dug into this, but it might be possible to use Typescript namespaces to distinguish a and b's versions of x. https://www.typescriptlang.org/docs/handbook/declaration-fil...
If a depends on x v1 and exports this dependency then your application also imports x v1. If b depends on x v2 and exports this dependency too that means your application is transitively importing both x v1 and x v2 which is not possible.
If a or b only use instances of x internally and do not accept instances of x as parameter or return an instance of x they don't have to export their dependency on x.
If either a or b do not export their dependency on x then there is no problem. Your application will only depend only on x v1 or x v2 directly.
There was actually a high profile bug when someone ended up with two versions of react included in the same page: https://github.com/facebook/react/issues/1939.
I'm not a React expert, but I don't see why that situation would only affect multiple React copies with different versions?
It's placing your own release cycle at the whims of your dependencies' release cycles. In the corporate world that would not be a viable solution.
For instance, when using gemified javascript libraries with bundler it is painful to upgrade to a different version of a javascript library for the user facing site while letting the backend admin interface continue using an older one for compatibility with other dependencies.
If so, different major versions of the same dep should be considered different libraries, for the sake of flattening. Consider lodash for example.
Personally I'm not really sure I like it. If I specify an exact revision of something, chances are I really do mean to install that exact revision. I don't see why I need an extra flag for that.
* LA: handles linear algebra and defines a matrix object.
* A: reads in a csv file and generates a matrix object using LA
* B: takes in a matrix object from LA, and does some operations on it
In this case, if B depends on version 5 of LA and the new version of A depends on version 6 of LA, then there's going to be a problem passing an object that A generated from version 6 and passing it to B which depends on version 5.
* Figure out early on (before 1.0) what your base interface will be.
For example, for a promise library, that would be `then` as specified by Promises/A+
* Check if the argument is an instance of the exact same version.
This works well enough if you use `instanceof`, since classes defined in a different copy of the module will have their own class value - a different unique object.
Its not easy, but its not always necessary either. Most JS libraries don't need to interoperate with objects from previous versions of themselves.However, since A exposes outside methods that take a matrix as an argument, it should not assume anything beyond the core interface and should use LA-6's cast() to convert the matrix.
The problem is partially alleviated when using TypeScript. In that case the inferred type for A demands a structure containing the `normalize` method, which TypeScript will report as incompatible with the passed LA-5 matrix (at compile time). That makes it clearer that `cast` would need to be used.
Another solution is to provide version overrides and make B depend on version 6.
However if there are differences in the matrix class between different versions of the library then you're forced to write a compatibility layer in any case.
And a shrewd move by FB: to not announce their new registry on the same day.
Time will tell whether they only want to be proxying NPM or will allow direct pushing to their own registry. If they do, JS ecosystem might see another big shift.
[0] http://blog.npmjs.org/post/151660845210/hello-yarn
So I don't think interests will ever align to create a new registry. Nobody wants to do that. That would have serious consquences for the JS community and would take years to recover, in my opinion.
For example, if I refer to 'left-pad', it would default to 'npmjs.org/left-pad'. If the author goes rougue, I think it would be great to enable people to publish and consume 'thirdparty.com/left-pad'
Disclosure: I'm a FB employee with no knowledge of our plans in this regard
2. you can already have your own version of thirdparty/left-pad by maintaining your own npm repository. i think what your parent post is referring to is facebook saying we're going to make our own public node/javascript package registry and you should publish to our registry.
doing this at the moment does nothing for the community other causing a lot of pain points. ex) now npm authors will have to publish to both registries so developers don't have to dig to find where it was published to, then they also have to hope that someone else didn't register the module name in one of the registries..
there is just too much splintering if facebook decided to become a competing registry rather than just using npm's registry and building on top of it.
Only to resurface again: http://status.npmjs.org/incidents/dw8cr1lwxkcr
And it will, no doubt, resurface again and again and again
Possibly not soon, at least from a technical perspective, but I could definitely see a PR fiasco (security, bias, etc.) causing a loss of confidence in its stewardship.
And I don't think it'd be that big of a disruption if it were to happen; for the hypothetical case of Facebook+Yarn, they're already proxying to NPM, so they could easily continue to do so while also accepting packages that are Yarn-only.
- strictly defined version update time intervals, e.g. you can't update your package more than once a week (or have to take some special actions for critical updates, e.g. contact support)
- "delayed" publishing, e.g. when you submit your package it will only be published in 24 hours, until then you can re-submit updates, etc.
- similar to above, but your package wont be published until it was tagged on github (or elsewhere) for a certain amount of time
- published packages can not be removed, but you can disassociate them with your account by marking them as "not maintained" and possibly assign new maintainers for it
- maybe introduce some way for developers to mark new versions as "backwards incompatible" if they do break backwards compatibility
I think there is definitely a "market" for some stricter node package repo.
This is due to the way that Ruby includes code, where it's available globally, vs Node where code is scoped to the including module. I'm not sure how Ruby could support multiple versions with changes to the language and/or RubyGems
http://blog.npmjs.org/post/141905368000/changes-to-npms-unpu...
The actual disruption however was solved by npm Inc disabling unpublishing (except for some very specific circumstances) and by creating a dummy account unpublished modules are reassigned to to avoid abuse (whereas previously anyone could simply claim unpublished modules and publish new versions).
No, I mean someone retroactively removing a package others were using out of spite.
For e.g. Facebook, I bet they have some online filesystem like NFS or similar, that hosts the global yarn cache for their continuous integration system. This way, their CI system doesn't call out to the cloud for every build.
Looks like they use SHA-1 hashes, though. Please, let's get rid of SHA-1 already.
why not mirror npm modules there then?!
warning electron-prebuilt-compile > electron-compilers > jade@1.11.0: Jade has been renamed to pug, please install the latest version of pug instead of jade
in npm, that would have just said the part after "jade@1.11.0" which was really vague and didn't really make you want to "fix" it because which npm module do you have to go into? who knows because npm (the package manager) didn't tell you.
That, to me, is what is wrong with npm. The problem stems from node.js not coming with "batteries included" so there is a proliferation of tiny libraries that do the most trivial things.
One problem is that packages often bundle everything, rather than including an npmignore. I'm guilty of this myself.
Do you think that the problem of over-dependence is reversible, and if so, how?
This is in no way a fault of node.js but of the whole JS standard. People these days use npm modules to run in a browser and ship stuff bundled together with webpack (or other bundlers), so even if nodejs had a proper stdlib, you'd still need to depend on a polyfill so that your stuff works in a browser environment, too.
And due to the fact that even if a sane stdlib would ever be standardized, it would take YEARS of time until it reaches significant market share (looking at you, Android, Safari and IE), so you'd always have to ship a polyfill.
Going further down shit creek, each project I work on has its own node_modules. Which duplicates all of this crap, ad nauseam.
Do I really need React and Babel and jQuery and Webpack installed for every goddamn repo? You're not a package manager if you're not managing packages!! You're a glorified wget.
Upgrade to npm v3, that should these problems.
I see no documentation nor code for the service powering this, nor is there a way to tell the command line tool to use a registry at a different address.
Aren't we just replacing the dependency on npm, Inc.'s registry with a new dependency on Facebook Inc's registry with this?
Yes. The client has a few advantages, but the registry is still closed and while with npm there's an option of running a private registry if you pay them, here, there's no option at all.
Is it really important that one knows that a tool dependency uses left-pad ? What could be the drawbacks of such bundling?
ES6 module imports and HTML Imports both require that dependencies are imported by URL. This means that the only reliable way to import another module is by relative URL, like:
This requires that packages are installed flat, as siblings. Yarn is going to enable native JS modules and projects like Polymer to use npm instead of Bower. :)This means that HTML imports and ES6 modules can force flat installs and start moving the front-end package ecosystem in that direction.
It's not very helpful if you are distributing an NPM package, but for people who aren't authoring libraries, its a godsend.
Even so, that requires a build tool. I want to be able to load working sources directly out of my packages directory.
Not if you use a build tool for your frontend.
> Even so, that requires a build tool.
That was basically what I was suggesting. I'm not sure what the other option really is. If you're working within the browser (implied by this being a frontend), then you have to use a build tool since javascript modules/imports haven't landed in any browser yet.
For now, but native JS module loading is right around the corner, and at that point I don't want to have a required tool to load my modules.
Currently I use HTML Imports, which don't require tools because they use relative URLs. It works really well, so I hope to use that pattern with JS modules.