Something like Haskell Stack fixes a lot of problems with versions but it is not suitable for all situations. It is a lot of work to produce a curated set of packages that work. But most of the time things just work so people work around it the rest of the time.
Stackage puts considerable effort on the whole community to ensure that packages remain compatible, and forces package publishers (developers) to update their packages in a timely manner (otherwise the package will be dropped from the snapshot). That works in the Haskell community, but I'd be hard pressed to believe any javascript developer who has a package on npm.org would put the same amount of effort into keeping it up to date.
Package manager is stack and the online repo is hosted at stackage. You manage packages from stackage using stack. You probably can't do it in a larger community like JS or Python because it is too big. Haskell is at the right size to do such things.
Not all Haskell packages (those in hackage.org) are in the stackage snapshots. Stackage is opt-in, there must be a maintainer willing to resolve the occasional build failures.
That means there is no such thing as 'too big', because the number of all existing packages in a ecosystem doesn't matter. It's the commitment that counts, and that's what I don't see in the JS community.
Such curated sets make sense for "core functionality", like text handling, streaming, encoding/decoding, etc., which are used by many 'mainstream' applications (Web backends, console programs, etc.). Yet they raise the barrier to entry by placing a larger burden on developers.
For example, there's a long tail of one-off packages on Hackage that are really valuable to keep around, even if they're not actively maintained and bumping versions would cause breaking changes. I'm thinking of all the various logic frameworks, theorem provers, etc. which are out there; maybe to accompany some paper from the 90s. They're certainly not "core functionality", but every now and then it may turn out that one of them is exactly what is needed in some app/library. Every few years the major repos may drop the last working version of GHC, so someone may have to step up and do the required fixes, but that's few and far between.
These days I use a combination of Hackage (all versions of all packages), Cabal (to solve dependencies), tinc (to make Cabal deterministic) and Nix (to cache everything).
That’s not right. Ranges are powerful and important, but you need a locking mechanism too otherwise you‘ll loose control. Npm and Yarn got finally right and dependency management in the npm ecosystem is now a breeze.
One reason is that many developers work with package managers / languages that are INCAPABLE of handling multiple versions of the same dependency. Without this you quickly get version deadlocks without semver.
Another reason is that many developers are obsessing on getting the latest version of their dependencies for fear of security issues or just missing out on the latest and greatest - and they often completely ignore retesting the application since they now have someone to blame if it fails (that other developer should not have pushed the breaking change with a minor version bump!)
I agree with you that it should be the standard to have fixed versions and update your dependencies at a time of your choosing so that everything can get tested properly - but it seems to be an uphill battle.
> many developers are obsessing on getting the latest version of their dependencies for fear of security issues
Getting the latest version is how you get new vulnerabilities.
Various software distributors, including some Linux distros let software bake in for this reason and can be even faster than the upstreams in developing and applying patches to known vulnerabilities.
Also, unfixed but known vulnerabilities are less dangerous: security and system engineers can work around them, also IDS/IPS can detect and often block attacks.
Very nice point. If you look at all npm examples in the article it shows the >= M.m.b pattern which is never found in the wild. You’ll find ~M.m.b for VERY old packages and ^M.m.b for modern ones (last 4 years). Those let the minor and build increase, but not the major.
Please note: I can speak only for the Node.js npm/Yarn ecosystem.
This article is terribly outdated. Version ranges are great and the only way to manage correctly your dependencies without having tons of duplicates. Also they are a security guarantee, because with them you can get security fixes “automatically”.
Yet there have been problems, that’s why both Yarn and npm@5 implement a lock file which gives you all the guarantees of a range-less package management with the power of a range-based dependency tree.
If module developers following this (IMHO not lucid) article start bundling specific versions in their npm packages everyone will suffer. It’s as stupid as it get.
Please, do your research before calling out on practices.
node+npm5/yarn, ruby+gemfile, rust+cargo (and maybe a few more) are the best examples for the whole workflow around packaging.
Semver ranges (dependency 1.0.x) along with version pinning (lock files that represent the currently installed versions) are the best-case scenario. One can avoid the exercise of "bumping versions" every once in a while (just upgrade all packages within supported semver ranges), get the same semver compatibility guarantees, makes sure all other team members are using same versions, and most importantly the CI process is reproduceable and can be sped up with reliable, non-broken dependency caching.
The elephant in the room is Java. Despite its maturity, both mvn and gradle have poor support for ranges and pretty much no concept of lock files. Everyone is left to "bumping versions" by hand all the time, and snapshot versions are pretty much hell and lead to irreproduceable builds.
Vendoring the whole tree is only an optional add-on to semver ranges. You can't vendor a whole tree and resolve all nested dependencies if everyone is using directly pinned versions! I think the article might have a fundamental misunderstanding between versioning as a library author and as a user.
TL;DR - Semver ranges are great when coupled with version pinning and lock files. If not, there are some trade-offs, and as long as people are aware of the trade-offs, its still better than no ranges.
Snapshot versions are great. We wish npm would have them. Only use them for packages you are developing yourself, of course, and never use them outside of development.
npm does have something close. You can tag versions with a name and a common convention is to use "next" as a tag. What is lacking is that I don't believe tags can be versioned so you can't have a snapshot per version as you can in Java.
The “right way” to do that is to publish alpha versions of your code. Snapshotting the state of the tree into a mutable release is calling for problems. Semver also defines a (poorly supported) M.m.b-release+tstamp pattern, so you can release immutable snapshots and have a semver range do the rest.
Maven-like snapshot are a requirement for an environment with poor support for transient semver-ranged dependencies, which maven is…
> Some justify "~1.8.0" because it includes important security updates automatically. ...
For what it's worth, I too disagree with using strict, static versions. But the author does specifically address this point, so it would be nice to see a rebuttal.
Yes I read it and it calls it a fallacy. Which is not, it’s a trade off.
We could argue about how many developers actually know the implications (which would be a great survey to do), but I believe having this trade off by default has been the right choice for npm/Yarn. Also we need to distinct devs from lead devs, who are the owners of such responsabilities in teams.
"Please don't insinuate that someone hasn't read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."
Getting security fixes automatically is not a good thing.
Too many open source projects fail to ensure non-breaking changes or long term support of multiple release versions.
So either you rely on greatest(>) that has no guarantees to not break and not just patches (~), you try patches and a new release is effectively end of support or you rely on greatest, fail to do a regular audit and realise the dependency is no longer in active development and will receive any updates (which upon typical discovery would require a change of dependency used).
Security isn't just about vulnerability, and I don't like to ask it (because it isn't universal), but is not a risk of system failure often worse than a risk of system vulnerability? It depends somewhat on how critical the system is and defaulting to explicit versions instead of latest is probably a better practice in my opinion.
I've done development using NodeJs and on the JVM and found I had far fewer problems with Java dependency management.
I think that should be qualified with "intentionally". Angular is supposed to be following semver but often has "regressions" in point releases which they fix in a following point release. Usually fairly quickly but it is still not guaranteed.
> Getting security fixes automatically is not a good thing.
Automatically when you request an upgrade to your dependencies. Older npm's defaults were terrible here, but as of npm5, it works more like Bundler/Cargo. You get deterministic installation, and "bundle update" lets you get a tool-assisted upgrade.
Author here. npm5 was released months after this was written. (A breath of fresh after the npm3 had -- look it up -- intentionally non-determinstic versioning based on install history.)
If you read the whole article, I conclude with recommending lockfiles and npm shrinkwrap, the traditional JS lockfile solution. Yarn is the new kid and awesome and deserves mention too.
Lockfiles are an appropriate choice for JS because of unique dependency loading. For Java and many other ecosystems, fixed versions are appropriate because they automatically deduplicate.
There are pros and cons. The absolute most important thing is to achieve determinism. Programming is hard enough as it is.
Hi Paul, thank you for your response here, could you please update the article with some references to the new options available? Would be great if included info for all platforms, not only npm/yarn :)
I can only imagine the amount of new version conflicts that this would generate (I've already seen more than I'd like) if we would no longer go "1.8.0 has a feature that we depend on, so 1.8.0 and newer". Every time every dependency gets an update, every dependent package would need updating too (or at least its metadata). Hell no.
Perhaps go for a range + last tested version, e.g. "1.9.7 worked" so when 2.0 comes out and breaks, you know what to use instead.
If you change the output of "ls", you know everything that uses ls will break and you can bump a major version. But if you change e.g. the date format to include a timezone (visible when -l is specified), does such a change really warrant a major version bump? It's just a minor change but it might still break things. Or maybe if you refactor some stuff in curl, the curl-dev package might suddenly be incompatible with something that depended on it.
If specific version are used then you can end up with multiple versions of the same dependency which can cause additional problems if they're passed around. In compiled languages you end up with errors like "can't assign typeA to typeA because they are different types", when they are different versions of the same type.
Not sure how node would handle the same type of error but I'm guessing it would involve subtle compatibility issues.
In npm all packages have their own dependencies so specifying a specific version sort of works. In Maven for example the dependency tree is flattened by the package manager so that at runtime there can only be a single version of libA. This is just how Java 8 and below works, you simply cannot import two versions of com.example.Foo at the same time. This will change in Java 9 with Jigsaw though. OSGi was a way to workaround this problem in <= Java 8. npm also has a "flattening" feature to save disk space but this is a completely different thing, an implementation detail that is never visible to the packages themselves.
Just want to point out that Jigsaw/Java 9 will not support multiple versions out-of-the-box. (disclosure: I'm currently authoring Java 9 Modularity for O'Reilly)
Yet another package management "advice" article from the point-of-view of a developer and not a maintainer.
You try maintaining an ecosystem where every dependency constraint is a hard specific version. You'll spend 80% of your time fiddling with bumping versions. Fuck that shit.
It also completely ignores security issues. It's worthless to require a library or package be a specific version number, if that version is behind on security fixes.
Yes, the developer might like a library, middleware, application server or whatever to be a specific version, but that ignores that in a production environment you can be behind on security fixes.
It's completely reasonable to lock your code a specific major release of some dependency, assuming it's still being patched, but minor release should be something your patch management solution just applies.
From an operational point of view, I think the original approach of Go, where HEAD of a dependency was just pulled from the repository of the dependency, is the right way to go. If something break, you fix it, because it needs to be done a some point anyway.
I see this all the time with Java middleware. Some developer specifies that we need JBoss version X.Y.Z and not patch unless they say so. At the same time we're required to patch the operation system with the latest security patches. Well what's the point of that if the only service exposed to the Internet is the only thing not being patched?
Paul Draper from TFA totally ignored the perspective of a packager.
It would be a completely unacceptable situation for application A1 to depend on DB abstraction library v0.8651, A2 on v0.8653, A3 on v0.8670, A4 on v0.87…
How do you ship this to your distro users? Almost always there is no easy way to install several libraries of the same name but different versions in parallel, and it's never supported by the distro auto-packaging tools.
Patching to use the latest version of the dependencies and verifying the patched version works ok is a huge burden on package maintainers. The application author knows the code much better!
The current solution of specifying minimum versions of dependencies exists not because we like the suffering caused by occasional breakage. No, it exists because it is a practical, working solution for the real world that is the best balanced trade-off.
The problem is that nobody really tests all the versions they specify in a dependency list. They just install the dependencies, and if everything works, commit the file.
In PHP we usually have a version range in libraries' dependencies and lock file that specifies exact dependency versions for the entire application. So you can be sure that you get tested combinantion of libraries.
Can you elaborate on what you mean? Composer is the most popular PHP packager and its two core commands - update and install - make updates to and install from the composer.lock file respectively
> Almost always there is no easy way to install several libraries of the same name but different versions in parallel.
How is this possible is beyond my comprehension. Someone, anyone break something and I have no standard way to bypass "dll hell" (putting space problems aside)? what the actual f.
PHP's package manager (composer) has a lock file that contains exact versions of packages and can be commited to the filesystem. So we don't have problems with version ranges.
I feel like this is describing a solved problem, and what's more, the author knows the solution. Lock files eliminate the issue of mysterious build failures and allow almost trivial creation of automated canary builds to detect problems early.
This is right: unless you use exact versions, ideally vendored or cached in a local package server, you do not have even basic reproducibility.
However.
The reason people use ranges is because they don't want to handle the administrative burden of tracking upstream changes and updating their software.
The OWASP Top 10 (2017 draft) shows that using known-vulnerable components is a common security weakness in software. Nobody sets out to do this, but the flipside of having access to massive troves of dependencies is that you have dozens, hundreds, perhaps thousands of dependencies you might not know about.
I've worked on Cloud Foundry Buildpacks. A large amount of effort has gone into exactly this problem: tracking upstream versions, pulling new ones immediately, building them and testing them. But it's at varying levels of sophistication.
For some dependencies, you get structured data. NodeJS publish an index.json file[0] which is fairly trivial to keep up with. But for other sources, which I will leave unnamed, it's necessary to do flaky, unreliable HTML parsing to detect new releases. Data about what's been found so far is kept in a ci-robots[1] repo to ensure atomicity and simplicity (I'd prefer a Concourse resource that emits versions one by one, but that's just me).
What's needed is a uniform way to describe available versions of software packages. Uniform across all the major tributaries of versions: package managers, distributions, the works. If it becomes possible, as Buildpacks have after a great deal of effort, to automate fetching and testing new dependencies, then the need for ranges vanishes. You can rely on your CI/CD to run each dependency change through its paces and update the firmly fixed list for you.
there's one solution in the article that's valid: version lock files. these are a must have for any reasonable deployment strategy.
now when you talk about actually developing the software, this is a completely different set of requirements and that one is best served with very coarse grained (but safe) version ranges. different tool for a different use case.
It's sad to see many people quoting npm as a good example while version ranging has been mastered by package managers on Linux and earlier Unix systems for decades.
What I would like to see is support for existing package managers like rpm and deb that could be mirrored from these language repositories (e.g. Python, Ruby, Perl, etc). This would be more difficult for packages that have external language bindings, but a source rpm/deb package could work).
58 comments
[ 2.8 ms ] story [ 118 ms ] threadStackage puts considerable effort on the whole community to ensure that packages remain compatible, and forces package publishers (developers) to update their packages in a timely manner (otherwise the package will be dropped from the snapshot). That works in the Haskell community, but I'd be hard pressed to believe any javascript developer who has a package on npm.org would put the same amount of effort into keeping it up to date.
That means there is no such thing as 'too big', because the number of all existing packages in a ecosystem doesn't matter. It's the commitment that counts, and that's what I don't see in the JS community.
For example, there's a long tail of one-off packages on Hackage that are really valuable to keep around, even if they're not actively maintained and bumping versions would cause breaking changes. I'm thinking of all the various logic frameworks, theorem provers, etc. which are out there; maybe to accompany some paper from the 90s. They're certainly not "core functionality", but every now and then it may turn out that one of them is exactly what is needed in some app/library. Every few years the major repos may drop the last working version of GHC, so someone may have to step up and do the required fixes, but that's few and far between.
These days I use a combination of Hackage (all versions of all packages), Cabal (to solve dependencies), tinc (to make Cabal deterministic) and Nix (to cache everything).
I completely agree with the article. Version ranges are evil. Avoid them at all cost.
Because other people have different priorities and have come to a different cost/benefit ratio value than you?
Another reason is that many developers are obsessing on getting the latest version of their dependencies for fear of security issues or just missing out on the latest and greatest - and they often completely ignore retesting the application since they now have someone to blame if it fails (that other developer should not have pushed the breaking change with a minor version bump!)
I agree with you that it should be the standard to have fixed versions and update your dependencies at a time of your choosing so that everything can get tested properly - but it seems to be an uphill battle.
Getting the latest version is how you get new vulnerabilities.
Various software distributors, including some Linux distros let software bake in for this reason and can be even faster than the upstreams in developing and applying patches to known vulnerabilities.
Also, unfixed but known vulnerabilities are less dangerous: security and system engineers can work around them, also IDS/IPS can detect and often block attacks.
This article is terribly outdated. Version ranges are great and the only way to manage correctly your dependencies without having tons of duplicates. Also they are a security guarantee, because with them you can get security fixes “automatically”.
Yet there have been problems, that’s why both Yarn and npm@5 implement a lock file which gives you all the guarantees of a range-less package management with the power of a range-based dependency tree.
If module developers following this (IMHO not lucid) article start bundling specific versions in their npm packages everyone will suffer. It’s as stupid as it get.
Please, do your research before calling out on practices.
Semver ranges (dependency 1.0.x) along with version pinning (lock files that represent the currently installed versions) are the best-case scenario. One can avoid the exercise of "bumping versions" every once in a while (just upgrade all packages within supported semver ranges), get the same semver compatibility guarantees, makes sure all other team members are using same versions, and most importantly the CI process is reproduceable and can be sped up with reliable, non-broken dependency caching.
The elephant in the room is Java. Despite its maturity, both mvn and gradle have poor support for ranges and pretty much no concept of lock files. Everyone is left to "bumping versions" by hand all the time, and snapshot versions are pretty much hell and lead to irreproduceable builds.
Vendoring the whole tree is only an optional add-on to semver ranges. You can't vendor a whole tree and resolve all nested dependencies if everyone is using directly pinned versions! I think the article might have a fundamental misunderstanding between versioning as a library author and as a user.
TL;DR - Semver ranges are great when coupled with version pinning and lock files. If not, there are some trade-offs, and as long as people are aware of the trade-offs, its still better than no ranges.
Maven-like snapshot are a requirement for an environment with poor support for transient semver-ranged dependencies, which maven is…
Haven't worked with them myself, but colleagues are singing high praise.
> Some justify "~1.8.0" because it includes important security updates automatically. ...
For what it's worth, I too disagree with using strict, static versions. But the author does specifically address this point, so it would be nice to see a rebuttal.
We could argue about how many developers actually know the implications (which would be a great survey to do), but I believe having this trade off by default has been the right choice for npm/Yarn. Also we need to distinct devs from lead devs, who are the owners of such responsabilities in teams.
https://news.ycombinator.com/newsguidelines.html
"Please don't insinuate that someone hasn't read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."
Too many open source projects fail to ensure non-breaking changes or long term support of multiple release versions.
So either you rely on greatest(>) that has no guarantees to not break and not just patches (~), you try patches and a new release is effectively end of support or you rely on greatest, fail to do a regular audit and realise the dependency is no longer in active development and will receive any updates (which upon typical discovery would require a change of dependency used).
Security isn't just about vulnerability, and I don't like to ask it (because it isn't universal), but is not a risk of system failure often worse than a risk of system vulnerability? It depends somewhat on how critical the system is and defaulting to explicit versions instead of latest is probably a better practice in my opinion.
I've done development using NodeJs and on the JVM and found I had far fewer problems with Java dependency management.
Relying on those projects is not a good thing.
Automatically when you request an upgrade to your dependencies. Older npm's defaults were terrible here, but as of npm5, it works more like Bundler/Cargo. You get deterministic installation, and "bundle update" lets you get a tool-assisted upgrade.
If you read the whole article, I conclude with recommending lockfiles and npm shrinkwrap, the traditional JS lockfile solution. Yarn is the new kid and awesome and deserves mention too.
Lockfiles are an appropriate choice for JS because of unique dependency loading. For Java and many other ecosystems, fixed versions are appropriate because they automatically deduplicate.
There are pros and cons. The absolute most important thing is to achieve determinism. Programming is hard enough as it is.
Perhaps go for a range + last tested version, e.g. "1.9.7 worked" so when 2.0 comes out and breaks, you know what to use instead.
If you change the output of "ls", you know everything that uses ls will break and you can bump a major version. But if you change e.g. the date format to include a timezone (visible when -l is specified), does such a change really warrant a major version bump? It's just a minor change but it might still break things. Or maybe if you refactor some stuff in curl, the curl-dev package might suddenly be incompatible with something that depended on it.
Not sure how node would handle the same type of error but I'm guessing it would involve subtle compatibility issues.
You try maintaining an ecosystem where every dependency constraint is a hard specific version. You'll spend 80% of your time fiddling with bumping versions. Fuck that shit.
Yes, the developer might like a library, middleware, application server or whatever to be a specific version, but that ignores that in a production environment you can be behind on security fixes.
It's completely reasonable to lock your code a specific major release of some dependency, assuming it's still being patched, but minor release should be something your patch management solution just applies.
From an operational point of view, I think the original approach of Go, where HEAD of a dependency was just pulled from the repository of the dependency, is the right way to go. If something break, you fix it, because it needs to be done a some point anyway.
I see this all the time with Java middleware. Some developer specifies that we need JBoss version X.Y.Z and not patch unless they say so. At the same time we're required to patch the operation system with the latest security patches. Well what's the point of that if the only service exposed to the Internet is the only thing not being patched?
But version pinning is probably the best way at the moment.
This way the maintainers can use ranges and the developers can pin what they really need in the end-product.
There are more developers than maintainers, so this scales much better.
Wrong comparison. The majority of software is built, deployed, maintained and used by entirely different organizations over years.
Companies doing end-to-end CD and running their software only internally are the minority.
This kind of bad practices from few developers turns into countless hours of work to maintain systems years later.
It would be a completely unacceptable situation for application A1 to depend on DB abstraction library v0.8651, A2 on v0.8653, A3 on v0.8670, A4 on v0.87…
How do you ship this to your distro users? Almost always there is no easy way to install several libraries of the same name but different versions in parallel, and it's never supported by the distro auto-packaging tools.
Patching to use the latest version of the dependencies and verifying the patched version works ok is a huge burden on package maintainers. The application author knows the code much better!
The current solution of specifying minimum versions of dependencies exists not because we like the suffering caused by occasional breakage. No, it exists because it is a practical, working solution for the real world that is the best balanced trade-off.
In PHP we usually have a version range in libraries' dependencies and lock file that specifies exact dependency versions for the entire application. So you can be sure that you get tested combinantion of libraries.
Some do. Some others target the versions shipped in a stable Linux distribution.
And even if the upstream don't test some combinations of versions, distributions will test what they ship.
How can they check future versions of libraries that will match the range?
How is this possible is beyond my comprehension. Someone, anyone break something and I have no standard way to bypass "dll hell" (putting space problems aside)? what the actual f.
However.
The reason people use ranges is because they don't want to handle the administrative burden of tracking upstream changes and updating their software.
The OWASP Top 10 (2017 draft) shows that using known-vulnerable components is a common security weakness in software. Nobody sets out to do this, but the flipside of having access to massive troves of dependencies is that you have dozens, hundreds, perhaps thousands of dependencies you might not know about.
I've worked on Cloud Foundry Buildpacks. A large amount of effort has gone into exactly this problem: tracking upstream versions, pulling new ones immediately, building them and testing them. But it's at varying levels of sophistication.
For some dependencies, you get structured data. NodeJS publish an index.json file[0] which is fairly trivial to keep up with. But for other sources, which I will leave unnamed, it's necessary to do flaky, unreliable HTML parsing to detect new releases. Data about what's been found so far is kept in a ci-robots[1] repo to ensure atomicity and simplicity (I'd prefer a Concourse resource that emits versions one by one, but that's just me).
What's needed is a uniform way to describe available versions of software packages. Uniform across all the major tributaries of versions: package managers, distributions, the works. If it becomes possible, as Buildpacks have after a great deal of effort, to automate fetching and testing new dependencies, then the need for ranges vanishes. You can rely on your CI/CD to run each dependency change through its paces and update the firmly fixed list for you.
[0] https://nodejs.org/dist/index.json
[1] https://github.com/cloudfoundry/public-buildpacks-ci-robots
now when you talk about actually developing the software, this is a completely different set of requirements and that one is best served with very coarse grained (but safe) version ranges. different tool for a different use case.