anyone find a way to get rid of the constantly shifting icons at the bottom of the screen? I'm trying to read and the motion keeps pulling my attention away from the words toward the dancing critters.
NPM has, starting with version 0.5.1, an absolutely lovely feature where it simply ignores the package-lock.json file altogether. Or to be more precise, "npm install" regenerates package-lock.json based on package.json. What's the point of "npm upgrade" then? Eh.
What if your project also uses librupa, which also depends on liblupa? Follow the chain of reasoning from that thought, or maybe spend a couple of decades dealing with the horror created by people who didn't, and you'll get to lockfiles.
> But if you want an existence proof: Maven. The Java library ecosystem has been going strong for 20 years, and during that time not once have we needed a lockfile. And we are pulling hundreds of libraries just to log two lines of text, so it is actively used at scale.
Do not pretend, for even half a second, that dependency resolution is not hell in maven (though I do like that packages are namespaced by creators, npm shoulda stolen that).
The author of the article asked the same question in his personal blog in Telegram a year and a half ago [1], and received in response exactly the same example with maven. Probably, to build a personal brand and increase visibility, he needs to ask the same thing many times in different places.
I see lockfiles as something you use for applications you are deploying - if you run something like a web app it's very useful to know exactly what is being deployed to production, make sure it exactly matches staging and development environments, make sure you can audit new upgrades to your dependencies etc.
This article appears to be talking about lockfiles for libraries - and I agree, for libraries you shouldn't be locking exact versions because it will inevitably pay havoc with other dependencies.
Or maybe I'm missing something about the JavaScript ecosystem here? I mainly understand Python.
What if your program depends on library a1.0 and library b1.0, and library a1.0 depends on c2.1 and library b1.0 depends on c2.3? Which one do you install in your executable? Choosing one randomly might break the other library. Installing both _might_ work, unless you need to pass a struct defined in library c from a1.0 to b1.0, in which case a1.0 and b1.0 may expect different memory layouts (even if the public interface for the struct is the exact same between versions).
The reason we have dependency ranges and lockfiles is so that library a1.0 can declare "I need >2.1" and b1.0 can declare "I need >2.3" and when you depend on a1.0 and b1.0, we can do dependency resolution and lock in c2.3 as the dependency for the binary.
we've all learned about things, not understood them, and thought "wow, these people must be idiots. why would they have made this complicated thing? makes no sense whatsoever. I can't believe these people, idiots, never thought this through like I have."
Most of us, fortunately, don't post these thoughts to the internet for anybody to read.
Ok, but what happens when lib-a depends on lib-x:0.1.4 and lib-b depends on lib-x:0.1.5, even though it could have worked with any lib-x:0.1.*? Are these libraries just incompatible now? Lockfiles don't guarantee that new versions are compatible, but it guarantees that if your code works in development, it will work in production (at least in terms of dependencies).
I assume java gets around this by bundling libraries into the deployed .jar file. That this is better than a lock file, but doesn't make sense for scripting languages that don't have a build stage. (You won't have trouble convincing me that every language should have a proper build stage, but you might have trouble convincing the millions of lines of code already written in languages that don't.)
Let's play this out in a compiled language like Cargo.
If every dependency was a `=` and cargo allowed multiple versions of SemVer compatible packages.
The first impact will be that your build will fail. Say you are using `regex` and you are interacting with two libraries that take a `regex::Regex`. All of the versions need to align to pass `Regex` between yourself and your dependencies.
The second impact will be that your builds will be slow. People are already annoyed when there are multiple SemVer incompatible versions of their dependencies in their dependency tree, now it can happen to any of your dependencies and you are working across your dependency tree to get everything aligned.
The third impact is if you, as the application developer, need a security fix in a transitive dependency. You now need to work through the entire bubble up process before it becomes available to you.
Ultimately, lockfiles are about giving the top-level application control over their dependency tree balanced with build times and cross-package interoperability. Similarly, SemVer is a tool any library with transitive dependencies [0]
> All of the versions need to align to pass `Regex` between yourself and your dependencies.
In nominally typed languages, all types have to be nominally the same, yes, but it does not follow that semver compatible packages will not permit passing different versions of each around: https://github.com/dtolnay/semver-trick (the trick is to ensure that different versions have nominally the same type by forward dependency).
Anyways, even with this, cargo has a lock file because you want to run in CI what you ran when you developed the feature instead of getting non-deterministic executions in an automated build + verification. That's what a deterministic build is aiming for. Then next feature you develop you can ignore the lockfile, pull in the new dependencies and move on with life because you don't necessarily care for determinism there: you are willing to fix issues. Or you aren't and you use the lockfile.
There is absolutely a good reason for version ranges: security updates.
When I, the owner of an application, choose a library (libuseful 2.1.1), I think it's fine that the library author uses other libraries (libinsecure 0.2.0).
But in 3 months, libinsecure is discovered (surprise!) to be insecure. So they release libinsecure 0.2.1, because they're good at semver. The libuseful library authors, meanwhile, are on vacation because it's August.
I would like to update. Turns out libinsecure's vulnerability is kind of a big deal. And with fully hardcoded dependencies, I cannot, without some horrible annoying work like forking/building/repackaging libuseful. I'd much rather libuseful depend on libinsecure 0.2.*, even if libinsecure isn't terribly good at semver.
I would love software to be deterministically built. But as long as we have security bugs, the current state is a reasonable compromise.
Lockfiles are essential for somewhat reproducible builds.
If a transient dependency (not directly referenced) updates, this might introduce different behavior. if you test a piece of software and fix some bugs, the next build shouldn't contain completely different versions of dependencies. This might introduce new bugs.
> “But Niki, you can regenerate the lockfile and pull in all the new dependencies!”
> Sure. In exactly the same way you can update your top-level dependencies.
how does updating top-level deps help with updating leaf packages? Is the author assuming that whenever a leaf package is updated, every other package in the dep chain gets immediately new release? That is fundamentally impossible considering that the releases would need to happen serially.
The author seems to miss the point of version ranges. Yes, specific versions of dependencies get frozen in the lock file at the moment of building. But the only way to determine these specific versions is to run version resolution across the whole tree. The process finds out which specific versions within the ranges can be chosen to satisfy all the version constraints.
This works with minimal coordination between authors of the dependencies. It becomes a big deal when you have several unrelated dependencies, each transitively requiring that libpupa. The chance they converge on the same exact version is slim. The chance a satisfying version can be found within specified ranges is much higher.
Physical things that are built from many parts have the very same limitation: they need to specify tolerances to account for the differences in production, and would be unable to be assembled otherwise.
In the world of Python-based end-user libraries the pinned (non-ranged) versions result in users being unable to use your library in an environment with other libraries. I’d love to lock my library to numpy 2.3.4, but if the developers of another library pin theirs to 2.3.5 then game over.
For server-side or other completely controlled environments the only good reason to have lock files is if they are actually hashed and thus allow to confirm security audits. Lock files without hashes do not guarantee security (depending on the package registry, of course, but at least in Python world (damn it) the maintainer can re-publish a package with an existing version but different content).
The author of this piece doesn't understand why a top level project might want control of its dependencies dependencies.
That's the flaw in this whole article, if you can't articulate why it's important to be able to control those... don't write an article. You don't understand the problem space.
Semantic versioning isn't perfect, but it's more than a "hint", and it sure as hell beats having to manually patch (or fork) an entire dependency chain to fix a security problem.
The author is perhaps presenting a good argument for languages/runtimes like JavaScript/Node where dependencies may be isolated and conflicting dependencies may coexist in the dependency tree (e.g., "app -> { libpupa 1.2.3 -> liblupa 0.7.8 }, { libxyz 2.0 -> liblupa 2.4.5 }" would be fine), but the proposed dependency resolution algorithm...
> Our dependency resolution algorithm thus is like this:
> 1. Get the top-level dependency versions
> 2. Look up versions of libraries they depend on
> 3. Look up versions of libraries they depend on
...would fail in languages like Python where dependencies are shared, and the steps 2, 3, etc. would result in conflicting versions.
In these languages, there is good reason to define dependencies in a relaxed way (with constraints that exclude known-bad versions; but without pins to any specific known-to-work version and without constraining only to existing known-good versions) at first. This way dependency resolution always involves some sort of constraint solving (with indeterminate results due to the constraints being open-ended), but then for the sake of reproducibility the result of the constraint solving process may be used as a lockfile. In the Python world this is only done in the final application (the final environment running the code, this may be the test suite in for a pure library) and the pins in the lock aren't published for anyone to reuse.
To reiterate, the originally proposed algorithm doesn't work for languages with shared dependencies. Using version constraints and then lockfiles as a two-layer solution is a common and reasonable way of resolving the dependency topic in these languages.
> Imagine you voluntarily made your build non-reproducible by making them depend on time. If I build my app now, I get libpupa 1.2.3 and liblupa 0.7.8. If I repeat the same build in 10 minutes, I’ll get liblupa 0.7.9. Crazy, right? That would be chaos.
No; in fact it's perfectly reasonable, and at the core of what the author doesn't seem to get. Developers have motivations other than reproducibility. The entire reason we have version number schemes like this is so that we can improve our code while also advertising reasonable expectations about compatibility. If we have dependents, then hopefully this also improves their UX indirectly — whether by taking advantage of optimizations we made, not encountering bugs that were actually our fault, etc. Similarly, if we have dependencies, we can seek to take advantage of that.
Upgrading environments is an opportunity to test new configurations, and see if they're any better than what's in existing lockfiles.
> But this is what version ranges essentially are. Instead of saying “libpupa 1.2.3 depends on liblupa 0.7.8”, they are saying “libpupa 1.2.3 depends on whatever the latest liblupa version is at the time of the build.”
But also, developers aren't necessarily using the latest versions of their dependencies locally anyway. If I did pin a version in my requirements, it'd be the one that I tested the build with, not necessarily the one that was most recently released at the time of the build. Not everyone runs an industrial-strength CI system, and for the size of lots of useful packages out there, they really shouldn't have to, either. (And in the pathological case, someone else could re-release while I'm building and testing!)
> But... why would libpupa’s author write a version range that includes versions that don’t exist yet? How could they know that liblupa 0.7.9, whenever it will be released, will continue to work with libpupa? Surely they can’t see the future? Semantic versioning is a hint, but it has never been a guarantee.
The thing about this is that "work with [a dependency]" is not really a binary. New versions also fix things — again, that's the main reason that new versions get released in the first place. Why would I keep writing the software after it's "done" if I don't think there's anything about it that could be fixed?
For that matter, software packages break for external reasons. If I pin my dependency, and that dependency is, say, a wrapper for a third-party web API, and the company operating that website makes a breaking change to the API, then I just locked myself out of new versions of the dependency that cope with that change.
In practice, there are good reasons to not need a guarantee and accept the kind of risk described. Lockfiles exist for those who do need a guarantee that their local environment will be set in concrete (which has other, implicit risks).
I see it as much like personal finance. Yes, investments beyond a HISA may carry some kind of risk. This is worthwhile for most people. And on the flip side, you also can't predict the future inflation rate, and definitely can't predict what will happen to the price of the individual goods and services you care about most.
> The funny thing is, these version ranges end up not being used anyway. You lock your dependencies once in a lockfile and they stay there, unchanged. You don’t even get the good part!
??? What ecosystem is this author talking about? Generating a lockfile doesn't cause the underlying dependency metadata to disappear. You "get the good part" as a developer by periodically regenerating a lockfile, testing the resulting environment and shipping the new lock. Or as a user by grabbing a new lockfile, or by just choosing not to use provided lockfiles.
> “But Niki, you can regenerate the lockfile and pull in all the new dependencies!” Sure. In exactly the same way you can update your top-lev...
I disagree with this blogpost in its entirety. Lockfiles are neither unnecessary, nor are they complicated. The argument presented against lockfiles boils down to a misrepresentation. I also dislike the presentation using the godawful yellow color and the stupid websocket gadget in the footer.
The entire point of lockfiles is to let the user decide when the version resolution algorithm should execute and when it shouldn't. That's all they do and they do it exactly as promised.
Many comments talk about how top-level and transitive dependencies can conflict. I think the article is suggesting you can resolve those by specifying them in the top-level packages and overriding any transitive package versions. If we are doing that anyways, it circles back to if lock files are necessary.
Given that, I still see some consequences:
The burden for testing if a library can use its dependency falls back on the application developer instead of the library developer. A case could be made that, while library developers should test what their libraries are compatible with, the application developer has the ultimate responsibility for making sure everything can work together.
I also see that there would need to be tooling to automate resolutions. If ranges are retained, the resolver needs to report every conflict and force the developer to explicitly specify the version they want at the top-level. Many package managers automatically pick one and write it into the lock file.
If we don’t have lock files, and we want it to be automatic, then we can have it write to the top level package manager and not the lock file. That creates its own problems.
One of those problems comes from humans and tooling writing to the same configuration file. I have seen problems with that idea pop up — most recently, letting letsencrypt modify nginx configs, and now I have to manually edit those. Letsencrypt can no longer manage them. Arguably, we can also say LLMs can work with that, but I am a pessimist when it comes to LLM capabilities.
So in conclusion, I think the article writer’s reasoning is sound, but incomplete. Humans don’t need lockfiles, but our tooling need lockfiles until it is capable of working with the chaos of human-managed package files.
When discoverability and versioning of libraries is more-or-less standardized (a la Cargo/PyPI/NPM), automated tooling for dependency resolution/freezing follows naturally. The build tools for Java and C historically did not have this luxury, which is why their ecosystems have a reputation for caring a lot about backwards compatibility.
In case the author is reading, I can't read your article because of that animation at the bottom. I get it, it's cute, but it makes it too distracting to concentrate on the article, so I ended up just closing it.
Every time something from this website is shared, this topic comes up. It's one of those things where you can't tell whether it's genius or madness. Either way, I'm pretty sure the design is deliberate(ly obnoxious).
73 comments
[ 2.2 ms ] story [ 84.6 ms ] threadMaven, by default, does not check your transitive dependencies for version conflicts. To do that, you need a frustrating plugin that produces much worse error messages than NPM does: https://ourcraft.wordpress.com/2016/08/22/how-to-read-maven-....
How does Maven resolve dependencies when two libraries pull in different versions? It does something insane. https://maven.apache.org/guides/introduction/introduction-to....
Do not pretend, for even half a second, that dependency resolution is not hell in maven (though I do like that packages are namespaced by creators, npm shoulda stolen that).
1. https://t.me/nikitonsky_pub/597
This article appears to be talking about lockfiles for libraries - and I agree, for libraries you shouldn't be locking exact versions because it will inevitably pay havoc with other dependencies.
Or maybe I'm missing something about the JavaScript ecosystem here? I mainly understand Python.
The reason we have dependency ranges and lockfiles is so that library a1.0 can declare "I need >2.1" and b1.0 can declare "I need >2.3" and when you depend on a1.0 and b1.0, we can do dependency resolution and lock in c2.3 as the dependency for the binary.
Most of us, fortunately, don't post these thoughts to the internet for anybody to read.
I assume java gets around this by bundling libraries into the deployed .jar file. That this is better than a lock file, but doesn't make sense for scripting languages that don't have a build stage. (You won't have trouble convincing me that every language should have a proper build stage, but you might have trouble convincing the millions of lines of code already written in languages that don't.)
If every dependency was a `=` and cargo allowed multiple versions of SemVer compatible packages.
The first impact will be that your build will fail. Say you are using `regex` and you are interacting with two libraries that take a `regex::Regex`. All of the versions need to align to pass `Regex` between yourself and your dependencies.
The second impact will be that your builds will be slow. People are already annoyed when there are multiple SemVer incompatible versions of their dependencies in their dependency tree, now it can happen to any of your dependencies and you are working across your dependency tree to get everything aligned.
The third impact is if you, as the application developer, need a security fix in a transitive dependency. You now need to work through the entire bubble up process before it becomes available to you.
Ultimately, lockfiles are about giving the top-level application control over their dependency tree balanced with build times and cross-package interoperability. Similarly, SemVer is a tool any library with transitive dependencies [0]
[0] https://matklad.github.io/2024/11/23/semver-is-not-about-you...
In nominally typed languages, all types have to be nominally the same, yes, but it does not follow that semver compatible packages will not permit passing different versions of each around: https://github.com/dtolnay/semver-trick (the trick is to ensure that different versions have nominally the same type by forward dependency).
Anyways, even with this, cargo has a lock file because you want to run in CI what you ran when you developed the feature instead of getting non-deterministic executions in an automated build + verification. That's what a deterministic build is aiming for. Then next feature you develop you can ignore the lockfile, pull in the new dependencies and move on with life because you don't necessarily care for determinism there: you are willing to fix issues. Or you aren't and you use the lockfile.
When I, the owner of an application, choose a library (libuseful 2.1.1), I think it's fine that the library author uses other libraries (libinsecure 0.2.0).
But in 3 months, libinsecure is discovered (surprise!) to be insecure. So they release libinsecure 0.2.1, because they're good at semver. The libuseful library authors, meanwhile, are on vacation because it's August.
I would like to update. Turns out libinsecure's vulnerability is kind of a big deal. And with fully hardcoded dependencies, I cannot, without some horrible annoying work like forking/building/repackaging libuseful. I'd much rather libuseful depend on libinsecure 0.2.*, even if libinsecure isn't terribly good at semver.
I would love software to be deterministically built. But as long as we have security bugs, the current state is a reasonable compromise.
.NET doesn't have lock files either, and its dependency tree runs great.
Using fixed versions for dependencies is a best practice, in my opinion.
If a transient dependency (not directly referenced) updates, this might introduce different behavior. if you test a piece of software and fix some bugs, the next build shouldn't contain completely different versions of dependencies. This might introduce new bugs.
> “But Niki, you can regenerate the lockfile and pull in all the new dependencies!”
> Sure. In exactly the same way you can update your top-level dependencies.
how does updating top-level deps help with updating leaf packages? Is the author assuming that whenever a leaf package is updated, every other package in the dep chain gets immediately new release? That is fundamentally impossible considering that the releases would need to happen serially.
This works with minimal coordination between authors of the dependencies. It becomes a big deal when you have several unrelated dependencies, each transitively requiring that libpupa. The chance they converge on the same exact version is slim. The chance a satisfying version can be found within specified ranges is much higher.
Physical things that are built from many parts have the very same limitation: they need to specify tolerances to account for the differences in production, and would be unable to be assembled otherwise.
For server-side or other completely controlled environments the only good reason to have lock files is if they are actually hashed and thus allow to confirm security audits. Lock files without hashes do not guarantee security (depending on the package registry, of course, but at least in Python world (damn it) the maintainer can re-publish a package with an existing version but different content).
The algorithm can be deterministic, but fetching the dependencies of a package is not.
It is usually an HTTP call to some endpoint that might flake out or change its mind.
Lock files were invented to make it either deterministic or fail.
Even with Maven, deterministic builds (such as with Bazel) lock the hashes down.
This article is mistaken.
The author of this piece doesn't understand why a top level project might want control of its dependencies dependencies.
That's the flaw in this whole article, if you can't articulate why it's important to be able to control those... don't write an article. You don't understand the problem space.
Semantic versioning isn't perfect, but it's more than a "hint", and it sure as hell beats having to manually patch (or fork) an entire dependency chain to fix a security problem.
> Our dependency resolution algorithm thus is like this:
> 1. Get the top-level dependency versions
> 2. Look up versions of libraries they depend on
> 3. Look up versions of libraries they depend on
...would fail in languages like Python where dependencies are shared, and the steps 2, 3, etc. would result in conflicting versions.
In these languages, there is good reason to define dependencies in a relaxed way (with constraints that exclude known-bad versions; but without pins to any specific known-to-work version and without constraining only to existing known-good versions) at first. This way dependency resolution always involves some sort of constraint solving (with indeterminate results due to the constraints being open-ended), but then for the sake of reproducibility the result of the constraint solving process may be used as a lockfile. In the Python world this is only done in the final application (the final environment running the code, this may be the test suite in for a pure library) and the pins in the lock aren't published for anyone to reuse.
To reiterate, the originally proposed algorithm doesn't work for languages with shared dependencies. Using version constraints and then lockfiles as a two-layer solution is a common and reasonable way of resolving the dependency topic in these languages.
No; in fact it's perfectly reasonable, and at the core of what the author doesn't seem to get. Developers have motivations other than reproducibility. The entire reason we have version number schemes like this is so that we can improve our code while also advertising reasonable expectations about compatibility. If we have dependents, then hopefully this also improves their UX indirectly — whether by taking advantage of optimizations we made, not encountering bugs that were actually our fault, etc. Similarly, if we have dependencies, we can seek to take advantage of that.
Upgrading environments is an opportunity to test new configurations, and see if they're any better than what's in existing lockfiles.
> But this is what version ranges essentially are. Instead of saying “libpupa 1.2.3 depends on liblupa 0.7.8”, they are saying “libpupa 1.2.3 depends on whatever the latest liblupa version is at the time of the build.”
But also, developers aren't necessarily using the latest versions of their dependencies locally anyway. If I did pin a version in my requirements, it'd be the one that I tested the build with, not necessarily the one that was most recently released at the time of the build. Not everyone runs an industrial-strength CI system, and for the size of lots of useful packages out there, they really shouldn't have to, either. (And in the pathological case, someone else could re-release while I'm building and testing!)
> But... why would libpupa’s author write a version range that includes versions that don’t exist yet? How could they know that liblupa 0.7.9, whenever it will be released, will continue to work with libpupa? Surely they can’t see the future? Semantic versioning is a hint, but it has never been a guarantee.
The thing about this is that "work with [a dependency]" is not really a binary. New versions also fix things — again, that's the main reason that new versions get released in the first place. Why would I keep writing the software after it's "done" if I don't think there's anything about it that could be fixed?
For that matter, software packages break for external reasons. If I pin my dependency, and that dependency is, say, a wrapper for a third-party web API, and the company operating that website makes a breaking change to the API, then I just locked myself out of new versions of the dependency that cope with that change.
In practice, there are good reasons to not need a guarantee and accept the kind of risk described. Lockfiles exist for those who do need a guarantee that their local environment will be set in concrete (which has other, implicit risks).
I see it as much like personal finance. Yes, investments beyond a HISA may carry some kind of risk. This is worthwhile for most people. And on the flip side, you also can't predict the future inflation rate, and definitely can't predict what will happen to the price of the individual goods and services you care about most.
> The funny thing is, these version ranges end up not being used anyway. You lock your dependencies once in a lockfile and they stay there, unchanged. You don’t even get the good part!
??? What ecosystem is this author talking about? Generating a lockfile doesn't cause the underlying dependency metadata to disappear. You "get the good part" as a developer by periodically regenerating a lockfile, testing the resulting environment and shipping the new lock. Or as a user by grabbing a new lockfile, or by just choosing not to use provided lockfiles.
> “But Niki, you can regenerate the lockfile and pull in all the new dependencies!” Sure. In exactly the same way you can update your top-lev...
The entire point of lockfiles is to let the user decide when the version resolution algorithm should execute and when it shouldn't. That's all they do and they do it exactly as promised.
Given that, I still see some consequences:
The burden for testing if a library can use its dependency falls back on the application developer instead of the library developer. A case could be made that, while library developers should test what their libraries are compatible with, the application developer has the ultimate responsibility for making sure everything can work together.
I also see that there would need to be tooling to automate resolutions. If ranges are retained, the resolver needs to report every conflict and force the developer to explicitly specify the version they want at the top-level. Many package managers automatically pick one and write it into the lock file.
If we don’t have lock files, and we want it to be automatic, then we can have it write to the top level package manager and not the lock file. That creates its own problems.
One of those problems comes from humans and tooling writing to the same configuration file. I have seen problems with that idea pop up — most recently, letting letsencrypt modify nginx configs, and now I have to manually edit those. Letsencrypt can no longer manage them. Arguably, we can also say LLMs can work with that, but I am a pessimist when it comes to LLM capabilities.
So in conclusion, I think the article writer’s reasoning is sound, but incomplete. Humans don’t need lockfiles, but our tooling need lockfiles until it is capable of working with the chaos of human-managed package files.