Oh man I've needed this for a while to the point where I was about to build it my self so thanks! It would be great though if it also listed the tagged nixpkgs version in addition to the hash if applicable.
First off, amazing tool and thanks for doing the nix community a favor!
If you could expand more on the architecture of the tool I'd be very curious! Like what goes all on from parsing Hydra builds to getting a useful package versions DB and what were the challenges while building it? Also, how are you merging all the nodejs packages (nodejs_20, nodejs_18, ...) into one “nodejs”?
I'm also in the process of building something very similar (didn't know the idea was that popular haha), but I'm not parsing Hydra builds but the output from `nix-env -qa --json` to get all the package versions and it's proving more involved than I anticipated (:
First, we processed build outputs from Hydra to map package names + version numbers to commits in the Nixpkgs repo. We then created a service that stores this index as a prefix tree, making it easy to search for a specific package and version in Nix.
There's some extra cleanup required to make the index really usable, which we'll share in the follow up post
For the merged packages -- we have a hardcoded "canonical names", which map the various attribute paths like `nodejs_20` and `nodejs_18` to a single, easier to search name (like `nodejs`). We originally designed this so that Devbox users (who may not be familiar with Nix naming conventions) can just type `devbox add nodejs@20` and get the right package version.
Thanks for the answer! Looking forward to the followup post :D
And yeah, nix naming conventions are sometimes a mess. Your approach does seem much nicer.
Last question if you have a bit more time. How come that you don't have every version of a package available? Is it because you only parse the Hydra stable 23 builds (which I guess aren't updated on every version) or because you parse them only periodically like Lazamar's Nix Package Versions?
We're parsing periodically (though our index is more up to date than Nix Package Versions at the moment). We've also truncated pre-Flakes commits, though we are planning to include those in the future
Btw, regarding a bit older pre-Flakes commits, at least some packages will error out while nix-installing on M1 (or newer) macbooks because at that point nix packages didn't anticipate macbooks ever ditching x86 for aarch... Hope this info helps you out, at least took me some time to realize that the random obscure error was caused by this. Can be almost always fixed by appending "--system darwin-x86_64" to the nix command to use Apple's Rosetta binary translation, cheers.
It's interesting how Nix's selling point is that its non-standard file system structure allows multiple versions/flavors of the same package to be installed at the same time, yet if I understand correctly, it often has one version per package per Nix version? It's more like install multiple versions of the same package at a different time?
Compare this to the model we have in Spack (which is inspired by nix), where there's one recipe per package for many versions, and you can define different build flavors, conditional dependencies (and conditional pretty much everything).
The difference is that Spack has a dependency solver and Nix apparently does not?
For example the curl package [1] has tons of versions, and its `tls` package variant has conditional values depending on the version:
depends_on("gnutls", when="tls=gnutls")
with when("tls=mbedtls"):
depends_on("mbedtls@2:", when="@7.79:")
depends_on("mbedtls@:2", when="@:7.78")
This allows you both to install all sorts of curls, like `spack install curl tls=mbedtls` (gives you something >= 7.46), or `spack install curl@7.77 tls=mbedtls` gives you a curl with an old mbedtls, etc.
Probably because it's pretty easy to just use an older package from a different version of Nixpkgs (just use an override that defines the attribute for a package in your current Nixpkgs as the same package in an older Nixpkgs). It'll transparently pull the older version from the binary cache, too, if available.
The only difficulty is finding the right Nixpkgs commit hash, which this solves. I think I find the Nix way to be a bit cleaner, though I'm unaware of the full set of pros/cons.
Worth noting that the Nix package definition for curl at [1] is a little easier to read for me since it doesn't have the duplicated code for each version.
> yet if I understand correctly, it often has one version per package per Nix version?
Both yeah and no. Taking the example of nodejs, you have all the supported major releases like nodejs_16, nodejs_18, etc. for each Nix revision[1], but within each revision you only have a specific version of the package. E.g. nodejs_16 is 16.3.1 for revision at date X. There are benefits and drawbacks of managing a package repo like that, with revisions which implicitely pin all packages and it's dependencies at once, but it's too long of a topic to write about here.
> For example the curl package [1] has tons of versions [...]
So does it in nix (curlMinimal, curlWithGnuTls, curlHTTP3, etc)[2]. In that regard nix and Spack are similar.
[1]: A revision is like a shapshot of all the packages at a given point in time.
The main difference is that Spack has a single definition of a package, and in Spack language `curlMinimal`, `curlWithGnuTls`, and `curlHTTP3` are all "concretizations" of curl. That has the advantage that dependents can say `depends_on("curl")` if they don't care about the flavor -- only if they somehow require curl with GNU tls, they say `depends_on("curl tls=gnutls")`.
Nix also has a single definition of the curl package. The curl package itself can take tons of flags as inputs, indicating how it should be built, and you can then have curlMinimal, curlWithGnuTLS, and curlHTTP3 all as different 'names' for curl with those arguments applied. A package just says it has a 'curl' input and any user can "call" that package, providing any flavor of 'curl' they want as input. You can override those inputs at a global level and local level in various ways.
I suspect you mean some other semantic difference, but if that's all you mean, it's not different from Nix in practice, from what it sounds like?
Maybe I'm misunderstanding, but my impression is that Spack is more composable:
In Spack pkg A can say `depends_on("curl tls=gnutls")`, pkg B can say `depends_on("curl +nghttp2")`, and if you then request a package that depends on both A and B, you end up with `curl tls=gnutls +nghttp2` as a transitive dependency.
It sounds like in Nix it's either/or: if you need both curlWithGnuTLS and curlWithHTTP2, what do you do?
I'm getting the impression that the main difference here is that Nix's approach is more 'concrete'.
In Nix-world, you'd describe your package as a function, where the inputs are parameters. Something like `function myPackage (curl, gcc) { ... }`. When this `myPackage` is built, you can pass in what you want `curl` to be. So, you'd think that it's something like `curlWithGnuTLS = curl(..., GnuTLS = true)`, and then could have something like `myPackageWithCustomCurl = myPackage(curl = curlWithGnuTLS, ...)`.
To give more specific Nix examples,
here's where `curlWithGnuTLS` is defined https://github.com/NixOS/nixpkgs/blob/dbf6c323883ae7e00941f6... By convention, the packages that want curlWithGnuTLS specifically use that name as an argument. (That's because `callPackage` will use the argument names to find the values to assign to it).
In nix, customising these package arguments from the CLI isn't really ergonomic.
It sounds like with Spack, there's more emphasis on being able to specify those parameters from the command line when installing things.
That said, to the question about different dependencies: if package A depends on curl-with-foo, and package B depends on curl-with-bar.. it does depend on how it's declared, but Nix is comfortable to have package A linked against curl-with-foo and B linked against a separate curl-with-bar. -- Although it would also be possible to declare a curl-with-foo-and-bar and pass it to each.
e.g. on my machine, if I run `ls /nix/store/ | rg 'curl.-.-bin'`, I have dozens of lines like:
To add to that: Spack's solver effectively generates these 'concrete' packages for you.
In the example, suppose you're packaging `X`, it depends on already existing packages `A` and `B`, and `A` needs `curl +foo` and `B` needs `curl +bar`; then Spack is convenient as you don't have to worry about the transitive dependency curl, the solver will output `curl +foo +bar`.
That should make contributing to Spack potentially easier, cause you just have to define X, the solver will figure out the flavor of transitive dependencies, and there's no need to create the `curl-with-foo-and-bar` package (and you don't have to modify `A` and `B` to use `curl-with-foo-and-bar` either)
> It's interesting how Nix's selling point is that its non-standard file system structure allows multiple versions/flavors of the same package to be installed at the same time, yet if I understand correctly, it often has one version per package per Nix version?
You're talking about Nixpkgs, and it's a cultural choice. In the past there used to be more of this, where e.g. the Haskell community had lots of variants of the same library all at once, because users and other libraries wanted them. But it became untenable to maintain and you in practice have to do tons of effort to vendor things. And once many of the subcommunities began moving to auto-generated packages (e.g. generating Nix packages from Haskell packages), it becomes much easier to have a single globally consistent repository of versions that all are known to work together, generated by the host-language dependency resolver, with minor fixes applied on demand.
There are some things that do have multiple versions in the main repository, e.g. LLVM. But in practice having like 10 versions of everything is a huge chore and only done on demand in practice if users want it, and it has real overhead cognitively and in build time and in maintainer time.
You can define your own Nix flake with every version of every Python library in it if you want. But in practice people sort of pick versions of things that make QA a real chore.
EDIT: Also, for like 99% of cases, what users think is cool but hardly do is to run both pieces of software at the exact same time concurrently. They typically just want atomic upgrades and rollbacks. For example, you change your git revision at work and get whole new tools, your old tools still work if you go back to the old git revision at no loss. So the need to only have one version "globally" under one name isn't a big deal in practice, because people often are only dealing with one global name. They just don't want their packages to get corrupted during an upgrade or switchover or something otherwise breaks.
For most people, depending on historic versions of nixpkgs is an anti-pattern. The main reason I say this is security. You may want an old version of, say, ruby, but you probably don't want the old, insecure libraries it was built against (i.e. glibc, openssl).
It is often just as easy to use overrides to build a different version of a piece of software, and in the rare case that doesn't work you can fall back to a custom derivation (drawing inspiration from the old version of nixpkgs, perhaps).
I worry about tools like this which make doing the wrong thing too easy.
Yeah. People like to push that there are “legitimate reasons” to want old versions, but it usually comes down to businesses.
I would suggest instead that there are no legitimate reasons, and that businesses who are requiring old software are at best being negligent.
I’ve worked with people that refuse to modernize and update and it’s always the same stupid thing: “cost”. We really do need strong regulations on businesses that negligently refuse keep reasonably up to date.
1) In general, older code tends to have security patches
2) it costs money to run a business, and saving money through gross negligence, leading to real people having real impacts through massive data leaks while suffering zero repercussions for that needs to end. “It costs money to keep software up to date” is a shit argument for why businesses should not be managing their software.
> I would suggest instead that there are no legitimate reasons, and that businesses who are requiring old software are at best being negligent.
1. Backwards incompatible schema changes for data that the software works with which requires a migration path. The version for that software needs to be locked until that migration is complete.
2. A functionality regression in the software. The version needs to be locked until that patch lands.
3. A security vulnerability can be introduced with new features. The version needs to be locked until that security issue is patched out.
You can't just yolo to the latest version of software every time it drops.
28 comments
[ 3.3 ms ] story [ 104 ms ] threadIf you could expand more on the architecture of the tool I'd be very curious! Like what goes all on from parsing Hydra builds to getting a useful package versions DB and what were the challenges while building it? Also, how are you merging all the nodejs packages (nodejs_20, nodejs_18, ...) into one “nodejs”?
I'm also in the process of building something very similar (didn't know the idea was that popular haha), but I'm not parsing Hydra builds but the output from `nix-env -qa --json` to get all the package versions and it's proving more involved than I anticipated (:
We're planning a longer post that explains the architecture and process of building the index. An extremely basic description of what we do can be found here: https://www.jetpack.io/blog/0-5-0-install-nix-packages-by-ve...
There's some extra cleanup required to make the index really usable, which we'll share in the follow up postFor the merged packages -- we have a hardcoded "canonical names", which map the various attribute paths like `nodejs_20` and `nodejs_18` to a single, easier to search name (like `nodejs`). We originally designed this so that Devbox users (who may not be familiar with Nix naming conventions) can just type `devbox add nodejs@20` and get the right package version.
And yeah, nix naming conventions are sometimes a mess. Your approach does seem much nicer.
Last question if you have a bit more time. How come that you don't have every version of a package available? Is it because you only parse the Hydra stable 23 builds (which I guess aren't updated on every version) or because you parse them only periodically like Lazamar's Nix Package Versions?
Btw, regarding a bit older pre-Flakes commits, at least some packages will error out while nix-installing on M1 (or newer) macbooks because at that point nix packages didn't anticipate macbooks ever ditching x86 for aarch... Hope this info helps you out, at least took me some time to realize that the random obscure error was caused by this. Can be almost always fixed by appending "--system darwin-x86_64" to the nix command to use Apple's Rosetta binary translation, cheers.
Compare this to the model we have in Spack (which is inspired by nix), where there's one recipe per package for many versions, and you can define different build flavors, conditional dependencies (and conditional pretty much everything).
The difference is that Spack has a dependency solver and Nix apparently does not?
For example the curl package [1] has tons of versions, and its `tls` package variant has conditional values depending on the version:
And dependencies are specified as This allows you both to install all sorts of curls, like `spack install curl tls=mbedtls` (gives you something >= 7.46), or `spack install curl@7.77 tls=mbedtls` gives you a curl with an old mbedtls, etc.[1] https://github.com/spack/spack/blob/develop/var/spack/repos/...
The only difficulty is finding the right Nixpkgs commit hash, which this solves. I think I find the Nix way to be a bit cleaner, though I'm unaware of the full set of pros/cons.
Worth noting that the Nix package definition for curl at [1] is a little easier to read for me since it doesn't have the duplicated code for each version.
[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/netw...
Both yeah and no. Taking the example of nodejs, you have all the supported major releases like nodejs_16, nodejs_18, etc. for each Nix revision[1], but within each revision you only have a specific version of the package. E.g. nodejs_16 is 16.3.1 for revision at date X. There are benefits and drawbacks of managing a package repo like that, with revisions which implicitely pin all packages and it's dependencies at once, but it's too long of a topic to write about here.
> For example the curl package [1] has tons of versions [...]
So does it in nix (curlMinimal, curlWithGnuTls, curlHTTP3, etc)[2]. In that regard nix and Spack are similar.
[1]: A revision is like a shapshot of all the packages at a given point in time.
[2]: https://search.nixos.org/packages?channel=23.05&from=0&size=...
I suspect you mean some other semantic difference, but if that's all you mean, it's not different from Nix in practice, from what it sounds like?
https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/netw...
In Spack pkg A can say `depends_on("curl tls=gnutls")`, pkg B can say `depends_on("curl +nghttp2")`, and if you then request a package that depends on both A and B, you end up with `curl tls=gnutls +nghttp2` as a transitive dependency.
It sounds like in Nix it's either/or: if you need both curlWithGnuTLS and curlWithHTTP2, what do you do?
I'm getting the impression that the main difference here is that Nix's approach is more 'concrete'.
In Nix-world, you'd describe your package as a function, where the inputs are parameters. Something like `function myPackage (curl, gcc) { ... }`. When this `myPackage` is built, you can pass in what you want `curl` to be. So, you'd think that it's something like `curlWithGnuTLS = curl(..., GnuTLS = true)`, and then could have something like `myPackageWithCustomCurl = myPackage(curl = curlWithGnuTLS, ...)`.
To give more specific Nix examples, here's where `curlWithGnuTLS` is defined https://github.com/NixOS/nixpkgs/blob/dbf6c323883ae7e00941f6... By convention, the packages that want curlWithGnuTLS specifically use that name as an argument. (That's because `callPackage` will use the argument names to find the values to assign to it).
In nix, customising these package arguments from the CLI isn't really ergonomic.
It sounds like with Spack, there's more emphasis on being able to specify those parameters from the command line when installing things.
That said, to the question about different dependencies: if package A depends on curl-with-foo, and package B depends on curl-with-bar.. it does depend on how it's declared, but Nix is comfortable to have package A linked against curl-with-foo and B linked against a separate curl-with-bar. -- Although it would also be possible to declare a curl-with-foo-and-bar and pass it to each.
e.g. on my machine, if I run `ls /nix/store/ | rg 'curl.-.-bin'`, I have dozens of lines like:
To add to that: Spack's solver effectively generates these 'concrete' packages for you.
In the example, suppose you're packaging `X`, it depends on already existing packages `A` and `B`, and `A` needs `curl +foo` and `B` needs `curl +bar`; then Spack is convenient as you don't have to worry about the transitive dependency curl, the solver will output `curl +foo +bar`.
That should make contributing to Spack potentially easier, cause you just have to define X, the solver will figure out the flavor of transitive dependencies, and there's no need to create the `curl-with-foo-and-bar` package (and you don't have to modify `A` and `B` to use `curl-with-foo-and-bar` either)
Also here I'm referring to specific version of zig for this particular LSP's config file, but it never goes to my PATH https://github.com/Cloudef/nixos-flake/blob/master/modules/n...
You're talking about Nixpkgs, and it's a cultural choice. In the past there used to be more of this, where e.g. the Haskell community had lots of variants of the same library all at once, because users and other libraries wanted them. But it became untenable to maintain and you in practice have to do tons of effort to vendor things. And once many of the subcommunities began moving to auto-generated packages (e.g. generating Nix packages from Haskell packages), it becomes much easier to have a single globally consistent repository of versions that all are known to work together, generated by the host-language dependency resolver, with minor fixes applied on demand.
There are some things that do have multiple versions in the main repository, e.g. LLVM. But in practice having like 10 versions of everything is a huge chore and only done on demand in practice if users want it, and it has real overhead cognitively and in build time and in maintainer time.
You can define your own Nix flake with every version of every Python library in it if you want. But in practice people sort of pick versions of things that make QA a real chore.
EDIT: Also, for like 99% of cases, what users think is cool but hardly do is to run both pieces of software at the exact same time concurrently. They typically just want atomic upgrades and rollbacks. For example, you change your git revision at work and get whole new tools, your old tools still work if you go back to the old git revision at no loss. So the need to only have one version "globally" under one name isn't a big deal in practice, because people often are only dealing with one global name. They just don't want their packages to get corrupted during an upgrade or switchover or something otherwise breaks.
It is often just as easy to use overrides to build a different version of a piece of software, and in the rare case that doesn't work you can fall back to a custom derivation (drawing inspiration from the old version of nixpkgs, perhaps).
I worry about tools like this which make doing the wrong thing too easy.
I would suggest instead that there are no legitimate reasons, and that businesses who are requiring old software are at best being negligent.
I’ve worked with people that refuse to modernize and update and it’s always the same stupid thing: “cost”. We really do need strong regulations on businesses that negligently refuse keep reasonably up to date.
2) it costs money to run a business, and saving money through gross negligence, leading to real people having real impacts through massive data leaks while suffering zero repercussions for that needs to end. “It costs money to keep software up to date” is a shit argument for why businesses should not be managing their software.
1. Backwards incompatible schema changes for data that the software works with which requires a migration path. The version for that software needs to be locked until that migration is complete.
2. A functionality regression in the software. The version needs to be locked until that patch lands.
3. A security vulnerability can be introduced with new features. The version needs to be locked until that security issue is patched out.
You can't just yolo to the latest version of software every time it drops.