Seems the direnv plugin aims to resolve, but I struggled to get it working. 200ms for hasn't been a huge issue (yet), but not having the proper paths set up has been. Using shims isn't the greats experience (something direnv plugin also aims to address).
I mentioned in my other comment that the direnv plugin aims to resolve this. Personally I didn't get it working but I haven't been determined enough. It has been working for others.
If you are looking for a language agnostic version manager, the nix package manager [0] might be worth a try. In combination with lorri [1] you will be dropped into a shell with all required development dependencies available, when you enter the project directory. This does not only include the interpreter/compiler versions but also any other dependency you can think of, like specific libraries the project links against (well almost any, it has to be available as a nix expression, best case directly in upstream nixpkgs).
For me, it has come so far that I don't have any interpreters or compilers in my main OS environment and even for one off REPL sessions, I'll use `nix shell nixpkgs#python3 -c python3`.
For those who like the concept of Nix, but don't like the Nix language, the Guix package manager [0] might also be worth a try. It's an official GNU project, inspired by Nix, and it uses Guile Scheme instead of Nix language.
The code in nixpkgs does contain some code that uses relatively sophisticated programming language constructs, such as lazy evaluation, partial function application, or fixpoints. -- So I'll say there's at least some friction to Nix from the language being pure/functional.
Mostly I think Nix would be just as hard if packages were expressed in any language.
Stuff like "this program compiled by nix is using a different glibc than this library" is going to be confusing regardless of whether Nix expressions can be written with JavaScript; or stuff like some program assumes it can write to $HOME in its build script not playing nicely with Nix.
A lot of why Nix is hard is because it's weird, and that even as an end user when things go wrong it may require understanding more about what's going on than what other solutions require.
I mean to try it but some of these version managers work on Windows too, I believe Nix is specific to Linux. Lorri sounds amazing though! It's the kind of thing that made me fall in love with solutions such as virtualenv (which handles the "this directory needs these specific packages" installed locally, not globally problem that Debian gives you).
One thing that differentiates nix from things like virtualenv is that each (package + version) tuple exists at most once in your local nix store and is then linked into your project environments, while virtualenv will install new copies of the packages. On the other hand you have to run the nix garbage collector to get rid of unused packages.
As for Windows support: It seems to be possible to use nix in WSL [0] but I've never tried that. I have used nix alongside apt and pacman on Ubuntu/Arch before I decided to go all the way and install the NixOS distribution. Using it alongside other package managers worked really well.
I've used Nix on Linux (i686, x86_64 and ARM), macOS and Windows (WSL2). I believe it also works (although not officially supported) on BSDs, etc. too, although I've not tried that yet.
> Lorri sounds amazing though! ...which handles the "this directory needs these specific packages" installed locally, not globally problem that Debian gives you).
You're describing Nix, not Lorri. Specifically:
- nix-shell provides a directory-local environments (and more, e.g. #!nix-shell to fetch script dependencies automatically)
- direnv can enter/exit this nix-shell environment when cd-ing in/out of a directory
- lorri makes direnv+nix-shell faster, by building the environment asynchronously in the background (rather than causing our terminal to freeze)
shell.nix + nix install is _the_ killer linux dev tool imo. It replaces all the various nonsense with a hermetic, fully reproducible build environment. You can then use the same shell.nix in your CI with the nix docker image. Now you have exactly identical dev environments across CI and locally. It's awesome once you've experienced it.
The biggest drawback is installing Nix on OS X is a huge pain. Docker has lots of magical UX tooling to make it rather seamless in comparison.
Installing NixOS on modern macOS nowadays is super trivial. Copy+paste command from website, follow instructions, optionally enable flakes in a config file.
That's it.
The daemon is not powerful at all, almost all the logic happens in the client.
The client does the parsing, job ordering, and tells the daemon exactly what to do.
The daemon, only required in multiuser environments, does only two things really.
- ensure that the /nix/store is protected as the daemon should be the only process able to _write_ to it. Everything can read from the nix store; no nix required at all.
- execute build commands in a sandbox. Providing isolation between multiple builds.
Because the nix store is global per system, it would be unsafe to allow any user to write to the nix store in a multi-user system. So the daemon instead mediates writes to the store.
Based on other comments like this, I gave it a try.
Be prepared to sink dozens and dozens of hours learning the language, it’s features, and switching everything over. Be prepared to get stuck when you are required to use an old version of a language and things don’t install correctly — and sometimes blocked by nix itself from installing it. Be prepared to get stuck if you are not in the happy path. Be prepared to get lost in the sometimes scant documentation.
I love the idea of nix, but it did not work for me except for the simplest cases.
I agree that the documentation story could be better. I also think it's a great shame that the language isn't statically-typed, so to understand how to use something I have to inspect its source code.
I've found it to be quite flexible though. For example, here's a commit in which I apply a patch to a tool to solve a problem that the derivation hadn't taken into account (and absent a home-manager solution): https://github.com/samhh/dotfiles/commit/867dd3b4d4b3942a0aa...
Static types are documentative, and language servers often also show you things like JSDoc alongside the type signature. Nix has neither of these, hence I have to check the source code or run a build and see what happens.
I really like Nix but in this particular way it feels like taking a big step backwards from the other languages I frequently use, particularly for a language in which you'll necessarily be constantly interacting with bespoke interfaces.
Informative function signatures, allowing programmers to add per-function documentation, and tooling that displays those things are by no means limited to staticly typed languages.
I guess knowing allowed types for each parameter is informative (more if very specific types are used rather than string or integer), but documentation usually specifies that anyway. And even knowing all the types isn't usually enough without documentation.
My point is just that I've used plenty of libraries in dynamically typed languages without needing to read the source. And conversely, I've occasionally needed to look at the source of a function in a staticly typed language, to answer some question not answered by the types and documentation.
Being able to see a function's entire type signature at a glance is the single most useful way of documenting code I've ever come across. No alt-tabbing to docs or relying on an IDE to pop up hints, etc., it's just right there.
Haskell is the ideal here, since it separates the type signature into its own separate line at the top of the function declaration. That confers readability and reduces cognitive overhead more than any other language I've used.
It also changes the programming thought process. You can pseudo-code an entire program just with function type definitions. Then test that the type defs compile without error, and go back and implement the function definitions. As long as the function definitions adhere to their type signature, the program almost always works (barring some I/O errors).
Not all statically typed languages are created equal. The ones fundamentally oriented around function type signatures, rather than variable types, are the ones I think parent had in mind in his comment.
The Nix crowd is becoming a lot like the Rust crowd-- they hijack every conversation with "Have you tried Nix"?
Like you, I find Nix hard to use and the docs confusing (beyond trivial examples). The amount of time the Nix mafia spends proselyting could be better spent making better docs
I agree. I've got nothing against nix, but it frankly sucks that the top comment isn't about asdf which I had hoped to learn about. There should be special "-1 offtopic" comment flagging because it's a frequent phenomenon.
As a fan of Nix and NixOS, I agree with this assessment. Too often people hijack threads about a topic to discuss something related but totally different and off-topic. It's kind of annoying, even when I'm a fan of the off-topic thing.
I've never heard of asdf and popped in here to read about it, people's experiences with it, etc. If I want to read about Nix/OS I'll find a thread on that, or go to /r/nixos, etc.
> the top comment isn't about asdf which I had hoped to learn about
Oh I think you're using HN wrong. To learn about the subject you can read the actual article that's linked to. No guarantees about the comments section.
This criticism is getting old. Piping curl output to bash is no less secure than installing a package via any other mechanism. Either you trust the source or you don't. I trust it more than I trust npm or pip because I'm typically getting it from the primary source instead of relying on a middleman that's got a poor track record.
"Someone else replaces the good program with a malicious program" is the attack vector people are worried about when talking about sha256.
But, since you're downloading & running code written by someone else, it seems weird to talk about that but ignore "what if this program I'm downloading does something malicious".
Presumably you trust the authors or you wouldn't be downloading it to begin with. The primary concern isn't "what if the authors are out to get me" it's "what if someone impersonates or compromises the authors".
Yeah, frustration on both sides here. On one hand, I can see how the nix evangelism can get annoying, especially here on HN. On the other hand the nix crowd is seeing a parade of ad hoc, informally-specified, niche implementations of way-less-than-half of nixpkgs. This is an important problem, maybe one of this most important problems the tech world is grappling with right now, and there is already a solution.
The reality is that the docs are fine. The reference material is comprehensive, and there are some very good tutorials and introductory essays to get people started. The problem is that it's difficult. Nix is very, very different from what people already know, and it takes quite a bit of effort to learn how it works. Most people just aren't prepared to do that much work just to install some software.
The ad-hoc solutions you're talking about are often highly usable, a virtue often forgotten by software evangelists. Nix's learning curve will keep away people who are disciplined in avoiding yak-shaving, as the language specific version enables productivity now. Maybe adding a layer on top of Nix, like create-react-app is to webpack, can make it a better option for immediate productivity oriented developers.
If the problem is that it's difficult and the documentation is not making that difficulty simple enough to use, then the documents are not fine. Simple as.
If the problem is that it's difficult to learn to fly a fighter jet, and the documentation is not making it simple, the real problem is that the documentation is not fine. Simple as that.
When you frame it like that, the question then becomes "do you actually need the fighter jet?" and "why are all these people suggesting I use a fighter jet?"
That's pushing the analogy past where it breaks and missing the point. Take someone who has only ever written C-like code. Hand them a Scheme, maybe Racket. You expect them to initially have difficulty regardless of the quality of documentation. It's a different way of thinking about and doing things and there's a lot of it to deal with.
As someone who has a recreational pilot license (sure, not jets, but still) you just reinforced the point of the previous comment rather than refute it for me.
Most pilot text books are absolutely horribly written, finding a good one takes significant effort. Reading just an ok book shines a light on how absolutely trash the majority of pilot text books are and that it is absolutely bad "documentation" that is partially to blame for it being difficult to learn.
Nope nope nope. They're not fine. Nix is really really hard to use. I worked for a company that ran all the ops with Nix managed by some of the contributors. Every time I wanted to do something I'd have to get DevOps to do it for me. I've never ever had that level of difficulty doing stuff outside of a Nix environment and I really, really tried. Most of my colleagues who are fairly decent developers struggled with it too. It's the only tech I advise against when it's mentioned nowadays. And I want to like it.
On the other hand only Nix (to my knowledge) guarantees truly reproducible builds.
It helps that people who are into Nix most often use NixOS. NixOS is the only distro I've written packages for. It's not even particularly hard since there's a lot of inspiration to find in the official and community repos + people sharing personal configurations.
The UX wasn't the best but it's improving.
It remains extremely reliable which is not something that you could say of other systems.
I went all in and used NixOS too, but things like managing CI with Hydra, running perf tests on Nix ops clusters, debugging 10K+ Nix repos… you start to get it down and then things change like with flakes… and all the tools like Nix2Stack, Nix2Cabal, Nix2Docker, managing secrets, user profiles, none of it is existing knowledge you can bring or really take elsewhere. If you’re familiar with the ‘innovation token’ idea Nix would take all of them in my eyes for a project.
Docker is more pragmatic unless you spend hundreds of hours on Nix, in my opinion.
--arg name value
This option is accepted by nix-env, nix-instantiate and nix-build. When
evaluating Nix expressions, the expression evaluator will automatically try to
call functions that it encounters. [...]
Sorry but the notion that the docs are fine is laughable. Most nix functions don't even have doc comments. You have to figure them out by delving into examples in the depths of github. Nix is in dire need of a culture of providing documentation comments, along with a tool that builds them into a website, like rustdoc or haddock.
Hum... Every time I see Nix cited somewhere, it's somebody who tried it and didn't manage to get expertise out of the documentation maze. (You can add me there too.)
But it's no wonder that people keep commenting that Nix is an awesome idea that solves a lot of problems. That's because it's an awesome idea that solves a lot of problems (yeah, like Rust).
It is also completely undiscoverable (not completely unlike Rust), and that's why almost nobody uses it. But that's a matter of improving the documentation or maybe fixing one or two superficial problems on the language.
> But that's a matter of improving the documentation or maybe fixing one or two superficial problems on the language.
Based on a quick check of their Wikipedia page, Nix has been around for 15 years, since 2007.
So those for sure look like structural problems. I don't know which ones exactly, but paraphrasing Tolstoi, popular software products are popular in much the same way, while unpopular ones are unpopular in their own unique ways.
In my experience almost no open source project that hasn't entered the mainstream in its first 10 years manages to turn the ship around, unless it lucks into a major change of environment. Programming languages are sometimes exempt if they have a "killer" library or framework pop up.
Nix ain't trying to win marketshare it's trying to be amazing. But it takes guts to be amazing and that means a lot of people don't manage to use Nix.
HN tends to see success in terms of adoption rate & ranking. Nix is on another plane. It's beyond critical mass. And for people like me, it is a huge force multiplier. But I had to grind my way to the point where that is true.
I tend to not bother selling people on it. People who want what Nix does will find it at some point. So I just focus on using Nix myself in a world of worse tools. Feels great.
Nix just got a huge overhaul, with most functionality getting bundle under a single easy to use 'nix' command (i.e. similar to 'apt'), rendering a lot of the old commands obsolete. That currently creates a bit of a confusing situation, as lots of docu is about old commands, while new commands that look similar can behave quite a bit different (e.g. 'nix-shell' != 'nix shell'). Nix Flakes are another new thing and also create a bit of confusion and ugliness (e.g. packages being named 'nixpkgs#legacyPackages.x86_64-linux...').
That said, I haven't found Nix hard to use, quite the opposite. The new 'nix' command is pretty self explanatory, it's just a little incomplete in spots. And the Nix language is quite simple and easy to understand if you have ever touched a functional language. Compared to all the other Linux stuff I have played with over the years, it was a very pleasant experience so far.
That could just as easily be attributed to the word Nix being easy to search for. But even if everyone on HN is aware of what it is, that doesn't make it mainstream by any definition I can think of. There are countless examples in the world of things that most people (in a given audience) are aware of, but that almost none of them participate in.
Is Haskell mainstream? What about committing murder?
Your claim is that software doesn't go from non-mainstream into popular if it stays non-mainstream for a decade. I'm really not willing to discuss the definition of "mainstream", but there are plenty of cases of software that stayed a decade as "everybody knows it, nobody uses it" and eventually got popular.
Life is too short to spend hundreds or thousands of hours learning a tool to save you time. Nix might give you 'perfect' reproducible builds but does the time saved from this beat the time taken to learn it? I'm skeptical.
I have the exact opposite experience. Nix is the best solution out there especially in the sort of situations that you describe.
Nix provides a level of flexibility that other solutions simply do not offer. If a package in Nix doesn't fit your requirements for whatever reasons, you can create a modified version of a package with ease. For example, say that you need a version of Nginx built against a custom version of OpenSSL. You can do just that with a few lines of code:
Nix will even know which packages it'd need to build locally instead of downloading a prebuilt binary.
You can't do that with other common package managers. You're stuck with whatever the package manager provides you. So if you're not in the happy path, you're out of luck. You'd either have to give up or build from scratch. If you need a bugfix for a particular package, you'd have wait until the fix reaches the package repository. When I was using Ubuntu, that was often until the next major release. None of this stuff is a problem with Nix, which allows for customization with very little effort.
It's easy to learn about Nix if you know where to look for. Nix Pills [1] would be a good start. The core language is the easiest part. It's JSON, but with functions and variables for proper abstraction. The documentation, while imperfect, is quite extensive compared to a majority of other high profile open source projects. If the documentation fails you, the Nixpkgs repository [2] is an even more rich source of information. The code is well organized, and I was able to get familiar with writing Nix packages fairly quickly by grepping the codebase. And finally, Nix has a sizable community so you could always ask if you're stuck.
I've been using nix as my daily OS for six months and I'm able to be productive. At the same time, I'm often confused as to whether I should use nix-shell, nix-env or modify my configuration files.
I'm new to NixOS too. My general rule is to try to keep my core system build/description as minimal as possible. Default to nix-shell first, and only if I'm using that thing repeatedly do I then test-install it with nix-env, and if that succeeds then add it to configuration.nix.
> It's easy to learn about Nix if you know where to look for.
I understand you are saying it with the best intentions but it might sound like not the most correct way of defending a tool being criticized for its lack of documentation or easy of use.
I feel that the criticism regarding documentation is too harsh. Speaking from experience, Nix has more documentation and community resources than most open source projects that I've encountered.
It is both great and terrible, a shining example for me was that I wanted to install a package built off a specific commit. That was trivial, I just over rode the source and the expected hash and the exact same build & install steps worked.
Also rollbacks in nixos are amazing.
On the other hand newly packaging something can be really daunting.
With a shell script, you get to enjoy all the pains of building software and its dependencies from scratch. Even then, you wouldn't be even halfway done. You'd also need to go through the process of turning your ad-hoc procedures into a reproducible script only to find out that it broke months later.
Nix provides better abstractions and composability to make that experience far less painful. Note that in my previous example, I didn't need to specify how to build OpenSSL or Nginx. I was able to reuse the definition from Nixpkgs and apply my own customizations to it.
The example above is part of a declarative, reproducible, full system build. You include it in your build description file, and then Nix will rebuild the entire system in the exact same way across multiple machines.
That's less of a guarantee with a collection of shell scripts.
Your example looks great! Indeed I trolled every piece of nix documentation I could find -- it was the only way to figure out what I should be doing.
Here's the simplest example of an utter failure when I tried nix: installing ruby 2.2. It's very possible I did something totally wrong, but when I have to use ruby 2.2 and nix cannot do it, it sort of kills your awesome example of being able to override openssl.
Another extremely frustrating experience was the bifurcation between nix and nixos. Clearly one experience is preferred over the other. So many blogs and docs talk about things like HomeManager and other configurations that are nixos specific.
On top of nixos/nix, there's also shell.nix/flakes. There's just too much development in nix right now to make sense of what's going on.
Thanks for that clarification. My point was mainly that I was being led down paths that were talking about home manager which was something I didn’t want in my system. Nix, nixos, home manager, flakes, shell.nix … they all do different things and blog articles and documentation mix and march them to create a very confusing ecosystem to navigate.
Flakes are actually part of the piece for how to sanely get access to old versions of things, as your project flake can bring in multiple instances of nixpkgs from different “stable” branches and then pull the specific packages off each one that you need.
Flakes are undeniably useful but it's gated behind an experimental flag and documentation work is ongoing. For now, it just shouldn't be the first point of entry for beginners.
The documentation is the biggest gap for sure. I've only been using Nix for about a year, and I'm using flakes exclusively. Although I don't have direct experience with the legacy system, I could definitely picture it being very frustrating managing inputs via the NIXPATH envvar rather than explicitly with flake locking.
Fair, fair. A slightly more charitable take might be that flakes solve/streamline a set of problems that are particularly acute for power users, and installing multiple specific versions of software without needing to copy the definitions into your overlay is absolutely a power user move.
Did you ever try using Ruby 2.2 in an isolated nix-shell? That's arguably the best feature of nix-shell, to create an isolated environment with only the packages you need for that particular project.
> when I have to use ruby 2.2 and nix cannot do it
Considering that ruby 2.2 was released in 2014, I can't say I'm surprised. It's too much effort to continue maintaining something so old. But if you really want to use it, you could try using the package definition from an older commit of Nixpkgs:
> Another extremely frustrating experience was the bifurcation between nix and nixos.
I'm not sure what you mean. Nix is the package manager while NixOS is an entire distro based on Nix the package manager and its configuration language. They have clear separation of concerns. Keeping it that way has actual benefits too, namely portability. It allows you to use Nix on a wide range of platforms including non-NixOS Linux distros, macOS, and even BSDs to some extent.
> On top of nixos/nix, there's also shell.nix/flakes.
Flakes are currently an experimental feature and not meant for wide use yet. So if you're just getting started with Nix, I'd recommend looking into it later until you're sufficiently comfortable with Nix.
They're not fundamentally different, though. Flakes formalize the conventions for writing Nix expressions to make them more reusable.
I think an easy trap with nix is the immediate desire to have it manage everything, which usually involves installing NixOS. It's the equivalent of being airdropped into the Romanian countryside with working knowledge of French. I am trying to adapt nix for use with some teams I'm consulting on, and my goal is to make its usage as dead simple as possible for really high value tasks.
I also feel like nix is immediately tangible and beneficial in a way that makes us want to get to the power user level, but I don't feel like wanting to learn, say, kubernetes means one automatically wants to learn to be a cluster operator. The scope and complexity are great, and I have experienced all of the things you have described as well. Now that I am fairly confident in my abilities, I think it's important to make sure we find ways to help everyone get the most out of it, even if it doesn't involve scaling the wall that is the learning curve.
> I think an easy trap with nix is the immediate desire to have it manage everything, which usually involves installing NixOS.
Can you clarity what you mean?
What do you mean by "trap"? Which (if any) of the following concepts factor into your understanding? Broken expectations? Premature optimization? Over-engineering? Lock-in?
Are you suggesting that many users of the Nix package manager "fall into a trap" and begin using NixOS?
What do surveys and/or data show about usage of just the Nix package manager? Combined with NixOS?
Absolutely, and if it was not clear from the context, I call it a trap because that feels like what I'd fallen into at some point. Let me see if I can describe it better.
The Nix ecosystem is attractive to me at a high level because it looks like it has the possibility to revolutionize a lot of the problems that come up with an increased focus on devops culture. The problem with the Nix ecosystem itself is that it is larger and more varied than it might appear.
There are really three large entities in the ecosystem: nix the language, nix the package manager (nixpkgs), and nix the OS (NixOS). Despite the fact that there is a lot of overlap in some of the details of these three things, they are easily understood as separate entities. There are a bunch of smaller secondary entities such as nix-darwin, home-manager, devshell, and digga (formerly devos). The aim of these entities is usually to supplement, enhance, replace, or reproduce features of nixpkgs and/or NixOS.
Each of these entities has its own learning curve. It takes surprisingly little nix knowledge to use it as a dev environment generator, but nixpkgs knowledge will help a bit more. Even within nixpkgs itself, each major language has a large ecosystem built up around facilitating package building and deployment, and these are extremely heterogeneous. Some of them require that you regenerate the entire matrix of package dependencies for the package manager (javascript, haskell). Some of them have tooling that allow you to get started with almost no effort (mach-nix for python). These all have their own peculiarities. NixOS has a ton of modules, all rather heterogeneous, on top of requiring more careful knowledge of how to put together a system than most other distros. And nix the language without nixpkgs is very small, and some of what it does have built in is necessary so it can bootstrap an actual stdlib from nixpkgs.
The trap is sprung once you've been sold on the idea, and even seen it shine through some things, but feel like you have an entire wall to climb to get to a point of general competency. All of these things I've described, and even some of their subgroups, require new knowledge for a more complete understanding. I do not understand the details of many of the things I have described, and I would consider myself a power user at this point. What looks like one gigantic, insurmountable cliff is really dozens of much smaller cliffs, many of which are not necessary, at least not to most people.
At one point I wanted to test my skills in a really generic way, so I helped fix broken packages on macOS during one of the two yearly regression periods where the community tries to get the CI builds passing. I picked the packages completely at random. Most of what I ended up fixing was C/C++ based, and I don't normally work in it, or I haven't for many years, but I feel like I learned a good bit about cmake. I don't really write Haskell, but I was able to fix a bug in a solving library having to do with a missing portability shim. I didn't have help for any of this, and in some cases I was either too stuck to fix the build or unable to fix it due to problems beyond my control.
All of this is to say that I think getting to this point is a lot less painful if one takes a use case, preferably a narrow one, and focuses on learning whatever is necessary in the service of that use case. Some good topics are dev environments, using nix as an app/project builder, using nix to build containers, using nix to manage remote machines (the difference between these and personal machines is that the scope of these machines tends to be much smaller). I think people try to take on too many of these things simultaneously and end up overwhelmed. That's not to say that NixOS isn't worth learning, but I think many people could realize most or all of the real-world benefit if they use nix on an OS they're comfortable with. Coming back to my comparison of being dropped in the Romanian countryside, if ...
mixing arm64 and x86_64 on an Apple Silicon machine:
{
x86_64 ? import <nixpkgs> { localSystem = "aarch64-darwin"; },
aarch64 ? import <nixpkgs> { localSystem = "x86_64-darwin"; }
}:
let
foo = aarch64.foo;
in aarch64.mkShell { # this makes nix-shell drop to an arm64 shell, change it to x86_64 to be intel/Rosetta2
buildInputs = [
foo
x86_64.bar
aarch64.baz
];
}
Not as bad as the guix crowd IMO. They're like the nix/Rust "just use XYZ" crowd but also combined with the GNU zealots so you get the perfect overlap of annoying software purists.
Really, I like the idea and I actually like the fact that they use Guile (scheme) instead of a home-built DSL, but guix is a horrible, poorly documented, doesn't work even on the happy path mess.
"poorly documented" is a new one for me. It's got one of the best manuals out there. A little short on examples maybe, but there's a separate cookbook with tutorials, and lots of blog posts.
The documentation appears good, until you fall off the already very narrow happy-path to just get things working. I can't even get the hello-world program to work following the instructions exactly on a clean Ubuntu VM.
I didn't mean to imply everyone who uses it is a GNU zealot, but there's definitely a subset who are.
Many things in the world are possible. I often find probable to be a more useful concept than possible. Any tool with deterministic results (such as Nix) are highly likely to reduce a whole class of future problems.
With this in mind, the question shifts from rather vague talk about possible futures to various kinds of scenarios.
What happens when software developers have a deterministic build system? On the whole, the benefits are significant. Sure, there are costs to get there. In my view, the benefits often outweigh the costs.
Thanks for your thoughts. I worded my question poorly, I suppose, but you got my meaning.
I keep seeing nix mentioned alongside complaints about it's maturity but that seems par for the course for something new, doesn't it? Until a tool reaches critical mass sharp edges are to be expected.
I'll look into nix again when I have an appropriate use case.
> Many things in the world are possible. I often find probable to be a more useful concept than possible. Any tool with deterministic results (such as Nix) are highly likely to reduce a whole class of future problems.
I disagree with the whole "highly likely" bit. Just because something new has a large positive value, that does not mean that a large time-investment or effort-investment in it is likely to pay off.
After all, it may never get traction to make the cost of learning it worthwhile - some other, newer, better, easier system could come along and be adopted instead.
We've seen this play out again and again in the technology world[1]. In 2000, a large time and effort investment into UTF-16 would be wasted as the world adopted UTF-8 instead. A large time and effort invested into new languages with powerful features (Haskell, circa 2005) would be wasted as the industry largely ignored it in favour of (presumably) inferior alternatives.
Is nix highly likely to provide solutions to a whole class of problems?
Sure.
Is it highly likely to be the dominant, or even a common tool, in the future that solves those problems?
Doubtful; "easier-to-use" has always won against "can better solve problems". I see no evidence that this will change anytime soon.
[1] I've found that by procrastinating about learning a new technology, I can sometimes manage to avoid having to learn it altogether, because the tech dies off before my procrastination does. By putting off learning VRML in 2001, it went away before I ever got to it.
Likewise, I'll put off learning new things until I actually have a need for the solution they offer. At least then, even if they die out in a few years, I'd have at least solved my immediate problems.
> I disagree with the whole "highly likely" bit. Just because something new has a large positive value, that does not mean that a large time-investment or effort-investment in it is likely to pay off.
You are disagreeing with something I didn't write.
> Is it highly likely to be the dominant, or even a common tool, in the future that solves those problems?
Doubtful; "easier-to-use" has always won against "can better solve problems". I see no evidence that this will change anytime soon.
The way you've framed the question is notable. The argument structure ignores a common pattern in software: tools steal ideas from each other often.
For example, MySQL is very different than its 2002 version. There are more database back ends with various guarantees and performance tradeoffs.
> Likewise, I'll put off learning new things until I actually have a need for the solution they offer. At least then, even if they die out in a few years, I'd have at least solved my immediate problems.
Your choice of language: saying 'need' suggests a black and white decision. See what I mean?
Costs and benefits often cannot be estimated accurately 'in one go'. It is common to spend some time and money to increase understanding and thus reduce uncertainty.
What deterministic build systems have you used? If you have not, how confident should you be that you can accurately assess their costs and benefits?
PS. Your comments suggest to me that, like most people with experience, you have become overconfident. To disabuse yourself of this, I suggest you find a prediction market and make some bets. You'll find you have to make your claims testable. Make a specific prediction/bet on Nix and let's see what happens. Let us know how it goes.
As someone who has been using asdf daily for over a year now I can say from experience it's awesome and should absolutely replace all individual language managers as the standard way we manage language versions.
I recently gave nix a try because of a hacker news comment and agree with most of the points in this comment. The fact that new users are encouraged to read not 1, but 3 separate manuals to grok the tool is extremely discouraging for new users. Also the fact that nix is currently transitioning to a new feature called flakes and deprecating channels signals to me that the ecosystem is not stable enough for me to recommend that my team at work use it right now.
That being said, I intend to read at least 2 of the manuals and use nix for dependency management in my side projects and to manage my system dependencies. I think nix takes all the great things that asdf can do, to a whole new level. The dream of having a nix section in my projects 'Getting Started' that only has 2 steps (1. Install nix 2. Run `nix-shell`) is something I desperately want.
I think nix is much like erlang in that it's incredibly powerful and solves a lot of common problems, but it's used and maintained by an old guard that went through the trouble of learning how to use it and maintains the "I did it, why can't you" mindset. The erlang ecosystem has been given new life and been made easily available to new users thanks to the awesome work done by Jose Valim and the rest of the elixir team and community. However, I really hope that it doesn't take 30 years and a new language built on top of nix for it to become accessible to everyone.
For those working on nix and the new flake functionality please look to things like elixir and asdf for inspiration on documentation, ease of use and pragmatism. Help make the tool you love ubiquitous. It will ultimately make your life easier when working with new developers because they will already have nix installed.
I intended to do everything I can to help push nix in this direction (once I understand how the hell it works) if anyone reading this, has some time on their hands and wants to improve the state of software development for everyone please go down the nix rabbit hole and try to find ways to contribute to making it more accessible.
Same, nix seems like such a perfect solution- but it's prohibitively hard. Sure, I could use it and sink 12 hours into reading the documentation, or I could just install pyenv/nvm/etc and be done in 1 minute.
Look at Python with Nix: https://nixos.wiki/wiki/Python
And then consider the simplicity of `pyenv install x.x.x` and a requirements.txt.
One place where asdf wins in convenience is when you actually really care about versions though. It allows you to install both new and old specific versions. In nix, while you can achieve that, you'll be guided to use the currently supported major versions like python 3.8, 3.9. Making sure you build with exactly 3.7.1 as well as a module which was merged to nixpkgs later is a serious pita.
And you can do that in theory, but in practice it ends up like this:
- you want to use lang-x.y.z, so you'll just reference an old commit
- turns out that version was never packaged, so you copy x.y.z-1 to your flake/overlay
- you end up patching the build because it doesn't work by default with new environments
- on the other hand a module you're using needs the new environment so you do your best to make everything happy - more overlaying
- because you changed something crucial in a few steps, you're compiling llvm 20 times through this process and now it's 1am
Compare that to putting "rust 1.59" in your .tool-version and running "asdf install". Guess who tried to use 3 specific versions of rust at the same time recently and got very familiar with the process...
(That's also ignoring the issue like Ruby being packaged without the platform identifier in nix which makes it unusable for some development purposes)
Yup, the versions of the toolchain components required for a project to build should be specifiable at the project in an format actionable by the main build tool. Running a project should never require more than "git clone $project && $build-cmd".
It should never be Nix's business (or any other OS) to provide elements of a project's toolchain. Reciprocally, projects should go out of their way to not depend on OS installed toolchain components.
> It should never be Nix's business (or any other OS) to provide elements of a project's toolchain.
Why? I used to think the same way, but that's because of the scar tissue of all the tooling being terrible at accepting that build concerns and packaging concerns are one and the same.
Nix proves it and makes this ancient pain a non-issue: "My package depends on x, y and z, additionally, building it requires a, b, and c, please also make build dependencies available in my shell while I write code too". That's it.
> Running a project should never require more than "git clone $project && $build-cmd".
That's precisely what Nix allows you to achieve. Meanwhile doing this:
> Reciprocally, projects should go out of their way to not depend on OS installed toolchain components.
Means that you can never depend on anything you're not vendoring (read: maintaining a copy of, on your own). Or it means sacrificing productivity in the altar of shitty tools. Nix allows you to have your cake and eat it too.
Shared projects should not depend on a particular package manager. You may use Nix, but the next person might use something else because reasons. If you bind a project's lifecycle to a specific OS you'll be trading your convenience for those of others.
Nix, the package manager, is not an OS nor is it locked to one distro: it runs on any Linux distro, it natively runs on macOS, and it runs on WSL.
It is also not a package manager like "apt", "dnf" or "pacman" as it does not fill the same niche. It does not even have to fill the same niche on every project.
As an example: at my job we're using it exclusively for optionally managing shared developer dependencies. Everyone is free to manage their own dependencies if they don't want or can't use Nix. But a flake.nix is there so you can get everything configured for you automatically.
In a personal shared project of mine I use it for tracking developer and build dependencies, meaning that I can build the project through Nix without having any of the dependencies specific to this project installed globally on my machine, meaning I can work on other projects that use incompatible versions of the same library, and keep the project reproducible.
On my desktop and laptop I use it as part of NixOS (which is an OS), to configure both from a single, mostly-shared configuration I keep in Git, replacing a mountain of terrible Ansible and shell scripts that did not work even nearly as well.
You can also use it to generate Docker images, operating system ISO images, distribute your app as an AppImage, Snaps,... and many other things. And you get an actual programming language to do all that with all the tooling the community maintains for each usecase.
If you're just using it as an asdf replacement, as the comment you originally replied to compared it to, your project will not be any more locked in to Nix than it would be to asdf.
I see your point and will give the concept of Nix-as-a-build-platform-manager some more thought when imagining the "ultimate build system", something which comes to me every few weeks.
I think you're missing the point of nix. Unlike other systems, yes you can make it mix's business to provide that. And the threshold for "OS installed" becomes blurry enough that projects can include the definition of what they depend on from the system.
I don't believe there's any non-nixos project so integrated with nix that it's required. You'd need to show some example here. It's normally just a wrapper for the description of how to setup the system - which you can totally do on your own and that's how most projects roll: just a text description of dependencies / versions.
If it's Rust, you can use https://github.com/oxalica/rust-overlay to get any version you want very easily without pinning an instance of nixpkgs just for it.
asdf does not allow you to keep three different versions of the same language, so I'm not sure how that compares? It's not super-trivial to do in Nix, but at least you can do it.
asdf is also no different than Nix when it comes to minor/major versions. You're at the mercy of what the plugin does, other than that you have to create your own plugin from scratch or make a fork. Nix has the option to patch things up more easily at least.
> asdf does not allow you to keep three different versions of the same language
Sure it does. You can have any number installed and either switch by changing the tools version file, setting it for your shell, or referencing the right path directly.
> You're at the mercy of what the plugin does
Again theory vs practice. In theory, sure, it depends on the plugin. In practice, almost every plugin lists every single released version to choose from. Which is much different from nix where the official choices are limited and you have to work to enable the rest.
> You can have any number installed and either switch by changing the tools version file
Oh, that's cheating :) Nix can do the same thing! Derivations are just kept in the store until you garbage collect them. You can also configure multiple versions at the same time to present you with multiple binaries, which is what I was referring to.
> Again theory vs practice. In theory, sure, it depends on the plugin. In practice, almost every plugin lists every single released version to choose from. Which is much different from nix where the official choices are limited and you have to work to enable the rest.
In practice, nixpkgs supports every maintained version of most tools. Certainly far more tools than asdf can ever hope to maintain. If not, there's an overlay or flake out there to support it. Can you come out with specific examples and a usecase?
I just tried but is really a pain in macOS right now, all went smooth until I got an error while trying to install the bundler gem, I skimmed quickly GitHub and seems to be an issue of macOS and the root dir being read-only, I suppose is too much hassle to worth it, at least if you compare with docker.
As others pointed, the idea behind nix is cool, but it sometimes fall short, hope it get ironed all the issues in macOS so I could give a try in the future.
Were you trying to install the Bundler gem as root? If that is the case, I'd recommend against it even without Nix. In any case you can find Bundler in Nixpkgs instead:
Same, tried Nix on my macOS (MBA M1) back in January. The installation is painless but it ran into an issue in the first run after it. It looks like that I skipped a step somewhere, I was following the guide and it should be working but its not. I suspected it was the guide that someone forgot to include it in the documentation or edge case I experienced. Now the uninstalling part is painful because there bare minimum documentation about this. Tried all different way to remove it and I couldn't. Took me a month to finally remove Nix from my macOS, I found the information through SO.
Yes! You can swoop into any thread discussing other packaging / build tools, and recommend nix, immediately sidelining any conversation about the original topic.
The advantages are more pronounced for DevOps than for developers.
e.g. for DevOps, Nix allows copying a package of software (and all its dependencies) from one computer to another, with confidence that the programs will work on both machines.
For developers, Nix is more appealing if you ever work on more than one computer. e.g. with Nix, it's easy to ensure you've got the same version of packages installed regardless of whether you're on macOS or Linux.
It's pretty neat for side projects, in order to get "I was able to build it now, I'll be able to build it in 6 months". Whereas, if you're otherwise relying on Ubuntu's system packages, maybe the package versions got updated, and so you'll need to spend time to get your program to work with the latest compiler/libraries.
Here's a use case I pose every time Nix comes up in these discussions.
For $REASONS I want python 3.7.10, and the full scientific stack (numpy, scipy, pandas, numba, matplotlib, ...) at the latest compatible versions along with pytorch. I don't mind waiting for things to compile. How, if at all, can Nix help me get into an environment that meets those constraints? Can it help without requiring me to find and copy hashes of things?
Roughly, Nix's approach is to describe packages as a function of it dependencies as inputs. (And all the inputs have all their dependencies described in Nix, too). Nix doesn't do things like "install latest version of X". (One of Nix's principles is build reproducibility; the package should have the same behaviour if built with the same input).
Tools like mach-nix or poetry2nix wrap around nix.. as I understand, they adapt the requirements.txt or whatever and pass it into Nix in a way that Nix can understand.
I have a similar question and want to ask the nix crowd about an approach - how well does it work to have nix just manage the nativeish stuff like Python and maybe the compiler chain, but still use the language's package manager (pip/poetry) for language deps ?
Both for parent's quandary, and for when you're working with a group and don't want to force everyone else to use Nix.
I've used Nix. It solves an interesting problem. It is so complicated as to be unusable by someone who is just looking for a quick and simple solution to something like a virtual environment
There's a reason why, despite being almost 20 years old, Nix has at best a small cult following. It just doesn't solve the problems that most people are looking to solve, most other solutions are simply easier and in many ways better
My biggest gripe with nix is that you need sudo rights to get it running properly on another distro. I know you can do the nix-user-chroot thing, but it's not possible on machines where kernel namespaces aren't accessible for unprivileged users.
I spent time understanding it, and getting it working, but it really doesn't work just universally. Once the install process becomes more like, say, miniconda -- where an unprivileged user can install it locally with ease -- nix will get a massive boost in my books. Until then, I can only yearn.
It is possible to use from a completely unprivileged environment by putting the nix store in your home directory instead of `/nix`. But (and it's a big but) you'll then have to build all packages from scratch because the hashes for you packages will be different to the ones in the nix binary cache.
nix local package search sucks, and it is much more effective to use the online package search [1].
It's astounding because the local version is doing a naive, case-sensitive regex search, which is pretty bad when you're looking for, say, the Perl package MIME::Lite which can have various junctions and capitalizations. In fact, that package has a listed name of perl5.34.0-MIME-Lite but is installable as nixpkgs.perl534Packages.MIMELite!? I can't even
I found the code a few months back (but can't be bothered to look it up just for a comment). I didn't fix it then because I had other priorities than learning how to modify my local copy of nix to test it.
I know there are other comments about nix and I agree that its a pain to setup. It doesn't give you the ability to pin versions, and good luck if your dependencies are not available.
But, I would encourage people to try out NixOS. Its much better than the package manager, yet it follows the same basic principles.
- Uses a simple DSL language
- 1 config file describes your entire environment, you basically use nix the language to tell your computer if you want wifi enabled or not, or if you want sound, or if you want i3 etc.
- Its easy to roll back, since the config file is versioned (like git)
I personally use a mac machine, running nixos in vmware fusion and I can't believe how long I had been using a separate machine for development and another for personal use.
I get the best of both worlds with this setup; my messages and notes are here, I get world class hardware (come at me bro), and I get to control every single aspect of my development machine.
I still use docker inside nixos, and yes I know its almost supposed to replace docker and all but I will get there one day, right now is not it.
Areas for improvement: - They should have come up with a better name: nix, Nix and NixOS are confusing (language, package manager and operating system - in that order. Yes, they went with an upper case letter for the pkg manager) - Docs NEED to be better, this is not an easy task. The language is shared between every component. - Give playgrounds to beginners, I shouldn't have to download the minimal iso and boot it on my machine to get a sense of how powerful it is. There has to be an easier way.
End of my rant. I started using it a month ago, I have run into issues with dependency pinning but I remain optimistic.
I have tried Nix, and am actually still using it for some simple things. But it is one of the most user hostile experiences I have had outside of some enterprise software. ASDF is significantly better when it comes to user experience.
is there a reason why this couldn't just be `nix shell python3`?
like "nixpkgs" is almost always the default, from what I can tell, so it could be omitted and simply be assumed to be the default, and not sure what the -c param does but perhaps it could also assume that the specified package(s) are also the ones to install/choose, in which case the whole thing could be like `nix shell -c python3`?
It seems nice. When programming envs tend too complicated though, I tend to switch to docker: a given image will handle a given version and it's mostly hassle free. That probably would not make sense on macOS or Windows, if you want to run native versions.
Same. Getting a python environment up and running can be a multi-day endeavor if you're unlucky. Just using docker compose on the existing Dockerfile for the project and mounting the local filesystem, means everything just works. PyCharm now works nicely with a remote interpreter inside docker compose.
I've even tried doing that with all the devtools we have. Like normally a developer here has to install gcloud, terraform, kubectl, apt-get lots of stuff, and then 3-4 internal tools. Then configure all of it, and of course some stuff will just not work on someone's computer. But creating a docker image including all that means that I can grab a new computer and be up and running in minutes.
Hijacking this: does anyone know how to get vscode to use a python interpreter inside a docker container for linting? I’ve resorted to using the docker container, but also having a python env so that I can tell vscode to point to the python interpreter in the env to get linting. Otherwise, dependencies come up unresolved.
I did some googling, but I’m unfortunately not experienced enough to be able to describe the problem succinctly to google.
The trouble with this is that the python environment is in the container - In step 2, id need to type path to the interpreter that it inside the container.
Although, now you’ve got me thinking what would happen if I mounted the interpreter inside the container to something local that I can point to? Might try this later, thanks for the help!
I’ve seen these and got it to work this way. I’m not too fond of working inside the container as it’s just a little more friction, but I’ve been working this way for a little bit now. I was hoping to avoid working in the container.
This. Docker not only solves the language-specific venv problem, but also encapsulates other system-level dependencies that may not be covered by stuff like pyenv or nvm. Plus with docker-compose you can trivially satisfy other, way more complicated dependencies like "fully functional PostgreSQL DB is accessible over the network", and you can deploy the entire thing to production with a few simple commands.
One thing that I realized after finally dabbling with Python after years in other languages was just how much Python needs Docker in development. There are so many details to get the right version of python working on a project because of system level assumptions that using Python without Docker is a huge pain.
Conversely, the experience of using Ruby within Docker for development is often clunky and awkward by comparison.
Installing dependencies, per the docs, before installing python works well. Installing library dependencies for packages also works well. It's not dissimilar to compiling c/c++ where you have to read the installation docs. And the docs better be updated.
Happy asdf user for several years now. Unfortunately, there are still cases where it either can't be used or it would take a great deal of hacky stuff to make it work. One such case is getting PyCharm to understand it.
I found it works well for Ruby, Elixir/Erlang, and Node.
And PostgreSQL too. I tend to use the PostgreSQL plugin for projects in which I use asdf. It creates the data directory somewhere and I can use that exact version of the database without docker. Then it's pg_ctl start / stop from within the project directory. The version coming with the OS is never the right one.
So is asdf good then? I have been using SDKMAN! to install JDKs, but i noticed that Gradle will automatically detect JDKs installed using SDKMAN!, asdf, and Jabba [1], which signals to me that the other two are worth a look. Although now i dig into it, asdf does not have a first-party Java plugin, so the support is through a third-party plugin maintained by three people and a robot [2].
Jabba never worked on windows for me. So now I just install jdks globally via scoop and have a powershell plugin that automatically detects a .env file in each directory.
TBH I've had a bit mixed experiences with asdf and java (mostly big enterprise apps like Oracle SQL Developer needing some hand-holding to work properly, and issues with http kit intercepting network traffic used by SoapUI).
Not sure the error lies at the feet of asdf, though.
For ruby/node it's a relief to be able to easily "activate" a certain version of ruby and a certain version of nodejs - at the same time, via simply "asdf local ruby ... and asdf local node ...".
I think of myself as a pretty experienced hacker but asdf has never really worked for me on any platform. It’s always a kludge. Love the idea, but I’m sticking with pyenv for now.
I actively dislike this project even though I've never used it because Common Lisp asdf is a de facto standard that's been around since forever. There's no way you fail to notice the name clash with even a cursory search. They're both software projects and they're both build tools it's scummy behavior in my view to say the least.
My first reaction was to see if the project is made with Commnon Lisp's ASDF, and cool now it has plugins for different languages. Common Lisp ASDF is very well thought out, but not the easiest package management system.
Yeah I was really wondering who found it easier to hack ASDF to work with other languages than just use a few separate package management solutions. It's an incredible library, but man I would not want to push its limits or it would go from odd to unusable pretty quick.
asdf is not a replacement for pyenv, rbenv, goenv and nvm, it's yet another abstraction on top of them. Nvm is a fork of rbenv for node, and pyenv is a fork of rbenv for python, the core of all of these asdf plugins are exactly ruby-build, python-build and node-build, which came from rbenv, pyenv and nodeenv. For go, you don't need any kind of environment, you can switch between versions with just a change of an environment variable. Also, a much simpler solution is to simply install these language runtime versions with your favorite package manager like Homebrew or Macports, and use direnv and dotenv to switch between them as you cd into different directories. All of these languages have convenient environment variables you can control to put the right things in your paths.
> asdf is not a replacement for pyenv, rbenv, goenv and nvm, it's yet another abstraction on top of them
No, asdf is a replacement for them. It's not an abstraction on top of them. It uses its own plugin system, and everything is written purely in bash. Everything gets installed in `~/.asdf/installs`
Their "plugins", are using exactly the same build plugins pyenv, rbenv and nodeenv use to build the language runtimes. All asdf does is set the point things to the right directories using various tricks like symlinks and shims and whatnot. This is something easily accomplished by direnv.
> asdf version resolution is slow which makes every command execution pay that penalty. asdf reshim is needed for finding new executables, and some tools are not happy with their executables being masked by shims.
> Perform asdf version resolution only once and defer environment loading to direnv.
Our infrastructure runs on Ubuntu 20.04 across the board but, for example, our main Ruby on Rails monolith runs Ruby 3.0, but Canonical only packages 2.7 for Focal. We have a legacy service which runs on 2.5 and a few microservices that run on 2.7. Having language versions tied to specific OS versions was a massive PITA for us in the past, now it doesn't even really matter all the much what OS is running the code. If we just used apt versions we'd be stuck on Ubuntu 18.04 for ruby 2.5 and we'd have to ugprade the entire application in order to get it to run on 20.04 once it goes out of support. Yuck
Nevermind trying to setup a development environment where you have multiple dependencies on different language runtimes.
I also prefer `direnv`. I am not entirely sure how `asdf` actually works; `direnv` is pretty straightforward: it hooks into the shell and modifies environment variables. (`PATH`, but also tool-specific variables if required.) So it's pretty easy to debug or replicate by hand if necessary.
It can't actually install different versions though (looks like `asdf` can?), which is somewhat inconvenient.
Just install them with Homebrew or Macports. They all get installed into different paths anyway and you don't even need to build from source on your system. Building python and node are major PATA.
Nit: nvm is fork of rvm. nodenv is fork of rbenv. The rbenv model works much better than nvm model because it uses PATH executables instead of heavy shell functions.
Regarding the suggestion to “do things manually” — what is the value of doing things manually? For me, it is very far away from how I want to spend my energy, which is doing computer things that only I can do. I am happy to outsource environment variable fiddle faddling to someone else.
Configuration is a necessity for any tooling. Direnv is something you likely already use with dotenv anyway to configure your CI/CD pipelines. All you need to do is to drop in a few more extra environment variables into your git worktree.
> install these language runtime versions with your favorite package manager like Homebrew or Macports
Package managers often don't have a way to install specific old versions of a package, or multiple versions of a single package. That's the whole point.
Yes there is, but may not be granular enough to the patch level, they usually end at the major or minor level. But then again, I've never ran into or heard of anyone that needs a very specific semver of a language runtime for a project. It could happen, and when it does, you switch to pyenv, rbenv or nvm. There's still not a reason to use asdf, as a poster pointed out, asdf version resolution is slow and you need to reshim, which is yet another manual intervention you'll need to do.
I've found asdf to be a reasonable choice when needing to manage more than one language - especially where there are good "native" version managers that asdf wrap (eg: python, ruby, node, to a certain extent rust/cargo).
I see comments about Nix and I've heard rumblings about it before, so I might look into that. But Docker covers my needs for version management completely. To me it seems much more explicit, more portable, less error prone, less complex than any of these virtual environment managers. Never mind that I don't even want node.js on my system if I can help it at all (I can't, generally).
This is a reasonable point, and something that is being worked on. I definitely think Docker can provide LSP servers as necessary, we're just not quite there. This is something that I'm interested in working on, though I'm not the only one: https://github.com/emacs-lsp/lsp-docker
For the most part, latest version linting is good enough for me, currently.
It's obviously doable, but it's very heavy for languages. Like someone else said, you immediately run into basic issues like readline, LSP compatibility, volume persistence, port forwarding, etc.
Docker is better for version managing services like databases, caches (Postgres, Redis, etc), things that have little need for different configuration or exploration and persist through multiple sessions. Once they're up, they're done.
Asdf is the next best thing for local development, where you need that kind of exploration.
The main issue with docker is that you have to start from scratch with your environment. With Nix you can layer it over your existing Linux environment (or you can have it from-scratch by running `nix-shell --pure`).
There are advantages to the docker approach, and Nix can be painful to get going, but a significant portion of the community (including me) find the pain to be worth it.
I’ve used ASDF since 2016 and it’s been working beautifully. The only hiccups I’ve had with it aren’t related to ASDF, rather stuff like Erlang 24.0 not compiling on macOS (which was subsequently fixed). Every engineer at my company uses ASDF
Same. We switched when we started an elixir project and pretty quickly switched all our codebases over. On top of languages we've used it for database systems in some cases.
asdf is indeed the best solution in the elixir/erlang world. I use it for node, ruby, and python now too and it's been great to have a consistent experience across languages.
I recently discovered Hermit[1], which is a really straightforward tool for managing versions of tools for a given project. From the FAQ[2]:
> Hermit is probably most similar to asdf, but their goals differ. Hermit's goal is to make isolated cross-platform tooling consistent, self-bootstrapping, and reproducible at the project level. asdf's primary goal is to allow developers to install and switch between multiple versions of languages and tooling.
The downside for the author of this post, though, is that Hermit doesn't yet support Ruby because it's not static/relocatable.
The lack of Ruby support instantly rings an alarm for me because CPython (on POSIX) also is not relocatable, but is listed as support. Turns pit Hermit is actually using a third-party build script[1] instead of the official one. While the python-build-standalone project is quite awesome and indeed useful for a lot of things, it has enough quirks I would recommend against any generic package distributor to advertise it as Python for general use. This in turn makes me lose most confidence on Hermit, unfortunately.
Be careful if you’re also interested in Hermit. These kinds of things bit you up way down the road when you least expect them to.
This feels like Trillian for virtualenvs. So typical that as engineers we keep abstracting everything we can until we run out of people that understand the basics. This tool looks like a great resource until step 4 when we create a virtualenv protocol. In chat clients, the protocols came first. For development IDEs adding auto-complete, it seems the protocol LSP came second. It's happening to language virtualenvs now as well.
1. You need a tool for X to solve an issue so you write it. (AIM)
2. Guy over there needs a tool for Y to solve the same issue, so he writes it. (Yahoo Chat)
3. So many tools exist, we need to merge them all into one. (Trillian, Adium)
4. Merging all the tools created a monster, we need a common protocol they all understand.
5. No one remembers how tool X works anymore, updates become more sparse.
6. Both X and Y are no longer popular, big company took over. (Slack)
Not sure the comparison makes sense in this case. It doesn't have to do with virtualenvs. It's just a tool to install multiple interpreters/language runtimes in parallel and switch between them in an interactive shell with shims. In this case there are multiple tools that do the exact same for specific languages (pyenv/rbenv/goenv/...), this one does multiple runtimes, that's all. You'd still use your language's tooling for everything else.
I use direnv myself, and it's great. However, for anyone trying it out I'd caution that it doesn't play well with conda - the Python package management system (see this open issue https://github.com/direnv/direnv/issues/326). It can be a dealbreaker for some.
Indeed, direnv and asdf integrates, such that PATH is set to the values exposed by your .tool-config. This avoids the asdf shim and makes executing the tools slightly faster. Good when you have to run them over and over in a script.
direnv is nice but it really makes me uneasy if it catches on for modern collaborative development.
You are giving people a _ton_ of trust when cloning a repo or pull request that has a .envrc which will be automatically executed by direnv. This file could do anything from steal your SSH keys to delete your entire home directory or worse. And the only thing preventing that from happening is the suggestion of "oh be sure to eyeball the .envrc file and make sure it looks ok, then run this command to enable it". People will just google for the first "how to I unblock .envrc file" and blindly run the command, then oops. Or a determined attacker will just obfuscate what they're doing so the .envrc looks fine but you failed to realize some esoteric bash-isms were actually invoking an obfuscated script and... oops. It's just begging for a supply chain disaster on a similar scale as npm's infamous issues.
VS code and some vim extensions have similar issues with automatic project-specific configurations that allow arbitrary code execution too, and neither have a good solution beyond 'just read all the code and understand it to make sure it's not going to hurt you'. It's not really a unique thing to direnv.
We really need a better model for collaboration and consistent dev environments. I think containers and docker/chroot/etc. environments with explicit opt in of your local resources, configs, files, etc. are a much more sane and security conscious path towards it. A script executing to setup my python environment for a project should never, ever have access or even know about my personal password store database for example (unless I explicitly decide it should).
Direnv maintains a local cache of the scripts it's allowed to execute. It will not execute a brand new script and instead ask you to manually run "direnv allow".
> And the only thing preventing that from happening is the suggestion of "oh be sure to eyeball the .envrc file and make sure it looks ok, then run this command to enable it". People will just google for the first "how to I unblock .envrc file" and blindly run the command, then oops. Or a determined attacker will just obfuscate what they're doing so the .envrc looks fine but you failed to realize some esoteric bash-isms were actually invoking an obfuscated script and... oops.
You need to be absolutely sure you've read and understand every single line in the .envrc file, and every single line in any script it may be invoking, before you run direnv allow. Otherwise you are giving code from the internet a blank check to do anything on your machine with your user account.
I personally do not think bash/shell is a sensible language to trust that you can quickly or easily determine from a visual check alone if a script might be doing nefarious things.
I use direnv to source environment variables - the only scripts I use it for consist of "export" statements. Anything beyond that and I'll be questioning it.
Sure, you can trick idiots into running malicious code, but I don't think you need direnv for that - any pip/npm install can run anything and so does curl|bash.
The pull request scenario is what scares me. Someone sends you a drive by pull request that has a ton of changes and you don't notice they slipped an obfuscated bash quirk into the .envrc (along with legitimate changes to the file) that invokes their obfuscated nasty script. You pull down the change, approve the envrc (looks fine!), and poof there goes your home directory or worse.
It seems like a lot of these virtual environment tools are designed to work around the assumption that packages are installed globally on a system.
I don’t see why this is so hard to avoid with per-project version pinning. I’ve never had this problem with Java.
This seems like a lot of effort and complexity to work around global package installation. Just stop assuming that packages are global and this would all go away. How do we know that? Because other languages don’t require this kind of workaround!
Java (well Maven) I believe does this the most clever and simple way. It does have a global folder of dependencies but will gladly stash multiple versions of the same library. Local projects just link to the right version. That saves downloads and allows for simple version pinning. And JVMs are backwards compatible so having a new JVM can still run bytecode for past versions with just a flag. Features like this are so much more valuable than anything that goes into the actual language syntax.
I come from the Java world and it's mindblowing to see that mature, widespread languages like Ruby, Python or JavaScript require all kinds of hackery to run properly. Dependency management with Maven is a breeze in comparison.
I'll agree that how to do things properly with Ruby isn't obvious when you're new to the language - just as it isn't obvious how to do things in Java (or other JVM languages) when you're coming from Ruby or another language.[0]
But Bundler doesn't work all that differently from Maven, in the end. Instead of running your commands through `./mwnw`, you run them through `bundle exec`[1]. Dependency management is straightforward and mostly hassle-free (the problems arise "only" when you use C extensions which, to be fair, is not that rare, but I don't really see a way around this in a language where you need C for certain fast things). In particular, Bundler by default[2] also stores dependencies centrally like maven to avoid duplication and then selects the appropriate version of a given dependency (unlike e.g. NPM which stores everything locally).
Rbenv (or alternative tools such as chruby or asdf) solves a problem completely distinct from Bundler or Maven, it's about selecting the right Ruby version for your project, a problem that also needs to be solved with Java. IME, many developers just let the IDE deal with that, but Ruby developers are more used to running things from CLI, so that's why rbenv exists. In any case, the most popular analogue for Java would be sdkman which works, mostly, the same way (and which, IME, is even a bit hackier than rbenv).
JS and Python are different. Node is a mess IMHO, and Python isn't able to agree on which package manager to use (I would hope people would migrate to Poetry, which seems to be doing things right).
[0]: For example, one thing that I find very annoying over in Java-land is that it's a real hassle to make sure things behave the same way when called from the CLI as opposed to from the IDE, such as setting the active spring profile. This is because, mostly, the IDE circumvents maven/gradle (you can run things through the build system, but typically the output, e.g. test output, is better when using the IDE-internal system, at least with IntelliJ).
[1]: There are "hackeries" to get rid of having to run everything through "bundle exec", such as binstubs and direnv, although I would mostly call them "shell script tricks", and anyway, if you instead just alias "bundle exec" to something shorter or create a wrapper script, it's not really worse than `./mwnw`.
[2]: This can be overriden and is actually a good idea for production builds, especially if you use a multi-stage docker build since then it's easy to copy over all your dependencies.
I think you're missing that Maven does actually solve the rvm/nvm/pyenv problem. You can specify the bytecode version in the config and compile/run any backward compatible version. Your actual JDK need only be new enough, not an exact match and it doesn't need to change between projects. I have never had a single compiler or runtime version compatibility issue even with native dependencies ever with Java. The IDE situation can have some subtle edge cases, but no other language (save C#) actually gains so much utility from an IDE. Stack traces, break points, runtime inspection and profiling are all nearly flawless in IntelliJ (or Visual Studio) in ways that are monstrously difficult with any dynamic language.
Yes, but now you're not comparing maven vs. Bundler, you're comparing statically vs. dynamically typed languages. It's almost impossible to get the strong guarantees you get in a statically typed language in a dynamically typed one.
If that's the discussion you want to have, well, I personally also prefer statically typed languages, but that's beyond the point.
That said, I've had issues, e.g. on CI, that didn't manifest locally because of a mismatch in Java versions (something with JAXB, IIRC), and Java is maybe sometimes a bit too cavalier about the issue.
By Ruby, Python and JavaScript hackery I was meaning rvm, venv / pip, and npm. I have the most experience with rvm, and the hackery part comes from the experience when I created a web interface that could instruct the server to do ruby tasks. Turned out that rvm (and also bundler) does a bunch of stuff to the environment of the user, that's sometimes inaccessible, even if you invoke a "login shell" programatically. One of these "hacks" is doing a bunch of stuff when the user navigates to a Ruby project directory with cd, which is when rvm creates or uses the existing virtual env for the project - named is a specific way based on the directory name. I solved this of course but the discovery was quite annoying.
Regarding Python, I don't have much experience, other than the general feeling that it's fragile and that specific projects work with specific dependency managegers / virtual environments, and they are not really interchangeable or anything. I either could replicate the env or I couldn't. I definitely don't have much of a Python experience.
Now regarding your Java IDE annoyance, I share the same experience. Also the case with Tomcat. IDEs do so much hackery so that user experience is convenient, that you would be hard pressed to replicate the results, and often need to test both because it just doesn't work the same way, due to the different dependencies used, deployment method etc.
Now that I think about it, every single one of the programming ecosystems I used were quite annoying.
> By Ruby, Python and JavaScript hackery I was meaning rvm, venv / pip, and npm. I have the most experience with rvm, and the hackery part comes from the experience when I created a web interface that could instruct the server to do ruby tasks. Turned out that rvm (and also bundler) does a bunch of stuff to the environment of the user, that's sometimes inaccessible, even if you invoke a "login shell" programatically. One of these "hacks" is doing a bunch of stuff when the user navigates to a Ruby project directory with cd, which is when rvm creates or uses the existing virtual env for the project - named is a specific way based on the directory name. I solved this of course but the discovery was quite annoying.
Keeping the "right context" when you do things in different directories with the same shell config is, unfortunately, a problem that I don't think is easy to solve. My preferred approach is to use direnv (https://direnv.net/), but that's not perfect either.
As for rvm, I think that a lot of people at some point discovered that rvm was maybe doing a bit too much (in particular, it would also manage gem sets, something that seems superfluous with Bundler). I think that's a reason why rbenv and chruby emerged as lighter-weight alternatives which are a bit less magic. I haven't seen many people use rvm anymore in recent years. By contrast, Bundler doesn't really do anything to your shell.
> Now that I think about it, every single one of the programming ecosystems I used were quite annoying.
That's something I agree with. As a more recent convert to the JVM world (and mostly just for lack of better alternatives), I'm still sometimes annoyed at the IDE-centrism (and UI-centrism, see also sonarqube) in this space. Luckily, things are already better than they were when I learned Java in university and "it runs in my IDE" was the default state of affairs...
Thanks for the suggestions. I'll look into direnv, as I only heard about it, but never really investigated.
If people don't use rvm, do you know what they use to install a specific Ruby version? We basically use it so that we can use a Ruby that's not bundled with the distros anymore (large legacy project).
2. For ensuring Maven itself is run with the correct JDK, it is a commom recommendation to use the Enforcer plugin to specify the required JDK version and/or vendor, as well as the Maven version:
I only spent two years in contact with Java. I was initially very pleased with how easy dependency management was but it soured over time.
I'd respond to production disasters with hacky bash scripts and then we'd spend the next sprint incorporating that functionality into the server so the hacks weren't necessary. We kept catching ourselves working really hard to optimize what we thought at first needed to be custom code but what ended up being easily achievable by leaning on well-known non-java tools like `diff`. There's something about the way Javs encapsulates everything into a JAR that discourages Java devs from using anything on the outside.
If you can live in a world where everything you need is in the JAR, life is good, but that hasn't frequently been my world.
I spent many years working with Java and never used bash for deployment. Maven covers all the bases. And when you have something really custom, you write a maven plugin. Which can also be managed with Maven.
Oh the bash scripts had nothing to do with deployment, they just fixed the data that the buggy deployment created.
It's only relevant because the high level of encapsulation that made deps easy to manage also represented a barrier. Using diff and awk was fast enough, the reimplementation in java wasn't (happened just a few times). Language ecosystems with a more permeable encapsulation style make it easier to reference tools outside of that ecosystem at the cost of making it harder to get your deps straight.
I’m in the same spot. I’ve spent my career in Java and as I’ve started to enter the modern JS world through using React in projects… wow.
I was amazed to find such a mess. Each project has full copies of all sources of all modules you use? They’re not shared in a global cache? There’s just hundreds of thousands of individual files?
Yarn does a better job on some of this, but it all feels like stepping into an alternate reality where no one has ever heard of Maven.
Maven predates Node by 5 years. Even if Node was developed without knowledge of Maven surely someone has brought this up in the last 13 years, right?
It’s all very surprising to me, and I guess an interesting lesson in “parallel” evolution.
It's less about package versions (which are indeed versioned on a per-project basis in sane ecosystems - python being a notable exception) and more about versions of the language toolchain itself. e.g. you might want one project on Java 17 and another on Java 8. These tools allow you to seamlessly switch between those.
Java tooling already provided that and it’s trivial to install multiple JDK versions on a system. I probably have 4. Most IDEs have a dropdown to pick the one you want, there are CLI tools for it too, and overall it’s about as complex as a $PATH variable.
It’s also possible to install multiple versions of Python on a system. The dependency management is the real problem — having more than one version of a runtime is not the hard part.
If Java provides good first-party support for this then it may not make sense to use asdf for Java. Rust is similar in that rustup is first-party the recommended tool to install Rust with. I don't bother with asdf for Rust. Other ecosystems like python/ruby/javascript don't provide this as a first party tool meaning there are lot's of 3rd party alternatives. For these ecosystems asdf is nice in that it allows you to use one tool across all the ecosystems.
> The dependency management is the real problem
I'm not sure how this works in Java, but in many ecosystems like JavaScript and Rust dependency management isn't an issue at all. Installing a package defaults to a local install and even things like dependencies transitively depending on different versions of the same package are supported out the box and just work.
What do you refer to by saying Python is an exception? I haven't used anything but different language and package versions on a per-project basis since around 2007. I had some project with deployments on client servers where their sysadmin insisted on having everything installed as os packages, but this at least from my perspective hasn't been the norm for quite a while now.
Not to say that I like the Python package management story.
Commonly used tools like virtualenv allow you to do this, but it's not the default in the python ecosystem: `pip install foo` will install foo globally. In most other ecosystems the equivalent command (e.g. npm install foo) defaults to a local install with no effort on your part.
It's not a virtual environment tool, though. It's a tool to manage installed runtimes and switch between different versions in a shell. You'd still manage your virtual environments in whatever way your language prescribes.
There is an issue that delays commands by 200ms, which isn't a massive issue for my usecase, but you might want to look into asdf-exec https://github.com/danhper/asdf-exec
I like the idea. A package manager that can be used on several languages but simple enough to pick up in 15mins or less.
Just today on trying to get rvm to do something I got angry at the number of language-specific package managers I have installed: nvm, jenv, rvm, rbenv(after I couldn't get rvm...)
Of course, I know about Nix but it is too complex for my use cases.
Tried asdf but can’t stand the global shims. I only want it to work on a per-directory basis with direnv. I tried to follow the asdf-direnv instructions but IIRC still wound up with asdf’s shim directory in my global PATH, and it needed to install its own copy of direnv. Disappointing that asdf expects to take over my system like it does.
305 comments
[ 4.3 ms ] story [ 263 ms ] threadPersonally that's not something I can live with as these binaries are sometimes launched many times in subprocesses.
They just released a major new version (asdf-direnv), so might be worth giving that a shot and see if it works for you with asdf?
To be clear since I think it might not be obvious: lapser is talking about asdf-direnv, not direnv by itself. See https://github.com/asdf-community/asdf-direnv
For me, it has come so far that I don't have any interpreters or compilers in my main OS environment and even for one off REPL sessions, I'll use `nix shell nixpkgs#python3 -c python3`.
[0]: https://nixos.org/
[1]: https://github.com/nix-community/lorri/
[0] https://guix.gnu.org/
Mostly I think Nix would be just as hard if packages were expressed in any language.
Stuff like "this program compiled by nix is using a different glibc than this library" is going to be confusing regardless of whether Nix expressions can be written with JavaScript; or stuff like some program assumes it can write to $HOME in its build script not playing nicely with Nix.
A lot of why Nix is hard is because it's weird, and that even as an end user when things go wrong it may require understanding more about what's going on than what other solutions require.
(Tongue-in-cheek, but I'd actually like to see this happen: Nix is a fantastic concept obscured by space-alien syntax.)
https://lists.gnu.org/archive/html/guix-devel/2022-03/msg001...
https://www.tweag.io/blog/2020-10-22-nickel-open-sourcing/
As for Windows support: It seems to be possible to use nix in WSL [0] but I've never tried that. I have used nix alongside apt and pacman on Ubuntu/Arch before I decided to go all the way and install the NixOS distribution. Using it alongside other package managers worked really well.
[0]: https://nixos.org/download.html#nix-install-windows
I've used Nix on Linux (i686, x86_64 and ARM), macOS and Windows (WSL2). I believe it also works (although not officially supported) on BSDs, etc. too, although I've not tried that yet.
> Lorri sounds amazing though! ...which handles the "this directory needs these specific packages" installed locally, not globally problem that Debian gives you).
You're describing Nix, not Lorri. Specifically:
- nix-shell provides a directory-local environments (and more, e.g. #!nix-shell to fetch script dependencies automatically)
- direnv can enter/exit this nix-shell environment when cd-ing in/out of a directory
- lorri makes direnv+nix-shell faster, by building the environment asynchronously in the background (rather than causing our terminal to freeze)
The biggest drawback is installing Nix on OS X is a huge pain. Docker has lots of magical UX tooling to make it rather seamless in comparison.
Did it on an aarch64 mac the other day. One command (two?), 15 minutes, done.
I don't understand the concept/motivation behind it, everything that nix does should be able to be done in userland, like podman does, which I love.
The daemon is not powerful at all, almost all the logic happens in the client.
The client does the parsing, job ordering, and tells the daemon exactly what to do.
The daemon, only required in multiuser environments, does only two things really.
- ensure that the /nix/store is protected as the daemon should be the only process able to _write_ to it. Everything can read from the nix store; no nix required at all.
- execute build commands in a sandbox. Providing isolation between multiple builds.
I'm not sure if there are other reasons.
Be prepared to sink dozens and dozens of hours learning the language, it’s features, and switching everything over. Be prepared to get stuck when you are required to use an old version of a language and things don’t install correctly — and sometimes blocked by nix itself from installing it. Be prepared to get stuck if you are not in the happy path. Be prepared to get lost in the sometimes scant documentation.
I love the idea of nix, but it did not work for me except for the simplest cases.
I've found it to be quite flexible though. For example, here's a commit in which I apply a patch to a tool to solve a problem that the derivation hadn't taken into account (and absent a home-manager solution): https://github.com/samhh/dotfiles/commit/867dd3b4d4b3942a0aa...
I understand everything in that sentence except the word "so".
I really like Nix but in this particular way it feels like taking a big step backwards from the other languages I frequently use, particularly for a language in which you'll necessarily be constantly interacting with bespoke interfaces.
I guess knowing allowed types for each parameter is informative (more if very specific types are used rather than string or integer), but documentation usually specifies that anyway. And even knowing all the types isn't usually enough without documentation.
My point is just that I've used plenty of libraries in dynamically typed languages without needing to read the source. And conversely, I've occasionally needed to look at the source of a function in a staticly typed language, to answer some question not answered by the types and documentation.
Haskell is the ideal here, since it separates the type signature into its own separate line at the top of the function declaration. That confers readability and reduces cognitive overhead more than any other language I've used.
It also changes the programming thought process. You can pseudo-code an entire program just with function type definitions. Then test that the type defs compile without error, and go back and implement the function definitions. As long as the function definitions adhere to their type signature, the program almost always works (barring some I/O errors).
Not all statically typed languages are created equal. The ones fundamentally oriented around function type signatures, rather than variable types, are the ones I think parent had in mind in his comment.
Like you, I find Nix hard to use and the docs confusing (beyond trivial examples). The amount of time the Nix mafia spends proselyting could be better spent making better docs
I've never heard of asdf and popped in here to read about it, people's experiences with it, etc. If I want to read about Nix/OS I'll find a thread on that, or go to /r/nixos, etc.
And what are you doing :-)
Oh I think you're using HN wrong. To learn about the subject you can read the actual article that's linked to. No guarantees about the comments section.
But, since you're downloading & running code written by someone else, it seems weird to talk about that but ignore "what if this program I'm downloading does something malicious".
[1] https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...
The reality is that the docs are fine. The reference material is comprehensive, and there are some very good tutorials and introductory essays to get people started. The problem is that it's difficult. Nix is very, very different from what people already know, and it takes quite a bit of effort to learn how it works. Most people just aren't prepared to do that much work just to install some software.
Most pilot text books are absolutely horribly written, finding a good one takes significant effort. Reading just an ok book shines a light on how absolutely trash the majority of pilot text books are and that it is absolutely bad "documentation" that is partially to blame for it being difficult to learn.
Just like inherent complexity in some software or business problems.
Maybe airplanes wasn't the best thing to compare with -- the comment below compares with C instead, maybe that's better. I never flew a plane.
It helps that people who are into Nix most often use NixOS. NixOS is the only distro I've written packages for. It's not even particularly hard since there's a lot of inspiration to find in the official and community repos + people sharing personal configurations.
The UX wasn't the best but it's improving.
It remains extremely reliable which is not something that you could say of other systems.
Docker is more pragmatic unless you spend hundreds of hours on Nix, in my opinion.
then show me where `nix-env`'s `--arg` and `--argstr` is documented.
https://nixos.org/manual/nix/stable/command-ref/opt-common.h...
But it's no wonder that people keep commenting that Nix is an awesome idea that solves a lot of problems. That's because it's an awesome idea that solves a lot of problems (yeah, like Rust).
It is also completely undiscoverable (not completely unlike Rust), and that's why almost nobody uses it. But that's a matter of improving the documentation or maybe fixing one or two superficial problems on the language.
Based on a quick check of their Wikipedia page, Nix has been around for 15 years, since 2007.
So those for sure look like structural problems. I don't know which ones exactly, but paraphrasing Tolstoi, popular software products are popular in much the same way, while unpopular ones are unpopular in their own unique ways.
In my experience almost no open source project that hasn't entered the mainstream in its first 10 years manages to turn the ship around, unless it lucks into a major change of environment. Programming languages are sometimes exempt if they have a "killer" library or framework pop up.
HN tends to see success in terms of adoption rate & ranking. Nix is on another plane. It's beyond critical mass. And for people like me, it is a huge force multiplier. But I had to grind my way to the point where that is true.
I tend to not bother selling people on it. People who want what Nix does will find it at some point. So I just focus on using Nix myself in a world of worse tools. Feels great.
That said, I haven't found Nix hard to use, quite the opposite. The new 'nix' command is pretty self explanatory, it's just a little incomplete in spots. And the Nix language is quite simple and easy to understand if you have ever touched a functional language. Compared to all the other Linux stuff I have played with over the years, it was a very pleasant experience so far.
Evidence of that is that you can just talk about it by name here on HN without explaining what it is.
Is Haskell mainstream? What about committing murder?
And yes, this is 100% Nix's fault
I would say that HN is far from mainstream.
Nix provides a level of flexibility that other solutions simply do not offer. If a package in Nix doesn't fit your requirements for whatever reasons, you can create a modified version of a package with ease. For example, say that you need a version of Nginx built against a custom version of OpenSSL. You can do just that with a few lines of code:
Nix will even know which packages it'd need to build locally instead of downloading a prebuilt binary.You can't do that with other common package managers. You're stuck with whatever the package manager provides you. So if you're not in the happy path, you're out of luck. You'd either have to give up or build from scratch. If you need a bugfix for a particular package, you'd have wait until the fix reaches the package repository. When I was using Ubuntu, that was often until the next major release. None of this stuff is a problem with Nix, which allows for customization with very little effort.
It's easy to learn about Nix if you know where to look for. Nix Pills [1] would be a good start. The core language is the easiest part. It's JSON, but with functions and variables for proper abstraction. The documentation, while imperfect, is quite extensive compared to a majority of other high profile open source projects. If the documentation fails you, the Nixpkgs repository [2] is an even more rich source of information. The code is well organized, and I was able to get familiar with writing Nix packages fairly quickly by grepping the codebase. And finally, Nix has a sizable community so you could always ask if you're stuck.
[1]: https://nixos.org/guides/nix-pills/
[2]: https://github.com/NixOS/nixpkgs
I've been using nix as my daily OS for six months and I'm able to be productive. At the same time, I'm often confused as to whether I should use nix-shell, nix-env or modify my configuration files.
Thanks for the great comment.
I understand you are saying it with the best intentions but it might sound like not the most correct way of defending a tool being criticized for its lack of documentation or easy of use.
Ultimately documentation doesn’t solve usability problems it just makes note of them.
Also rollbacks in nixos are amazing.
On the other hand newly packaging something can be really daunting.
Nix provides better abstractions and composability to make that experience far less painful. Note that in my previous example, I didn't need to specify how to build OpenSSL or Nginx. I was able to reuse the definition from Nixpkgs and apply my own customizations to it.
That's less of a guarantee with a collection of shell scripts.
Here's the simplest example of an utter failure when I tried nix: installing ruby 2.2. It's very possible I did something totally wrong, but when I have to use ruby 2.2 and nix cannot do it, it sort of kills your awesome example of being able to override openssl.
Another extremely frustrating experience was the bifurcation between nix and nixos. Clearly one experience is preferred over the other. So many blogs and docs talk about things like HomeManager and other configurations that are nixos specific.
On top of nixos/nix, there's also shell.nix/flakes. There's just too much development in nix right now to make sense of what's going on.
Considering that ruby 2.2 was released in 2014, I can't say I'm surprised. It's too much effort to continue maintaining something so old. But if you really want to use it, you could try using the package definition from an older commit of Nixpkgs:
> Another extremely frustrating experience was the bifurcation between nix and nixos.I'm not sure what you mean. Nix is the package manager while NixOS is an entire distro based on Nix the package manager and its configuration language. They have clear separation of concerns. Keeping it that way has actual benefits too, namely portability. It allows you to use Nix on a wide range of platforms including non-NixOS Linux distros, macOS, and even BSDs to some extent.
> On top of nixos/nix, there's also shell.nix/flakes.
Flakes are currently an experimental feature and not meant for wide use yet. So if you're just getting started with Nix, I'd recommend looking into it later until you're sufficiently comfortable with Nix.
They're not fundamentally different, though. Flakes formalize the conventions for writing Nix expressions to make them more reusable.
I also feel like nix is immediately tangible and beneficial in a way that makes us want to get to the power user level, but I don't feel like wanting to learn, say, kubernetes means one automatically wants to learn to be a cluster operator. The scope and complexity are great, and I have experienced all of the things you have described as well. Now that I am fairly confident in my abilities, I think it's important to make sure we find ways to help everyone get the most out of it, even if it doesn't involve scaling the wall that is the learning curve.
Can you clarity what you mean?
What do you mean by "trap"? Which (if any) of the following concepts factor into your understanding? Broken expectations? Premature optimization? Over-engineering? Lock-in?
Are you suggesting that many users of the Nix package manager "fall into a trap" and begin using NixOS?
What do surveys and/or data show about usage of just the Nix package manager? Combined with NixOS?
The Nix ecosystem is attractive to me at a high level because it looks like it has the possibility to revolutionize a lot of the problems that come up with an increased focus on devops culture. The problem with the Nix ecosystem itself is that it is larger and more varied than it might appear.
There are really three large entities in the ecosystem: nix the language, nix the package manager (nixpkgs), and nix the OS (NixOS). Despite the fact that there is a lot of overlap in some of the details of these three things, they are easily understood as separate entities. There are a bunch of smaller secondary entities such as nix-darwin, home-manager, devshell, and digga (formerly devos). The aim of these entities is usually to supplement, enhance, replace, or reproduce features of nixpkgs and/or NixOS.
Each of these entities has its own learning curve. It takes surprisingly little nix knowledge to use it as a dev environment generator, but nixpkgs knowledge will help a bit more. Even within nixpkgs itself, each major language has a large ecosystem built up around facilitating package building and deployment, and these are extremely heterogeneous. Some of them require that you regenerate the entire matrix of package dependencies for the package manager (javascript, haskell). Some of them have tooling that allow you to get started with almost no effort (mach-nix for python). These all have their own peculiarities. NixOS has a ton of modules, all rather heterogeneous, on top of requiring more careful knowledge of how to put together a system than most other distros. And nix the language without nixpkgs is very small, and some of what it does have built in is necessary so it can bootstrap an actual stdlib from nixpkgs.
The trap is sprung once you've been sold on the idea, and even seen it shine through some things, but feel like you have an entire wall to climb to get to a point of general competency. All of these things I've described, and even some of their subgroups, require new knowledge for a more complete understanding. I do not understand the details of many of the things I have described, and I would consider myself a power user at this point. What looks like one gigantic, insurmountable cliff is really dozens of much smaller cliffs, many of which are not necessary, at least not to most people.
At one point I wanted to test my skills in a really generic way, so I helped fix broken packages on macOS during one of the two yearly regression periods where the community tries to get the CI builds passing. I picked the packages completely at random. Most of what I ended up fixing was C/C++ based, and I don't normally work in it, or I haven't for many years, but I feel like I learned a good bit about cmake. I don't really write Haskell, but I was able to fix a bug in a solving library having to do with a missing portability shim. I didn't have help for any of this, and in some cases I was either too stuck to fix the build or unable to fix it due to problems beyond my control.
All of this is to say that I think getting to this point is a lot less painful if one takes a use case, preferably a narrow one, and focuses on learning whatever is necessary in the service of that use case. Some good topics are dev environments, using nix as an app/project builder, using nix to build containers, using nix to manage remote machines (the difference between these and personal machines is that the scope of these machines tends to be much smaller). I think people try to take on too many of these things simultaneously and end up overwhelmed. That's not to say that NixOS isn't worth learning, but I think many people could realize most or all of the real-world benefit if they use nix on an OS they're comfortable with. Coming back to my comparison of being dropped in the Romanian countryside, if ...
package manager (whether on Darwin or any linux distro), to have some tools globally available:
per project, most shell.nix can just look like this: e.g asdf, only much more generic with full non-leaking package management: ruby/rvm/rbenv/bundle exec (example for rails >= 6): python/pyenv/venv: mixing arm64 and x86_64 on an Apple Silicon machine: using an unstable/pinned/git package: selecting a particul...Thanks to a chatty Romanian taxi driver I had earlier today, I understand this.
Really, I like the idea and I actually like the fact that they use Guile (scheme) instead of a home-built DSL, but guix is a horrible, poorly documented, doesn't work even on the happy path mess.
But maybe I'm just a "GNU zealot".
I didn't mean to imply everyone who uses it is a GNU zealot, but there's definitely a subset who are.
"possible future efficiency gains".
With this in mind, the question shifts from rather vague talk about possible futures to various kinds of scenarios.
What happens when software developers have a deterministic build system? On the whole, the benefits are significant. Sure, there are costs to get there. In my view, the benefits often outweigh the costs.
I keep seeing nix mentioned alongside complaints about it's maturity but that seems par for the course for something new, doesn't it? Until a tool reaches critical mass sharp edges are to be expected.
I'll look into nix again when I have an appropriate use case.
More broadly, even if it isn't as popular, Nix is a viable tool and also encourages us (developers) to seek out repeatability in our dependencies.
I disagree with the whole "highly likely" bit. Just because something new has a large positive value, that does not mean that a large time-investment or effort-investment in it is likely to pay off.
After all, it may never get traction to make the cost of learning it worthwhile - some other, newer, better, easier system could come along and be adopted instead.
We've seen this play out again and again in the technology world[1]. In 2000, a large time and effort investment into UTF-16 would be wasted as the world adopted UTF-8 instead. A large time and effort invested into new languages with powerful features (Haskell, circa 2005) would be wasted as the industry largely ignored it in favour of (presumably) inferior alternatives.
Is nix highly likely to provide solutions to a whole class of problems?
Sure.
Is it highly likely to be the dominant, or even a common tool, in the future that solves those problems?
Doubtful; "easier-to-use" has always won against "can better solve problems". I see no evidence that this will change anytime soon.
[1] I've found that by procrastinating about learning a new technology, I can sometimes manage to avoid having to learn it altogether, because the tech dies off before my procrastination does. By putting off learning VRML in 2001, it went away before I ever got to it.
Likewise, I'll put off learning new things until I actually have a need for the solution they offer. At least then, even if they die out in a few years, I'd have at least solved my immediate problems.
You are disagreeing with something I didn't write.
The way you've framed the question is notable. The argument structure ignores a common pattern in software: tools steal ideas from each other often.
For example, MySQL is very different than its 2002 version. There are more database back ends with various guarantees and performance tradeoffs.
Your choice of language: saying 'need' suggests a black and white decision. See what I mean?
Costs and benefits often cannot be estimated accurately 'in one go'. It is common to spend some time and money to increase understanding and thus reduce uncertainty.
What deterministic build systems have you used? If you have not, how confident should you be that you can accurately assess their costs and benefits?
PS. Your comments suggest to me that, like most people with experience, you have become overconfident. To disabuse yourself of this, I suggest you find a prediction market and make some bets. You'll find you have to make your claims testable. Make a specific prediction/bet on Nix and let's see what happens. Let us know how it goes.
I recently gave nix a try because of a hacker news comment and agree with most of the points in this comment. The fact that new users are encouraged to read not 1, but 3 separate manuals to grok the tool is extremely discouraging for new users. Also the fact that nix is currently transitioning to a new feature called flakes and deprecating channels signals to me that the ecosystem is not stable enough for me to recommend that my team at work use it right now.
That being said, I intend to read at least 2 of the manuals and use nix for dependency management in my side projects and to manage my system dependencies. I think nix takes all the great things that asdf can do, to a whole new level. The dream of having a nix section in my projects 'Getting Started' that only has 2 steps (1. Install nix 2. Run `nix-shell`) is something I desperately want.
I think nix is much like erlang in that it's incredibly powerful and solves a lot of common problems, but it's used and maintained by an old guard that went through the trouble of learning how to use it and maintains the "I did it, why can't you" mindset. The erlang ecosystem has been given new life and been made easily available to new users thanks to the awesome work done by Jose Valim and the rest of the elixir team and community. However, I really hope that it doesn't take 30 years and a new language built on top of nix for it to become accessible to everyone.
For those working on nix and the new flake functionality please look to things like elixir and asdf for inspiration on documentation, ease of use and pragmatism. Help make the tool you love ubiquitous. It will ultimately make your life easier when working with new developers because they will already have nix installed.
I intended to do everything I can to help push nix in this direction (once I understand how the hell it works) if anyone reading this, has some time on their hands and wants to improve the state of software development for everyone please go down the nix rabbit hole and try to find ways to contribute to making it more accessible.
Look at Python with Nix: https://nixos.wiki/wiki/Python And then consider the simplicity of `pyenv install x.x.x` and a requirements.txt.
- you want to use lang-x.y.z, so you'll just reference an old commit
- turns out that version was never packaged, so you copy x.y.z-1 to your flake/overlay
- you end up patching the build because it doesn't work by default with new environments
- on the other hand a module you're using needs the new environment so you do your best to make everything happy - more overlaying
- because you changed something crucial in a few steps, you're compiling llvm 20 times through this process and now it's 1am
Compare that to putting "rust 1.59" in your .tool-version and running "asdf install". Guess who tried to use 3 specific versions of rust at the same time recently and got very familiar with the process...
(That's also ignoring the issue like Ruby being packaged without the platform identifier in nix which makes it unusable for some development purposes)
It should never be Nix's business (or any other OS) to provide elements of a project's toolchain. Reciprocally, projects should go out of their way to not depend on OS installed toolchain components.
Why? I used to think the same way, but that's because of the scar tissue of all the tooling being terrible at accepting that build concerns and packaging concerns are one and the same.
Nix proves it and makes this ancient pain a non-issue: "My package depends on x, y and z, additionally, building it requires a, b, and c, please also make build dependencies available in my shell while I write code too". That's it.
> Running a project should never require more than "git clone $project && $build-cmd".
That's precisely what Nix allows you to achieve. Meanwhile doing this:
> Reciprocally, projects should go out of their way to not depend on OS installed toolchain components.
Means that you can never depend on anything you're not vendoring (read: maintaining a copy of, on your own). Or it means sacrificing productivity in the altar of shitty tools. Nix allows you to have your cake and eat it too.
It is also not a package manager like "apt", "dnf" or "pacman" as it does not fill the same niche. It does not even have to fill the same niche on every project.
As an example: at my job we're using it exclusively for optionally managing shared developer dependencies. Everyone is free to manage their own dependencies if they don't want or can't use Nix. But a flake.nix is there so you can get everything configured for you automatically.
In a personal shared project of mine I use it for tracking developer and build dependencies, meaning that I can build the project through Nix without having any of the dependencies specific to this project installed globally on my machine, meaning I can work on other projects that use incompatible versions of the same library, and keep the project reproducible.
On my desktop and laptop I use it as part of NixOS (which is an OS), to configure both from a single, mostly-shared configuration I keep in Git, replacing a mountain of terrible Ansible and shell scripts that did not work even nearly as well.
You can also use it to generate Docker images, operating system ISO images, distribute your app as an AppImage, Snaps,... and many other things. And you get an actual programming language to do all that with all the tooling the community maintains for each usecase.
If you're just using it as an asdf replacement, as the comment you originally replied to compared it to, your project will not be any more locked in to Nix than it would be to asdf.
"Nix shouldn't provide compiler toolchain" is not.
Nix's whole design is oriented around explicit description of inputs in programmatic descriptions of package.
asdf does not allow you to keep three different versions of the same language, so I'm not sure how that compares? It's not super-trivial to do in Nix, but at least you can do it.
asdf is also no different than Nix when it comes to minor/major versions. You're at the mercy of what the plugin does, other than that you have to create your own plugin from scratch or make a fork. Nix has the option to patch things up more easily at least.
Sure it does. You can have any number installed and either switch by changing the tools version file, setting it for your shell, or referencing the right path directly.
> You're at the mercy of what the plugin does
Again theory vs practice. In theory, sure, it depends on the plugin. In practice, almost every plugin lists every single released version to choose from. Which is much different from nix where the official choices are limited and you have to work to enable the rest.
Oh, that's cheating :) Nix can do the same thing! Derivations are just kept in the store until you garbage collect them. You can also configure multiple versions at the same time to present you with multiple binaries, which is what I was referring to.
> Again theory vs practice. In theory, sure, it depends on the plugin. In practice, almost every plugin lists every single released version to choose from. Which is much different from nix where the official choices are limited and you have to work to enable the rest.
In practice, nixpkgs supports every maintained version of most tools. Certainly far more tools than asdf can ever hope to maintain. If not, there's an overlay or flake out there to support it. Can you come out with specific examples and a usecase?
As others pointed, the idea behind nix is cool, but it sometimes fall short, hope it get ironed all the issues in macOS so I could give a try in the future.
Are there any other advantages to using nix?
The advantages are more pronounced for DevOps than for developers.
e.g. for DevOps, Nix allows copying a package of software (and all its dependencies) from one computer to another, with confidence that the programs will work on both machines.
For developers, Nix is more appealing if you ever work on more than one computer. e.g. with Nix, it's easy to ensure you've got the same version of packages installed regardless of whether you're on macOS or Linux.
It's pretty neat for side projects, in order to get "I was able to build it now, I'll be able to build it in 6 months". Whereas, if you're otherwise relying on Ubuntu's system packages, maybe the package versions got updated, and so you'll need to spend time to get your program to work with the latest compiler/libraries.
For $REASONS I want python 3.7.10, and the full scientific stack (numpy, scipy, pandas, numba, matplotlib, ...) at the latest compatible versions along with pytorch. I don't mind waiting for things to compile. How, if at all, can Nix help me get into an environment that meets those constraints? Can it help without requiring me to find and copy hashes of things?
Put an incorrect hash in. Run the build. The error will tell you what hash it actually got. Copy that. Run build again.
This makes me wonder: does Nix use libsolv, or a similar idea?
Roughly, Nix's approach is to describe packages as a function of it dependencies as inputs. (And all the inputs have all their dependencies described in Nix, too). Nix doesn't do things like "install latest version of X". (One of Nix's principles is build reproducibility; the package should have the same behaviour if built with the same input).
Tools like mach-nix or poetry2nix wrap around nix.. as I understand, they adapt the requirements.txt or whatever and pass it into Nix in a way that Nix can understand.
Both for parent's quandary, and for when you're working with a group and don't want to force everyone else to use Nix.
There's a reason why, despite being almost 20 years old, Nix has at best a small cult following. It just doesn't solve the problems that most people are looking to solve, most other solutions are simply easier and in many ways better
I spent time understanding it, and getting it working, but it really doesn't work just universally. Once the install process becomes more like, say, miniconda -- where an unprivileged user can install it locally with ease -- nix will get a massive boost in my books. Until then, I can only yearn.
It's astounding because the local version is doing a naive, case-sensitive regex search, which is pretty bad when you're looking for, say, the Perl package MIME::Lite which can have various junctions and capitalizations. In fact, that package has a listed name of perl5.34.0-MIME-Lite but is installable as nixpkgs.perl534Packages.MIMELite!? I can't even
I found the code a few months back (but can't be bothered to look it up just for a comment). I didn't fix it then because I had other priorities than learning how to modify my local copy of nix to test it.
[1] https://search.nixos.org/packages
Could it solve the issues we're seeing lately with NPM?
But, I would encourage people to try out NixOS. Its much better than the package manager, yet it follows the same basic principles.
- Uses a simple DSL language
- 1 config file describes your entire environment, you basically use nix the language to tell your computer if you want wifi enabled or not, or if you want sound, or if you want i3 etc.
- Its easy to roll back, since the config file is versioned (like git)
I personally use a mac machine, running nixos in vmware fusion and I can't believe how long I had been using a separate machine for development and another for personal use.
I get the best of both worlds with this setup; my messages and notes are here, I get world class hardware (come at me bro), and I get to control every single aspect of my development machine.
I still use docker inside nixos, and yes I know its almost supposed to replace docker and all but I will get there one day, right now is not it.
Areas for improvement: - They should have come up with a better name: nix, Nix and NixOS are confusing (language, package manager and operating system - in that order. Yes, they went with an upper case letter for the pkg manager) - Docs NEED to be better, this is not an easy task. The language is shared between every component. - Give playgrounds to beginners, I shouldn't have to download the minimal iso and boot it on my machine to get a sense of how powerful it is. There has to be an easier way.
End of my rant. I started using it a month ago, I have run into issues with dependency pinning but I remain optimistic.
like "nixpkgs" is almost always the default, from what I can tell, so it could be omitted and simply be assumed to be the default, and not sure what the -c param does but perhaps it could also assume that the specified package(s) are also the ones to install/choose, in which case the whole thing could be like `nix shell -c python3`?
Previously: https://news.ycombinator.com/item?id=26018684 (Feb 2021: 98 comments)
I've even tried doing that with all the devtools we have. Like normally a developer here has to install gcloud, terraform, kubectl, apt-get lots of stuff, and then 3-4 internal tools. Then configure all of it, and of course some stuff will just not work on someone's computer. But creating a docker image including all that means that I can grab a new computer and be up and running in minutes.
"Setting up a Python environment" takes about 10 minutes and it's dead nuts reliable.
It is a very effective package manager.
I did some googling, but I’m unfortunately not experienced enough to be able to describe the problem succinctly to google.
Using pylance, btw.
TIA!
This works in VSCode over SSH perfectly fine - the locally running VSCode is able to detect and select Python environments in the remote server.
Although, now you’ve got me thinking what would happen if I mounted the interpreter inside the container to something local that I can point to? Might try this later, thanks for the help!
Conversely, the experience of using Ruby within Docker for development is often clunky and awkward by comparison.
I found it works well for Ruby, Elixir/Erlang, and Node.
[1] https://docs.gradle.org/current/userguide/toolchains.html#se...
[2] https://github.com/halcyon/asdf-java/graphs/contributors
The only plugins under the asdf-vm github org are for Ruby, Node.js, Elixir and Erlang, because those are the only languages I needed.
Not sure the error lies at the feet of asdf, though.
For ruby/node it's a relief to be able to easily "activate" a certain version of ruby and a certain version of nodejs - at the same time, via simply "asdf local ruby ... and asdf local node ...".
https://asdf.common-lisp.dev/
- https://github.com/asdf-vm/asdf/issues/383 - https://github.com/asdf-vm/asdf/issues/547
They did do something to ameliate this though: now they refer to the proect as asdf-vm as much as possible, to distinguish it from Common Lisp asdf.
No, asdf is a replacement for them. It's not an abstraction on top of them. It uses its own plugin system, and everything is written purely in bash. Everything gets installed in `~/.asdf/installs`
> asdf version resolution is slow which makes every command execution pay that penalty. asdf reshim is needed for finding new executables, and some tools are not happy with their executables being masked by shims.
> Perform asdf version resolution only once and defer environment loading to direnv.
Nevermind trying to setup a development environment where you have multiple dependencies on different language runtimes.
It can't actually install different versions though (looks like `asdf` can?), which is somewhat inconvenient.
Regarding the suggestion to “do things manually” — what is the value of doing things manually? For me, it is very far away from how I want to spend my energy, which is doing computer things that only I can do. I am happy to outsource environment variable fiddle faddling to someone else.
Package managers often don't have a way to install specific old versions of a package, or multiple versions of a single package. That's the whole point.
Not sure I'd do this, though:
But more probably add a symlink to the completions managed by git/asdf: ln -s ~/.asdf/completions/asdf.fish ~/.config/fish/completions/pyenv I can understand. dunno the others.
but goenv?? you deserve whatever hell this is. it's a single binary configurable with an envvar for gopher sake. just RTFM.
For the most part, latest version linting is good enough for me, currently.
Docker is better for version managing services like databases, caches (Postgres, Redis, etc), things that have little need for different configuration or exploration and persist through multiple sessions. Once they're up, they're done.
Asdf is the next best thing for local development, where you need that kind of exploration.
There are advantages to the docker approach, and Nix can be painful to get going, but a significant portion of the community (including me) find the pain to be worth it.
FYI, the link to asdf there is a link to google.com
> Hermit is probably most similar to asdf, but their goals differ. Hermit's goal is to make isolated cross-platform tooling consistent, self-bootstrapping, and reproducible at the project level. asdf's primary goal is to allow developers to install and switch between multiple versions of languages and tooling.
The downside for the author of this post, though, is that Hermit doesn't yet support Ruby because it's not static/relocatable.
[1]: https://cashapp.github.io/hermit/ [2]: https://cashapp.github.io/hermit/faq/#asdf
Be careful if you’re also interested in Hermit. These kinds of things bit you up way down the road when you least expect them to.
[1] https://github.com/indygreg/python-build-standalone
1. You need a tool for X to solve an issue so you write it. (AIM)
2. Guy over there needs a tool for Y to solve the same issue, so he writes it. (Yahoo Chat)
3. So many tools exist, we need to merge them all into one. (Trillian, Adium)
4. Merging all the tools created a monster, we need a common protocol they all understand.
5. No one remembers how tool X works anymore, updates become more sparse.
6. Both X and Y are no longer popular, big company took over. (Slack)
I'm quite happy with direnv to avoid this problem.
https://direnv.net/
Funny enough, it was submitted to HN most recently just yesterday:
https://news.ycombinator.com/item?id=30908321
direnv is complementary to the various language-version tools, so it obviates needing asdf:
https://github.com/direnv/direnv/wiki#project-layouts
But it seems some folks combine the two:
https://blog.mikecordell.com/2021/12/18/better-project-envir...
You are giving people a _ton_ of trust when cloning a repo or pull request that has a .envrc which will be automatically executed by direnv. This file could do anything from steal your SSH keys to delete your entire home directory or worse. And the only thing preventing that from happening is the suggestion of "oh be sure to eyeball the .envrc file and make sure it looks ok, then run this command to enable it". People will just google for the first "how to I unblock .envrc file" and blindly run the command, then oops. Or a determined attacker will just obfuscate what they're doing so the .envrc looks fine but you failed to realize some esoteric bash-isms were actually invoking an obfuscated script and... oops. It's just begging for a supply chain disaster on a similar scale as npm's infamous issues.
VS code and some vim extensions have similar issues with automatic project-specific configurations that allow arbitrary code execution too, and neither have a good solution beyond 'just read all the code and understand it to make sure it's not going to hurt you'. It's not really a unique thing to direnv.
We really need a better model for collaboration and consistent dev environments. I think containers and docker/chroot/etc. environments with explicit opt in of your local resources, configs, files, etc. are a much more sane and security conscious path towards it. A script executing to setup my python environment for a project should never, ever have access or even know about my personal password store database for example (unless I explicitly decide it should).
> And the only thing preventing that from happening is the suggestion of "oh be sure to eyeball the .envrc file and make sure it looks ok, then run this command to enable it". People will just google for the first "how to I unblock .envrc file" and blindly run the command, then oops. Or a determined attacker will just obfuscate what they're doing so the .envrc looks fine but you failed to realize some esoteric bash-isms were actually invoking an obfuscated script and... oops.
You need to be absolutely sure you've read and understand every single line in the .envrc file, and every single line in any script it may be invoking, before you run direnv allow. Otherwise you are giving code from the internet a blank check to do anything on your machine with your user account.
I personally do not think bash/shell is a sensible language to trust that you can quickly or easily determine from a visual check alone if a script might be doing nefarious things.
Sure, you can trick idiots into running malicious code, but I don't think you need direnv for that - any pip/npm install can run anything and so does curl|bash.
Seems safe to me
If I wanted to include it, I use something like git-crypt and encrypt the contents
I don’t see why this is so hard to avoid with per-project version pinning. I’ve never had this problem with Java.
This seems like a lot of effort and complexity to work around global package installation. Just stop assuming that packages are global and this would all go away. How do we know that? Because other languages don’t require this kind of workaround!
I'll agree that how to do things properly with Ruby isn't obvious when you're new to the language - just as it isn't obvious how to do things in Java (or other JVM languages) when you're coming from Ruby or another language.[0]
But Bundler doesn't work all that differently from Maven, in the end. Instead of running your commands through `./mwnw`, you run them through `bundle exec`[1]. Dependency management is straightforward and mostly hassle-free (the problems arise "only" when you use C extensions which, to be fair, is not that rare, but I don't really see a way around this in a language where you need C for certain fast things). In particular, Bundler by default[2] also stores dependencies centrally like maven to avoid duplication and then selects the appropriate version of a given dependency (unlike e.g. NPM which stores everything locally).
Rbenv (or alternative tools such as chruby or asdf) solves a problem completely distinct from Bundler or Maven, it's about selecting the right Ruby version for your project, a problem that also needs to be solved with Java. IME, many developers just let the IDE deal with that, but Ruby developers are more used to running things from CLI, so that's why rbenv exists. In any case, the most popular analogue for Java would be sdkman which works, mostly, the same way (and which, IME, is even a bit hackier than rbenv).
JS and Python are different. Node is a mess IMHO, and Python isn't able to agree on which package manager to use (I would hope people would migrate to Poetry, which seems to be doing things right).
[0]: For example, one thing that I find very annoying over in Java-land is that it's a real hassle to make sure things behave the same way when called from the CLI as opposed to from the IDE, such as setting the active spring profile. This is because, mostly, the IDE circumvents maven/gradle (you can run things through the build system, but typically the output, e.g. test output, is better when using the IDE-internal system, at least with IntelliJ).
[1]: There are "hackeries" to get rid of having to run everything through "bundle exec", such as binstubs and direnv, although I would mostly call them "shell script tricks", and anyway, if you instead just alias "bundle exec" to something shorter or create a wrapper script, it's not really worse than `./mwnw`.
[2]: This can be overriden and is actually a good idea for production builds, especially if you use a multi-stage docker build since then it's easy to copy over all your dependencies.
If that's the discussion you want to have, well, I personally also prefer statically typed languages, but that's beyond the point.
That said, I've had issues, e.g. on CI, that didn't manifest locally because of a mismatch in Java versions (something with JAXB, IIRC), and Java is maybe sometimes a bit too cavalier about the issue.
Regarding Python, I don't have much experience, other than the general feeling that it's fragile and that specific projects work with specific dependency managegers / virtual environments, and they are not really interchangeable or anything. I either could replicate the env or I couldn't. I definitely don't have much of a Python experience.
Now regarding your Java IDE annoyance, I share the same experience. Also the case with Tomcat. IDEs do so much hackery so that user experience is convenient, that you would be hard pressed to replicate the results, and often need to test both because it just doesn't work the same way, due to the different dependencies used, deployment method etc.
Now that I think about it, every single one of the programming ecosystems I used were quite annoying.
Keeping the "right context" when you do things in different directories with the same shell config is, unfortunately, a problem that I don't think is easy to solve. My preferred approach is to use direnv (https://direnv.net/), but that's not perfect either.
As for rvm, I think that a lot of people at some point discovered that rvm was maybe doing a bit too much (in particular, it would also manage gem sets, something that seems superfluous with Bundler). I think that's a reason why rbenv and chruby emerged as lighter-weight alternatives which are a bit less magic. I haven't seen many people use rvm anymore in recent years. By contrast, Bundler doesn't really do anything to your shell.
> Now that I think about it, every single one of the programming ecosystems I used were quite annoying.
That's something I agree with. As a more recent convert to the JVM world (and mostly just for lack of better alternatives), I'm still sometimes annoyed at the IDE-centrism (and UI-centrism, see also sonarqube) in this space. Luckily, things are already better than they were when I learned Java in university and "it runs in my IDE" was the default state of affairs...
If people don't use rvm, do you know what they use to install a specific Ruby version? We basically use it so that we can use a Ruby that's not bundled with the distros anymore (large legacy project).
1. For selecting the correct JDK to use with standard plugins, including the compiler plugin, there's support for toolchains:
https://maven.apache.org/guides/mini/guide-using-toolchains....
2. For ensuring Maven itself is run with the correct JDK, it is a commom recommendation to use the Enforcer plugin to specify the required JDK version and/or vendor, as well as the Maven version:
https://maven.apache.org/enforcer/enforcer-rules/index.html
3. It is increasingly common to use Maven Wrapper to supply Maven, which you seem to be aware of.
4. Maven will run with the JDK pointed to by JAVA_HOME, of course.
I'd respond to production disasters with hacky bash scripts and then we'd spend the next sprint incorporating that functionality into the server so the hacks weren't necessary. We kept catching ourselves working really hard to optimize what we thought at first needed to be custom code but what ended up being easily achievable by leaning on well-known non-java tools like `diff`. There's something about the way Javs encapsulates everything into a JAR that discourages Java devs from using anything on the outside.
If you can live in a world where everything you need is in the JAR, life is good, but that hasn't frequently been my world.
It's only relevant because the high level of encapsulation that made deps easy to manage also represented a barrier. Using diff and awk was fast enough, the reimplementation in java wasn't (happened just a few times). Language ecosystems with a more permeable encapsulation style make it easier to reference tools outside of that ecosystem at the cost of making it harder to get your deps straight.
I was amazed to find such a mess. Each project has full copies of all sources of all modules you use? They’re not shared in a global cache? There’s just hundreds of thousands of individual files?
Yarn does a better job on some of this, but it all feels like stepping into an alternate reality where no one has ever heard of Maven.
Maven predates Node by 5 years. Even if Node was developed without knowledge of Maven surely someone has brought this up in the last 13 years, right?
It’s all very surprising to me, and I guess an interesting lesson in “parallel” evolution.
It’s also possible to install multiple versions of Python on a system. The dependency management is the real problem — having more than one version of a runtime is not the hard part.
> The dependency management is the real problem
I'm not sure how this works in Java, but in many ecosystems like JavaScript and Rust dependency management isn't an issue at all. Installing a package defaults to a local install and even things like dependencies transitively depending on different versions of the same package are supported out the box and just work.
Not to say that I like the Python package management story.
Just today on trying to get rvm to do something I got angry at the number of language-specific package managers I have installed: nvm, jenv, rvm, rbenv(after I couldn't get rvm...)
Of course, I know about Nix but it is too complex for my use cases.