Is anyone is the AI/ML area finding success with anything other than conda, where installation of CUDA/CUDnn is required? Although I often have to pip install a lot of packages, I find conda's nvidia/pytorch/conda-forge channels are still by far the easiest way to get a deep learning stack up and running, and so I just stick with conda environments. I've tried poetry in the past but getting the NVidia deep learning stack up and running was really tough.
So fast it finally made virtual environments usable for me.
But it's not (yet) a full replacement for conda, e.g. it won't install things outside of Python packages
How are you installing Pytorch with CUDA with Poetry? I stopped using Poetry because it wouldn't automatically get the CUDA version; instead, it would install the CPU version. I migrated to PDM, which does the right thing.
was this before torch 2.0? With the very notable exceptions of a few mispackaged versions, torch now includes all the relevant Nvidia libs, and I haven't seen it grab the CPU version on a GPU box yet, though I'm not sure what it looks for.
A notable open issue in poetry is we can't currently specify one dependency on torch, and have it grab CPU version on some systems and GPU on others. Does PDM solve that?
I don't think PDM solves that directly. What I do is have different lock files for different platforms (e.g. Linux/CUDA and macOS/Metal), but pyproject.toml lists only "torch".
I stand corrected. I was familiar with the first option, which coupled the dependencies with the platform, whereas I wanted a CUDA version on Linux and a Metal version on macOS.
However, this works perfectly with Poetry 1.8 and Pytorch 2.2. I suppose the only problem is what PDM also does, where the lock file is platform-dependent. I'm not sure whether Poetry allows you to select a specific lock file, however.
For anything related to CUDA/CuDNN, use one of NVIDIA base Docker images. Then whether you use Conda / Pip / Poetry / Pipenv does not matter much. Not at all a Conda fan myself and avoid it like the plague
What's surprising to me is that this isn't better known. The only reliable solution I've found is to go with the pytorch or deepstream images from NGC. Conda is probably a good idea for noobs who need Cuda installed for them on windows, but otherwise I find it an endless source of finicky issues, especially for unsavvy ML scientists who are looking for a silver bullet for package management.
10 years ago, « Data Science » work past the experimental stage was performed by SWE with a knack for applied maths. So investing in tooling to do things properly was a given.
Nowadays, most DS people only want to do ML at the experimental stage only and get lost when things get on the engineering side of things. But for their defense, nowadays the bare minimum skills require to do programming, containerization, CI/CD, etc. More experienced and swiss army knife SWE/MLE have to educate the willing.
It was already the same 10 years ago with MATLAB dudes not wanting to get dirty with C/C++/ASM SIMD. The history repeats itself, only at a faster pace
and it works. It used to not, but it's been fine for me for about a year now.
There's a very good chance I've installed cuda on my system before this though. And usually cudnn and some other packages because this is part of my standard install. And then I also never run into the issue where a package is looking for nvcc.
Pyenv just worked for me. I am actually using Fedora Silverblue and have GCC and the CUDA SDK available only inside a toolbox container. Therefore, I have to enter that toolbox to install things like FlashAttention.
I have successfully transitioned an ML/AI team of seasoned researchers away from conda and to poetry. Some also use pyenv, I suspect a lot don't bother but may get bitten eventually.
It's definitely a learning curve, but it turns out every conda user has been bit by the irreproducible tendencies of conda quite often. Nobody uses the conda env file, they just start an env and pip install things into it. They don't realize the base env has stuff, too, and conda envs are hierarchical rather than isolated. I know it's possible to use conda in an isolated and reproducible way, but have yet to meet someone that does so.
So it hasn't been hard to pitch poetry to these folks, and while many complain about the learning curve they appreciate the outcomes.
We're a pytorch shop, and torch mostly just works with pip or poetry these days, as long as you skip the versions the torch maintainers mispackaged. We rarely need anything higher-level that only conda could install.
We really like having more than two dependency groups as this allows us to keep research and production in the same repository. main, dev, research. Then researchers contribute to the core library of a project and keep research and production using the same code for running and evaluating models.
Have you tried https://pixi.sh/ ? It brings Cargo/NPM/Poetry like commands and lock files to the Conda ecosystem, and now can manage and lock PyPI dependencies alongside by using uv under the hood.
I haven't been using anything CUDA, but the scientific geospatial stack is often a similar mess to install, and it's been handling it really well.
OP - I'm curious why you've submitted this today, you seem experienced enough to guess this is not the first time you've used it. Has there been a major change or something happened in the community I need to know about? Or are you just spreading the love for a great piece of open source software?
I used Python a lot, but missed pyenv. Used it for some weeks now and was thinking this is something I would like to know about earlier (thats normally when I post it to HN).
I've used python lightly for a few decades now. I've never got far enough in to figure out how packages work. Once, at least 10 years ago, I briefly tried to use some published packages. I never figure out how to get it to work at all. Other than that, I've limited myself to what's available in the standard library. I've seen a lot of conflicting information about how python packages work, or should work. If there was a "one true way", I'd be interested. I don't want to have to compare a list of pros and cons. I want one officially blessed tool. I'll live with the constraints.
Packages is another story though. Pyenv is for Python versions. For me it’s either that or using Docker images. The later performed poorly with PyCharm when I tested a few years ago. I have seen an issue with Pyenv only once and not even on my machine.
I'm not saying pyenv is not a useful tool, but it is not a tool for beginners fighting with python packaging problems. It's a specialist tool to normalize your setup.
Very often, I see people tell me they don't have problems with pyenv, but later on have other unrelated problems with their dependencies. Analysis then prove it was because of pyenv, they just didn't know it. The cost is not obvious.
Drawbacks: late availability of patch versions, various quirks from how they are built (missing readline, missing some build info that self-compiled C python modules might need.)
I think PEP711 (https://peps.python.org/pep-0711/) is (eventually) a better alternative, because it builds on top of the proven manylinux approach to binary compatibility.
For now most people would do well to stick to python.org installers for mac and windows.
For linux, official repos are ideal. If you really, really can't (which is different than wanting to), ubuntu deadsnake and red hat epl are the best second plan, while already more finicky.
If you use something more exotic, you chose hardship, and you will have to be up to the task.
Anything else will come with bigger caveats than the people promoting them will really admit to.
The story is of course a bit richer because linux packaging is fun, so I'll complete with:
> but such advice is why so many beginners have painful Python experience
Sample of one, but I never encountered anything even broken in MacPorts. They seem to embrace the BSD ethos of doing everything right (even if at a slower pace, as some packages lag a few releases behind Homebrew).
Personally I only use MacPorts or Homebrew to install random smaller tools. It's easier to install Python, NodeJS, Postgres, etc binaries from the main website.
> Every solution that I know of on Linux requires you to build Python on the machine
Unless you need a Python that's not supported by your Linux distribution, you can just use what's available.
On macOS, MacPorts provides compiled versions for 3.2 all the way to 3.13, as well as 2.6 and 2.7. Right now, I have 3.8, 3.9, 3.10, 3.11, 3.12, and a 3.13 development build. The fact it's not Linux (or x86) might cause some frustration.
If you’re lucky enough to be on a Linux system that uses apt some thankless soul maintains a repo called deadsnakes with all these binaries. Fabulous if you’re using any somewhat old version of Python in CI for instance. Yum based systems are SOL as far as I can tell. Build and host your own binary for that. Apk doesn’t have this either IIRC
If you are deploying to such an ancient OS, it's perhaps easier to have the whole OS packaged as a container or a VM and use that. It's not only different Python versions that might bite you.
What makes you think that GP’s comment is talking about older OSes?
My understanding of their comment is that they’re talking about getting older Python interpreters to run on more modern OSes, modern enough that they don’t carry the older Python as a system package anymore. Hence, deadsnakes.
This is not about system packages or OSes, this is about your application needing Python 3.9.7 specifically, and locking to that, and 3.9.7 not being available in repositories anymore (3.9.7 is just an example). So normally you would either need to self host 3.9.7 somewhere or compile it from source on every new machine (which is terrible for CI but fine for local dev, a one off in local dev to build a Python version is nothing but paying 4 minutes every time on ci to compile your version of Python from source is a very angry amount of time for a lot of people).
That's indeed extremely specific. I'd imagine this pain could be self-inflicted with C-based extensions that were compiled (and can't be recompiled) with structures that don't exist in other versions.
I don't want to imagine what other eldritch horrors await developers working on this application.
Usually it is not the patch version being important eg being specifically 3.9.7, but rather picking one supported version and pinning it. Yes we should all be on the latest Python version that is available in the mainline repos and distros as a binary but a lot of orgs I’ve seen don’t invest the time and effort to properly be upgrading to be on the latest version of deps. So you are usually having one specific supported Python version that everyone builds against which may or may not be new enough to be in the official repo.
I also imagine the scenario you mentioned causing a lock to a patch version. But I see this as just a normal steady state a lot of orgs drift to if not staying on top of version updates
deadsnakes is useful, but also a bit of a misnomer. One can use it to get pretty much anything python version, even ones that are too new to be available in your default apt repos. That is, it's not just for 'dead'/EOL versions.
That's exactly why I don't really on that for Ubuntu though. It's a single point of failure, when I can just use `asdf` to install Python and easily switch between it, automatically if setup correctly on projects.
I use asdf on my local machine, stuff like deadsnakes is for when you need your CI to use VaguelyOldButStillSupported Python version and don’t want to either compile Python from source or host your own binary
My use case is that I'm almost always using the latest Python version, which many Linux distros don't have in their packaging yet, and I'm upgrading code using old versions to the latest. asdf is the best tool for that.
Actually, it only builds it locally if it can't find a pre-packaged version for your system/arch. Admittedly that's most of the recent ones on a Mac, but there is a difference (I've been using pyenv for nearly ten years[1] now).
The big advantage for me is that I can match whatever runtime and standard library a target has (and yes, that's needed more times than not, even in this new age of Docker).
Additionally, you can build an _optimized_ Python. I have this set for my builds:
I’ve quite literally never seen it download a pre-compiled Python - I didn’t actually know it had that functionality, and have been using it for probably a decade.
Actually, I'm pretty sure that binaries for Python 3.9 aren't available on Ubuntu 20.04 without adding an apt repository to get them from, so your statement that you "can apt install it" isn't entirely correct in this case - by default, binaries aren't available, and therefore pyenv has to compile Python 3.9.
> Actually, it only builds it locally if it can't find a pre-packaged version for your system/arch. Admittedly that's most of the recent ones on a Mac, but there is a difference (I've been using pyenv for nearly ten years[1] now).
Really? I regularly use this on Linux exactly because it compiles from source rather than using the system provided Python which often have patches in them that breaks pip's test suite.
I've never encountered it trying to use the system provided Python, but maybe there's a weird quirk in the way I am using it.
Currently going through this myself on a new mac I bought lol. Still haven't gotten anywhere. Oh well
$ PYTHON_CONFIGURE_OPTS='--disable-ipv6 --enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' pyenv install 3.12.2
python-build: use openssl@3 from homebrew
python-build: use readline from homebrew
Downloading Python-3.12.2.tar.xz...
-> https://www.python.org/ftp/python/3.12.2/Python-3.12.2.tar.x...
Installing Python-3.12.2...
python-build: use readline from homebrew
python-build: use ncurses from homebrew
python-build: use zlib from xcode sdk
BUILD FAILED (OS X 14.4 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/w9/xvxzj68j6kx7m480rnwq6hvh0000gn/T/python-build.20240325114837.43577
Results logged to /var/folders/w9/xvxzj68j6kx7m480rnwq6hvh0000gn/T/python-build.20240325114837.43577.log
Last 10 log lines:
./Include/internal/pycore_interp.h:193:24: error: field has incomplete type 'struct _dtoa_state'
struct _dtoa_state dtoa;
^
./Include/internal/pycore_interp.h:193:12: note: forward declaration of 'struct _dtoa_state'
struct _dtoa_state dtoa;
^
1 error generated.
make[2]: ** [Objects/boolobject.o] Error 1
make[1]: ** [profile-gen-stamp] Error 2
make: ** [profile-run-stamp] Error 2
If you use Homebrew: you can use ‘brew pyenv-sync’ to use Homebrew’s pythons with pyenv. Similar commands are available for rbenv/nodenv (which always feel like it is missing an ‘e’ to me)
Just curious: what are the downsides of poetry installed with pipx? The article mentions having to install poetry in another venv, but that's hardly an issue with pipx (you just add an 'x' after 'pip'), and installing pipx is as straight-forward as it can be.
I eventually landed on pipx as fighting with Pyenv and Anaconda - via Miniconda - was an exercise in frustration. There's some mucking about in `$HOME/.local`, but this is mostly self-contained and not a huge chore to keep running.
Coming from the Homebrew/Ruby ecosystem - Hey @mikemcquaid - installing a entirely separate package manager just to deal with a few projects felt like the wrong thing to do.
Occasionally, I have still needed to compile Python myself in order to get things to work, which isn't guaranteed not to blow up w/ `brew`, but this has become far less common of late.
Agreed pipx solves a lot of packaging issues with no downside to speak of. Not just with poetry but also for tools like virtualenv, ruff and black and non-dev command line tools.
So, what is "the tool for beginners fighting with python packaging problems"?
That is, the pattern seems to be that someone mentions a solution, then a zillion responses as to why it sucks.
Is there any tool or pair that sucks least for most cases and beginners? I get that every case is different, but perhaps there are some useful starting points?
> Python distribution and packaging is just fundamentally horribly broken
It's clearly not because most people successfully use it fine.
The problem of distribution and packaging is often a matter of user expectations vs. the actual problem.
The user expectations is that Python is a high level language and will run the same across different machines regardless of OS and Hardware.
The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. So people can end up moving the the glue of the project first and then expect everything else to just work.
Things are getting better (e.g. https://lukeplant.me.uk/blog/posts/python-packaging-must-be-...), a lot of people have put a lot of work into standards and improving the ecosystem, but it is a hard problem that most other popular languages don't interface with nearly as much.
I don't disagree with what you're saying but that article seems odd. It's just a story of someone installing something once that didn't break immediately. Not only is it anecdotal, it doesn't even seem to confirm whether the pip story is getting better or if they just got lucky.
I agree, but there is no large scale study on this.
As someone who has managed Python distributions in a large company and who triages issues on the pip github issue page that my anecdotal experience is things are getting better.
The only hard statistic I can point to is the number of top packages on PyPI that offer wheels has substantially gone up, and is close to 100% in the top 500.
They use it fine by using Docker. So many major Python repos on GitHub come with a Dockerfile compared to, say, NodeJS. It's unfortunate, but having dealt with Python packages before, I don't blame them.
Lots of analysts, data scientists, traders, engineers, etc., use Python third party packages successfully, and have never touched, or maybe even head of, Docker.
And yeah, in general there are significantly less NodeJS third party packages interfacing with packages that directly depend on OSes and hardware. Python has many third party packages that are older than NodeJS that depend on foreign function interfaces, win32 com apis, directly talking to graphics shaders, etc.
Python packages are painful even if nothing native is involved. There are NodeJS packages that rely on native code too, difference is the packaging system is a lot simpler to use, though it's starting to get worse with the `import` vs `require` nonsense and Typescript.
> It's clearly not because most people successfully use it fine.
Well, no because it's perfectly possible to successfully use a horribly broken system. I use Python "successfully", it just meant I have spend probably literal weeks of my life fighting pip and virtualenv and relative imports and finding I need flags like `--config-settings editable_mode=compat`.
> The problem of distribution and packaging is often a matter of user expectations vs. the actual problem.
Ha yes, I expect it to work reliably and simply and it doesn't!
> The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting.
That's completely irrelevant to the kind of problems I was talking about. I run into issues with Python failing to compile C libraries relatively rarely! Even compiling Python itself seems to work quite well (maybe not surprising since that's one of the only ways to get a new version on Linux).
It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go:
> It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go:
I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc.
Though the specific example you linked was largely solved with Python 3, there was a lot of confusion during the 2 to 3 transition because people had to support both behaviors, but most people don't have to think about Python 2 any more.
> Though the specific example you linked was largely solved with Python 3
I can assure you it absolutely was not.
> I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc.
Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad.
In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions.
It's similar to the situation with C/C++ - worse in some ways, better in others (at least there is a de facto package registry in Python). In some ways it's because both languages are very old and predate the idea that packaging should be easy and reliable. That's fine, but please don't pretend that it is easy and reliable now.
The question, as posted, was a confusion about how Python 2 relative importing worked, which was indeed bad. I don't know what you think you are pointing out, you haven't said, and the question *is* about Python 2.
> Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad.
>
> In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions.
I agree that a lot of the solutions were created when there was not an understanding, or thought out design, on what would be a good packaging solution.
But these ill thought out solutions were exactly because of trying to support this wide range of situations, from working on weird OSes, to integrating with strange build systems.
However, what you seemed to have missed is there is now a first party standard on:
* How package installers (pip, poetry, PDM, etc.) should interact with package builders (setuptools, hatchling, etc.)
* How and where build configuration and project metadata should be stored (pyproject.toml)
Almost every popular Python package tool now supports these standards, meaning they all interact with each other pretty well.
Dropping all legacy configuration, and updating the standards to support edge cases is still a long road, but it is a road being travelled and things are getting better.
Honestly this. I've moved three companies now to docker based dev for teams on python. It's better for teams who want to customize how they in particular work without inadvertently also blocking themselves from quickly contributing to another team because that team uses a totally different python management scheme. So a team can be conda-in-docker and another team can be virtualenv-in-docker and another team can be raw-python-in-docker and that one group of weirdos can be jython or whatever other sad shit they cook up.
The common interface is they all use `docker-compose up` and have their editors hooked into the containers.
Using docker is not free, you are effectively trading one type of complexity for another.
And of course you'll have to make sure people equally have good practices, since incorrectly using sudo pip install in docker and not a venv are very common.
So again, one possible solution for a certain context, but I wouldn't sell that to most people. Certainly not to beginners.
People writing their pythonanywhere website won't pop up a container, won't they ?
For a beginner trying to run someone else's Python project, it is free. You run one command and it works. There's tons more complexity under the hood, but you don't have to deal with it. This is why so many Python projects do resort to Docker.
And btw, JS doesn't have this problem, they use NPM.
Installing python with the python.org installer installs the entirety of Python and its standard library and tools, including pip, venv, etc., and the py launcher which supports selecting from multiple python versions. Linux distros may split this up into different packages, but on Windows (or Mac, afaik, though the py launcher is a Windows-only feature) using the official installers is one-stop shopping.
And, no, docker isn't just install and run on Windows. Before Docker made a heavy push for paid Docker Desktop for even personal use, it was close to that.
But now the way to get a free usable docker command line is to install WSL and a Linux environment, install docker there, and then invoke docker via wsl. (Which, of course, you will not find via Docker’s own information, which will try to sell you a paid subscription.)
I tried it just now on my spare Win10 desktop, and it went like this:
1. Download and install Docker desktop for Windows
2. Restart Windows (guess you don't have to do this with just Python)
3. Run Docker Desktop
4. Say "no" to signing into a Docker account
5. Wait for engine to start, which took a few minutes the first time, a bit annoying
6. Pull and run an image (I tried nginx)
It was weird being asked to log in, but the "no" button was pretty clear. I didn't feel like I was forced to use WSL to avoid paying; maybe they've backtracked from something. Unity was far more convincing that I had to pay.
I don't see what editing has to do with this. Whether you're running the main.py or the Dockerfile, if you want to edit the code then you gotta use Notepad.
While there are no silver bullet, for beginners, I found that "Relieving your Python packaging pain" (https://www.bitecode.dev/p/relieving-your-python-packaging-p...) is the Pareto solution. That is, the solution that has the best ratio of effort, reward, but also the lower risk of failure. It's not no risk, but I've been helping beginners for 15 years with Python, and have tried everything you can think of.
It does imply, on linux, to limit yourself to the choices of Python you can install. This constraint is, for most people, preferable than the alternative, even if it gets frustrating to our geeky soul.
The silver bullet is Docker. It's not the best solution in every situation, but at least it'll work.
For your own projects built from scratch (rather than big multi-dep projects off GitHub), there's a smaller learning curve if you go to the vanilla https://www.python.org/downloads/ , install the latest, and use the included pip to install packages. That'll probably get you very far. It's not like the older days when you needed both Py3 and Py2.
For experts working frequently in Python, tools like Pyenv can make more sense.
I think that you are painting a bit of an unreasonably bleak view of pyenv. I think that one can easily get value out of it without being a Python expert. I’m not sure how I can refute your “yes, but you’ll eventually run into trouble. You just haven’t clocked enough hours yet”. I can’t prove a negative. But I’ll say that I’ve been writing Python in my day job for a decade.
But, to add to the list of problems, these Python versions IIRC do not compile with optimisation turned on, so they’re by default quite a bit slower than they need to be.
When the God Of Programming made Python, all other languages were jealous of its elegance, simplicity and intuitive beauty. When the other programming languages went to complain he said..."Wait until you see what package systems I will give them...They will never get an environment properly setup..." :-)
If you don’t care which specific version of python you are using, do not use pyenv.
You will know when you care. These days, doing common tasks, the constraint solver in poetry will often tell you:
“Hey! I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2. Sorry! I give up!”
Now, if you aren’t concerned with using your bare metal for CUDA or fancy new MPS or AMD stuff. Just ignore this and use containers. I’d use podman compose.
However, I use pyenv on every machine. Because it compiles specific versions I actually need, even to create various virtual environments. If compiling python automatically sounds tough, you probably don’t need to anyway.
To describe the problem you’d see. I try to use poetry by default, though I think it became popular before it was PEP-compliant or useful in devops. It is impossible to control the behavior of other package managers, and poetry is/was strict about that. Which means you can’t force deploy in many cases. (Better lately.)
For the problem pyenv helps to solve, I back-up my pyproject.toml setuptools backend with pip and requirements.txt. These days, requirements-cuda.txt and requirements-mps.txt.
The landscape is still a disaster for binary compatibility, but it can be done lol. (I’ve been doing python packaging professionally since prom, which was python 2.6 give or take.)
> I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2
I'm a reasonably advanced python user. I've shipped web apps, Desktop GUIs, cli-tools, and even written cpython extension modules for custom hardware control. I typically target the system `python3` of whatever linux distribution I'm shipping to and I use the system 'python3-virtualenv' for venvs.
But I have never encountered a dependency resolution issue or been forced to use poetry. What am I doing wrong?
(Edit2 tl:dr: it’s that other package maintainers don’t always keep up with new semver constants in their metadata, particularly on anything cutting-edge)
It’s always the inclusion of a specific dependency we added for a feature, and based on that dev’s knowledge and experience. It’s often me, but not always.
This doesn’t happen in ecosystems with a base package versioning. This is arguably why anaconda became popular, and why we target base docker images of ubuntu.
Doesn’t work in complex deployments based on money rather than ideals, every time. At least in my career.
Edit: first time I dealt with this, we ended up forking a dependency chain instead of using pip. I lost that war and they ended up reviving a legacy PHP app instead of funding python dev.
Sometimes, an app or library developer wants to target multiple Linux distributions.
For example, I want my own Python apps to work equally well on Debian oldstable (which currently provides Python 3.9) and Arch Linux (currently on 3.11). That means that I’m going to choose the lowest common version (3.9) as the language level for my app or library.
And if I program against a 3.9 language level, I absolutely *refuse* to use any other interpreter than a 3.9 one at development time. (If I used a newer one, then my linter and type checker would helpfully give me false alarms all the time, unaware of the actual language level.)
Hence, I use pyenv and poetry to get exactly the interpreter for the language level I want, and to allow pylint and mypy to be perfectly aware of the language level I’m using at development time, allowing them to give me findings that actually make sense.
> But I have never encountered a dependency resolution issue or been forced to use poetry. What am I doing wrong?
Not using python on a system without the necessary build tools while trying to use dependencies with native code that don't have binaries built for your combination of python version and platform?
This used to be a huge problem with Python on Windows, as things that weren't pure Python would very often not have binary packages or have them for a narrow set of python versions, often not the same set as other dependencies. (Not just a windows problem, but it was definitely big on windows.)
Tooling and practices have advanced so that more packages are automatically built against a wider set of targets, so the problem is a lot smaller than it used to be.
Oh man, you just confirm that I’m not crazy. For every single time I use pyenv it just downloads source and build. I got tangled into so many issues such as TLS headers in my AL2 boxes. I thought the whole thing is about getting the official binary and install and not compile from source even for a 2 year old release.
What's the problem? The repo clearly states that you need to install such and such build prerequisites.
Also, the issues the article you link are things that damn near every programmer will eventually need to know. Things like PATH are IMO the basics -- if you don't understand this or how to `$packageManager install` something, you're gonna have a rough time programming in general, regardless of language.
Can I ask - why the dislike for virtualenvwrapper? If you are saying that it can have occasional problems, I agree with you. But it makes the process so much simpler. The occasional problems I’ve had (had an issue once or twice deleting a venv) pales in comparison to the advantage I get in remembering a couple fast commands. Is there something better or am I relegated to “source what/directory/was/it/again/bin/activate?
Homebrew really bugs me. The devs want you to use the most current version of Python. Except the requirement is set at the package level and there's no enforcement. So you just keep getting python versions installed.
I don't get why you just don't set a minimum version like every other manager. Or if you want to do this crazy thing, don't rely on package maintainers to enforce it. I just ends up in major bloat
This is misleading. The older versions get cleaned out as brew does the upgrades (if not, then check your brew settings). My take on brew's Python is that it is there to make sure there is a uniform runtime for everything with a Python dependency, but I am not required to use it.
If I pin a version of Python, isn't that going to wreck any tooling that depends on it? Unless you're saying have multiple versions of Python installed.
This is practically the only remaining annoyance I have with the Python ecosystem (relative imports aside). I use some tools, like Glances [0] whose formula relies on a much newer version (3.12) than the actual package requires (3.8) [1].
So when there's a Python update, all of those update as well. I thought I'd fixed this with pipx, but in a way that's worse, because the venvs it builds depend on a specific version of Python existing, which doesn't work well with brew always wanting to upgrade it.
I want a stable, system-level Python that I don't touch, don't add packages to, and which only exists as a dependency for anything that needs it. If an update would break a package I have installed (due to Python library deprecation, etc.), it should warn me before updating. Otherwise, I don't care, as long as any symlinks are taken care of.
Separately, I want a stable, user-level Python that I can do whatever I want to. Nothing updates it automatically. I can accomplish this by compiling Python and using `make altinstall`, but if there's a better way, I'd love to hear about it.
Thanks for taking the time. Can you help me understand? Because I didn't get this from the devs.
Just looking at a random formula, am I correct to understand that this will use python 3.10 and NOT 3.12?[0] I understand ones like this[1] where there's a note about the issue with newer versions.
What I'm trying to understand is if the python version is specified by the formula or it will default to the newest version. If it requires it to be specified in the formula then doesn't this make it contingent on the maintainer upgrading it every python version? I didn't verify [0], but it looks like it should work with later versions of python, and it it is still using 3.10 then isn't that essentially the maintainers "fault?" Because that's my concern. I can't see how something like this stays updated when it requires a maintainer to update. Seems better to have a >=3.10 and then do ==3.10 if only 3.10 works (odd) or >=3.10 <3.12 if it works for 10 and 11 but not 12. Especially since formulas are often made by people that are not the package developers themselves and we're just reliant upon someone keeping up. I'd rather break upon a new version than have many old pythons installed. I know there's no perfect solution, but we're talking about failure modes. And fwiw, I'd rather it try to use the system or environment python rather than installing a unique version. It just gets confusing when you need to add dependencies for optional stuff that wasn't included in the formula and is extremely non-obvious to a new user.
Yes you have to use the package version in the formula. Yes all python formulae will be moved to a newer version eventually. The other versions are for human consumption and use with tools like ‘brew pyenv-sync’.
You're right, but venv requires an existing Python install to create a venv. Pyenv is for installing multiple different versions of Python to create those virtual environments.
It is, but the problem with `venv` is that it can only use the same version of Python that you're using. E.g. if your Python installation is 3.8, your `venv` will also use 3.8.
Pyenv allows you to use whatever version of Python you want.
Most systems allow you to keep multiple python installations, no? I think the best way is to specify the python when you do this: python3.11 -m venv env
On most Linux distros you probably don't want to change your system python to whatever, because there are a lot of system programs that may not be compatible with a version other than what is on the system by default.
Then once you activate it you can make sure your python
just install the different python versions by downloading the binaries or through your package manager and run python3.<whatever_version_you_want> -m venv and the virtual environment will automatically, forever, use that python version when you activate it.
I used to use this a lot when working on a Windows machine. Worked pretty well. But nowadays Nix solves the same problem in a fully general way, for all software rather than a single language. You can have whatever versions of whatever you want, side by side, without interfering with each other.
I was more talking about version management of the Python interpreter itself. So far all the packages I need are already in nixpkgs so I haven't needed to stray off that path yet.
This is a truly, truly terrible idea. It adds several failure modes, some subtle so you can go a long way in a state of error, just so beginners can type `python` instead of e.g. `python3.10`.
Many developers, not just me, have a similar setup: we use virtual environments everywhere, and if you aren't in one, `python` doesn't even resolve to a symbol.
If I want to write a quick script with no dependencies, I directly call `python3.xx` on it. Otherwise, I create a virtualenv.
Yes, it's a bit harder for beginners, but from a huge amount of experience helping people who are starting up in programming, people have little issue in following a few more instructions. What demolishes beginners is getting into a bad state where nothing works and you don't know why.
though, with openinterpreter, you can just ask it to fix your python and it'll help you out of a weird situation that you as an newbie don't understand. Still, node's node_modules implementation isn't a bad one, at the expense of disk space, you don't run into the same problems as you do python.
This is not a terrible idea. It is a tool that makes it possible for people to either retrofit a newer version of Python onto an ageing OS or to develop for a specific older version of Python on a newer OS. If you don't need the features the tool uses, you shouldn't use it--I don't see why you're comparing it to virtualenvs, it provides completely different runtimes with a different engine and a standard library, and the ability to switch between those by just cd'ing to a folder with a .python-version file in them.
Virtual environments are for your installed dependencies, whilst pyenv is for installing python.
I have a client that uses Python X and another that strictly uses Python X+1.
The virtual environments are so that I can have the project dependencies installed and the pyenv lets different companies have different cadence for their Python upgrades.
I could be completely mistaken and mixing up my Python support utils as I've not had a client request Python for a couple of years.
Isn't that more of a limitation with virtualenv? I've been using virtualfish as my venv manager in the fish shell and it has a `vf upgrade` command for upgrading virtualenvs to newer versions, either by updating in-place or rebuilding the entire virtualenv. So far has worked nicely with my pyenv Python installs.
We use Pyenv successfully for developing the Flower open-source project. We use a few simple Bash scripts to manage virtual environments with different Python versions via pyenv and the pyenv-virtualenv plugin.
The main scripts are `venv-create.sh`, `venv-delete.sh` and `bootstrap.sh`. `venv-reset.sh` pulls these three scripts together to make reinstalling your venv a single command.
I've only used Python occasionally, and more than half of that time I ran into issues with the installed Python version or dependencies. Not only version 2 vs 3, but for example a certain project required exactly 3.11, not 3.10 or 3.12. Last time, I tried to install Caddy and broke all previously installed packages/commands - and after searching for a solution and trying different things, the best I could come up with was to switch versions (manually) before and after running a command to restore the environment.
Now I understand I should have been using Pyenv the whole time. But it sure felt unfriendly for a newcomer to the language/ecosystem. Granted my learning approach is to jump in without much preparation, I wish the language designers or the ecosystem provided a better experience when things don't work right.
How is it awful on the Ruby side? In my experience it is a solved problem since bundler was invented 15 years ago. It's been very free from pain during the last 10 years or so.
AFAIR it was mostly because Ruby still wanted to use not-100%-fixed versions so there was still an issue with something updating slightly breaking the build. Just a cursory experience when trying to deal with Ruby. Though it _felt_ somewhat more stable than Python.
In my experience, python has been far, far worse than ruby. Now, to be fair, I’m much more comfortable in the ruby ecosystem, but still, everything feels much more brittle in python package management.
After some trial and error I have now settled on the following "stack": 1)I manage python versions with pyenv 2)For each new project I create a new virtualenv with venv "PYENV_VERSION=3.10 python -m venv .venv" 3)Then I start jump into the venv and initiate the project with Poetry ("poetry init -n") and manage dependencies with Poetry.
If I'm keeping the projects under Dropbox, then I'll just add the .venv folder to the ignore list ("attr -s com.dropbox.ignored -V 1 .venv") to reduce the amount of files that need to be synced. If I need to get back to old project, I simply recreate the venv and install dependecies using Poetry.
A good habit would be to add a .pythonversion file on the project folder. Pyenv can pick that up and it is then obvious which version was used for the project.
I do development on Windows, but run all the Python stuff on WSL2 and use VS code for notebooks and everything. One thing I haven't looked at is using pipx to manage Poetry. Now I simply have one Poetry version installed.
Pyenv has worked fine once I managed to collect all the necessary dependencies to Ubuntu. For me it's not just to switch between Python 3.x. I like the fact that you can easily have different patch versions and match exactly the version I would use in production (via containers).
Personally I've had problems with poetry managing virtualenvs in the past so I just don't let it touch them any more. Maybe it's better now, but I don't see any reason to risk it, given how often Python seems to like to ruin my day.
I also don't like that by default it wants you to use `poetry run` to run things. Sure you can configure it not to, but it still annoys me
This creates the venv against the correct Python version, and I can now do without `pyenv exec` for this repository.
And every time that `.python-version` changes (which I at most do a few times per year per project,) I throw away the `.venv`, do `pyenv install -s` and start over.
Tools you can use to make sure the Python program you wrote keeps working: requirements.txt, pip, pipenv, pyenv, virtualenv, pyenv-virtualenv, virtualenvwrapper, pyenv-virtualenvwrapper, venv, pyvenv, conda, miniconda, poetry, docker, nix.
Which ones did I miss? Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
We still need a generated lock file with every top level dependency and sub-dependencies locked down to their most precise version commit to version control so that when you build your image today or in 6 months you end up with the same result.
This is not true. Nix actually solves the problem. If you package with Nix, you'll get the exact same version of Python with the exact same version of dependencies, including the exact same version of system libraries. Your build will work in a year just as it does today. You don't even need to keep any binary artifacts.
Of course this doesn't come free: packaging with Nix may involve nontrivial effort for some projects.
How does nix solves the problem of Python dependencies not specifying their dependencies well or exhaustively? Do you have to find them out and fix them manually or can you generate a lock file and hope for the best?
It depends. If the dependency is in nixpkgs (Nix' own package repository), someone will have done the work of figuring out the dependencies properly and you can just use that. If it's not, you can either do that yourself and pin particular versions, or you can integrate with a tool like Poetry: Poetry could generate a lockfile for you that you then reference from Nix to get the versions of Python packages. You'd still need to specify any native dependencies manually, though.
Ultimately the true bellwether of compatibility is the package's own test suite, which we try to get working (and integrated into the build process) for as many packages as possible. For packages which have poor/non-existent test suites, often a downstream package's test suite will expose compatibility problems (we've found bugs in zlib security patches using curl's test suite for instance).
nixpkgs maintainers are frequently the first to notify a project author of incompatibility with new versions of another package.
> How does nix solves the problem of Python dependencies not specifying their dependencies well or exhaustively?
In Nix lingo, a package successfully building or running against an implicit or unmanaged dependency is called 'impurity'. To help keep builds 'pure', Nix builds everything in a sandbox and does various other tricks to ensure that a package being built can't/doesn't find any dependencies at build time that you don't explicitly tell Nix to include in the build environment.
(This is also increasingly how Linux distros build their packages, to solve the same problem.)
For the most part, if building (and running the test suite during the build) a Python package with Nix succeeds, you can be confident that you've got all the dependencies sorted out— even the ones upstream forgot to tell you about.
> Do you have to find them out and fix them manually or can you generate a lock file and hope for the best?
At install time, you can be confident that Nix will bring along all of the system-level dependencies your Python package needs. You don't have to find and fill any gaps at install time 10 years from now or whatever.
When you're writing your Nix package for the first time, you'll be doing a mix of generating the Nix code that defines your package from some upstream, Python-specific lockfile and making manual corrections when the build fails. Nix doesn't have any magic for figuring out dependencies that are left out of poetry.lock or whatever, or for disambiguating guaranteed-compatible exact versions from requirements.txt.
Pyinstaller, provided you build it in a reasonably old glibc to avoid glibc incompatibilities. I know that is a caveat but it is a one time build time operation.
Doesn't Pyinstaller also require CPython to have been built with '--enable-shared'? I _think_ this is the default for most system-installed Pythons that I've seen, but it's not the default if you build Python from source or install via pyenv.
It's inherit problem for all languages based on some runtime. Java, PHP, Ruby, whatever have the same issue. And TBH it's not very major issue to start with.
No other solution besides fixed OS (containers, nix) could solve this problem.
I'm not sure what you might mean by runtime, but a python venv does make sure you're using the same runtime.
There are symlinks in the environments bin directory to the specific runtime. If you're plonking a python3.12 binary over the top of the python3.11 one, then yeah, you'll be using a different one, but that'd be an issue nomatter what you're using.
Just don't uninstall your python binaries and you're fine.
If you want to package everything up into am image or a zip, you can do that too if you want, but in my 10+ year career I've not had much of an issue just using boring venvs.
If you attempt to reproduce a project in a repo that only includes source and requirements.txt you will not have metadata about the version of Python used.
> but a python venv does make sure you're using the same runtime.
It does not.
> There are symlinks in the environments bin directory to the specific runtime.
Precisely. They symlink to a path. Which means that if you have a certain version of Python at one location (let’s say /usr/bin/python3) and later update that (let’s say by upgrading macOS and the Xcode developer tools), the same virtual environment will point to a different version of Python.
> Just don't uninstall your python binaries and you're fine.
That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
The point of the question was to reproduce the same thing without asterisks and you’ve introduced a major one and called it a day.
Like I said in the original comment you replied to and are now ignoring:
> That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
Apparently, “sometimes you don’t have a choice” is a foreign language.
— Hey, so you know how we have this requirement, which came about after years of dealing with and understanding a problem and the available solutions?
— Yes, what about it?
— Well, a random commenter on Hacker News who has zero context of the problem has suggested we use a method we already found inadequate for our specific use case.
— Oh wow, in that case let’s replace the whole system right now.
I think people are really looking for some examples of scenarios where this might be a requirement and your time might be better spent providing one vs. fighting with them. Maybe something like an airgapped gov network with very specific approved and audited software?
No what you're saying is "Doc I want to only live on bread", to which a reasonable response is "it sounds like you're in jail and have bigger problems."
I'm not ignoring you; I just don't think your use case of "I want to capture the Python from Xcode in my dev env so it's resilient to changes and upgrades" is something anyone wants (or should want) to do. Do you really want to ship on that version? Are you asking your users to install Xcode?
> That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
In other words: when are you forced to use Xcode's Python for development, and why would that be a good idea? I'm earnestly asking; there may be reasons; I'm just not aware of any.
> I just don't think your use case of "I want to capture the Python from Xcode in my dev env so it's resilient to changes and upgrades" is something anyone wants (or should want) to do.
Then you are wrong. Simple as that. I’m describing a very real scenario.
> Do you really want to ship on that version?
Holy moly, is it really that hard to understand the difference between wanting and having to? For the use case, and older version in a consistent place which is easy to install is the best solution.
> Are you asking your users to install Xcode?
Triggering an Xcode CLI tools installation is simple and done graphically. And it’s one step removed from installing Homebrew or pyenv, which both need them (even for the scripted installation, pyenv requires git).
> In other words: when are you forced to use Xcode's Python for development, and why would that be a good idea?
See, for a moment there you understood it’s about users, not just your own dev environment, but then went back. Unfortunately, after this conversation with you two I no longer have the energy to go through it in detail in an uphill explanation. Another time, maybe.
>> I just don't think your use case of "I want to capture the Python from Xcode in my dev env so it's resilient to changes and upgrades" is something anyone wants (or should want) to do.
> Then you are wrong. Simple as that. I’m describing a very real scenario.
>> Do you really want to ship on that version?
> Holy moly, is it really that hard to understand the difference between wanting and having to? For the use case, and older version in a consistent place which is easy to install is the best solution.
Tone is hard on the internet, but I'm honestly trying to understand your use case. The way I understand it now is "I want to be able to develop Python programs locally using only Xcode's Python." I'm still not sure why you want to (generally when people ship Python programs they bundle a runtime) but let's set that to the side. For this use case, I don't understand why symlinks don't work for you. Xcode installs to versioned folders so you can have multiple Xcode installs side by side, that way new versions won't overwrite things. I'm obviously not an expert here though; am I missing something?
> For this use case, I don't understand why symlinks don't work for you.
I never said that. My assertion was that venv does not ensure the same Python runtime.¹ That’s it. I don’t have a problem with that. But I do know of one situation where it could make a difference and “do it another way” is not a reasonable answer.
If we ever meet in person I’ll gladly explain it in detail.
The binary installs aren't that generic in the machines I've used (mac and debian based, so covers a fair bit).
It's python3.11, not python3, and the python3 "executable" is, itself a symlink to the particular binary (in this example python3.11). Upgrading just changes the symlink, which wouldn't affect the venv, which isn't using python3, it's using python3.11.
I didn't introduce anything, I explained how the links are to the versioned binaries, which isn't what you're stating is happening.
Edit to add: as someone else also points out, you don't even have to use the symlinked versions, you can use --copies
> It's python3.11, not python3, and the python3 "executable" is, itself a symlink to the particular binary (in this example python3.11).
Not for the example I gave. If you’re seeing Python 3.11, you’re definitely not using the /usr/bin/python3 on macOS with is made available by the Xcode CLI tools. That’s at 3.9.6 even on Sonoma.
> as someone else also points out, you don't even have to use the symlinked versions, you can use --copies
Sorry, but yes I am using macOS for work. I use homebrew as xcode is the issue here, not python or macOS. There are solutions to your problem, you just seem resistant to using them.
> There are solutions to your problem, you just seem resistant to using them.
No, it is you who are failing to understand. I’m describing to you a real scenario, but it’s not one that bothers me. I don’t need you to come up with a solution and didn’t ask you for it. You need to understand not everyone has the same requirements and tradeoffs you do.
This is not the way you're supposed to be using pyenv
You're supposed to install a specific version of python in a specific place, with a specific name. Say, /usr/local/python-3.10.6
Use pyenv to use that python. Control that by creating a `.python-version` file that says 3.10.6
You now have a project that uses 3.10.6. Unless, of course, somebody installs a different version in that path - at which point you've got bigger issues
Using pyenv to use `/usr/bin/python3` and hoping for the best misses the point
In this case you don’t count the standard library as part of your Python environment. Since Python version is not specified neither is the standard library.
You've missed: pdm, uv, pip-tools, pipx, rye, and probably some others.
Only pdm and poetry generate cross-platform lock files by default as far as I know, but there are a lot of people trying to solve this problem right now.
It's not an easy problem to solve. Python's package management predates package managers from most other programming languages and Python itself predates Linux. There is a lot of baggage so change is very slow.
Shameless plug: don't forget the wonderful pip-chill for simplifying gigantic requirements files (and for stripping out version numbers to make canaries easier to do).
I am always curious how many people, outside of those building code for third party clients, actually hit this problem? In the 10+ years of using Python I have never had a problem using the core tools. The ecosystem is far from perfect but it has never cause me a problem.
Edit: Wow y'all are some sour people for voting down this question. I truly wonder how often people run into this problem compared to just complaining about it.
I am not sure about what 'core' means in this context.
In my edperience I had many many problems with OS packages vs pip installed ones. There were really strange dependency issues.
In some cases I encountered dependency hell.
Even if somebody said to me that the core means also using virtualized en I would disagree as places over Internet not always explicitly guide you in that route.
Just getting me and one coworker in sync on our dev machines often causes problems. Even more when we try to deploy to a cloud service which relies on pip.
If I were writing applications only for myself it'd be fine, but I've definitely run into issues when writing code with others - teammates, open source repos, etc.
Working in CI in a company that has couple dozens of Python packages: I'd say about once a week. There are some weeks with no incidents, and there are weeks when everything is broken for many days straight.
NB. The latest incident was Friday when I've discovered that some CI pipeline ran `setup.py install` that down the lane invoked easy_install, which doesn't have a policy of ignoring bizarre versions s.a. X.Y.Zrc1 or X.Y.Zb2 etc. It ran aground when it was trying to install scikit-learn which wanted NumPy >=X.Y.Z, but it already installed X.Y.Zb1, and it didn't realize that this version should be OK (also, it shouldn't have installed non-release versions anyways).
I just hit it a few days ago. 3rd party script that I did not write. The dev has lost interest in it 3 years ago. It is written for 2.7.x (and it is a nontrivial amount of code). So my choices are some sort of pyenv thing (or one of them), or fix up the script myself so it runs on 3.12. I lucked out and someone else had already done the second thing. One project I worked on was 2.4.x. If they ever have to update that (which I hope they have) to something more recent I could see them doing it until they port it over. That is not even a lib or anything that is just the main prog.
Python Foundation support ended that long ago, but vendors maintained it for longer. Red Hat support for it through RHEL 7 ends 30 June 2024, however, and my guess is they’re the last ones.
For this use case it is probably fine to have 2.7.x installed to use for this. Just annoyed. The downside to having it that way is this weird stray python installed. Would rather use something to switch it around to make it easier to keep track. But in the end it didnt mater as someone else had already updated it and I did not have to bother. Maybe next time :)
Just a requirements.txt that you can install with pip in a venv works 95% of the time. Unless you need a specific python version, which you only figure out halfway through by reading the documentation. Or you are using pytorch, because on windows and linux the version on pypi lacks GPU support, but there's no good way to encode the download url in your dependencies. Or when people don't properly maintain the requirements.txt, because you are supposed to somehow manually keep it in sync.
I run into some kind of packaging or dependency problem for basically every non-trivial project. Hence the many "solutions", but somehow most end up worse than the problems they attempt to solve
What package managers are people using in other languages to make sure that software "always works the same as when you first wrote it, without asterisks"? I'd like to understand how they solve the "package no longer exists in a central registry" problem.
This is not as much about a package manager as it is about conventions and necessary dependencies.
Standards ensure that going forward language semantics and syntax don't change. Having minimal dependencies ensures program longevity. Package manager cannot solve these problems, no matter how good it is at its job.
Python doesn't have a standard, it's heavily reliant on dependencies which are very plentiful and similarly unregulated. A program written in C that uses only functionality described in some POSIX standard will endure decades unmodified. Even Python helloworld program went stale sometime ago, even though it's just one line.
Dotnet uses Nuget[1]. Packages in the system are immutable, & never changing. They can be unlisted, but never deleted (except in limited & extreme cases, like malware), which means even if a package maintainer stops publishing new versions to the repository, existing packages will continue to be publicly available for as long as Microsoft continues to exist.
Often a package says it works with OS version X, and not X+1. It may be true or false. But what you described does not solve either version of that problem.
I think you're referring to vendoring dependencies? In python/pip for example, you can download the source for a package and point to the folder directly as a dependency instead of the version or a git URL. Most package managers/languages support some version of that. I suppose if you wanted to vendor all dependencies by default, keep them updated etc it would take a little more scripting or extra tools.
> I'd like to understand how they solve the "package no longer exists in a central registry" problem.
This is, of course, an infrastructure/maintenance issue as much as a package manager design issue. But in Nix's case, the public 'binary cache' (for Nixpkgs/NixOS) of build outputs includes not only final build outputs but also the source tarballs that go into them. As Nix disallows network access at build time, all dependencies are represented this way, including jar files or source tarballs, or whatever— Nix itself must be the one to fetch your dependencies. Consequently, everything you fetch from the Internet for your build is a kind of intermediary Nix build that can be cached using the usual Nix tools. The Nix community's public cache has a policy of retaining copies of upstream sources forever (there is recently talk of limiting storage of the final built packages to a retention period of only 2 years, but sources will continue to be retained indefinitely. So far the cache reaches back to its inception around a decade ago.)
Taken together, these things mean that when a project disappears entirely from GitHub or Maven Central or whatever, people building against old versions of it with Nix/Nixpkgs don't even notice. Nix just fetches those upstream sources from the public cache without even reaching out to that central repository from which those sources have been removed.
For private use cases where your project and its dependencies won't be mirrored to the public cache of Nixpkgs builds, you can achieve the same effect by running your own cache or paying a hosted service to do that.
For builds outside the Nix universe, you can make special arrangements for each type of package your various builds fetch, and mirroring those repos. Then configure your builds to pull from your mirrors instead of the main/public ones.
asdf can be used as an alternative to pyenv. (In fact, it is not only meant for the Python ecosystem, so it can also replace nvm and others.)
For me, the combination of asdf and Poetry has worked quite well recently: I use asdf to pin the Python & Poetry version and then use Poetry to pin everything else.
You didn’t miss Poetry, but I have to say up until I started using Poetry python in large projects was a pain in the tooling department to setup and maintain over longer periods of time.
It’s no panacea, but feels more stable and usable (especially from onboarding new team members PoV) than other tooling I’ve tried
To add to this, with poetry, you basically do `poetry env use python3.12` to create a python virtualenv on python3.12 (you can use whatever python version pyenv supports, doesn't even have to be CPython).
The generated virtual env name is a little wonky. I'm not sure exactly what the scheme is, but it's basically `$(package)-$(some sort of hash?)-$(pyversion)`. I can't speak for all tools, but at least VS Code detects the poetry env and suggests it (and indicates it as a Poetry env) when you go to configure the project interpreter.
> Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
There aren't such tools. Python not being a standard and heavily reliant on the OS that runs it and on third-party components that are also not standard leaves you with no choice by to "be at the wheel" all the time. Virtually anything written in Python will go stale in a mater of few years. In other words, you need to constantly update and test as the environment changes just to stand still.
Looks like what could be a shell alias to cd and activate? I always keep a few tabs open to my repos folder, so not sure it would help me much. Doesn't seem to support fish, though activate does.
I modified my bash prompt to detect and auto-activate Python environments, and show if one is currently active. Haven’t thought about activate in years. It’s great.
Yes, I did that on my work laptop with fish, since I only ever used it with the work project.
On my personal machine I don't bother with venvs, so also haven't thought about it in years. But was trying to figure out what the "workon" command did for GP.
`workon` lists all virtualenvs you have in your virtualenvs folder and `workon foo` activates environment foo
this way I have all my envs in ~/virtual and all my projects in ~/projects and just `workon xyz` when I want to be in a certain venv for a given project (which doesn't always map one-to-one)
I remember when I was a childish Perl fan trading insults with the childish Python fans of the time, one of the things "they" always chided "us" about was that we had too many different ways to accomplish the same thing...
I have been using pyenv for years and it's the best tool for managing Python versions. You can install any version, switch between them, use different versions for different projects, have multiple versions installed at the same type without problems. It really solved all my Python version managament problems, I never looked for any other tool like this since I started using it.
I've been using pyenv for several years now, but for some reason, the basic commands and overall integration don't feel as smooth as Node's nvm package. I wonder if that's because Python setup is technically harder than Node.
I've given up on trying to manage runtime dependencies with language specific tooling. Instead, I've moved to mise which handles the majority of them (Python, Node, Ruby, Terraform, etc.) the same way. It will also activate Python virtual envs:
I'm using pyenv to manage different projects on my machine. I use --system-site-packages to share libraries like torch, and some others -- especially to ensure those libraries are built with the same cuda version, avoid some issues when upgrading drivres, and to avoid having to keep multiple copies of several gigabyte libraries around.
I'm not using things like anaconda for a similar reason I'm not using docker. -- I don't want a separate OS install for each service I'm running on my server. It works perfectly fine.
Just use the builtin "python -m venv" and life is good, why the others?
I tried almost all of them, with various issues, now staying with the default venv, it's solid and get the job done.
The problem there is that by default a virtual environment version of Python is just a symbolic link to the actual binaries installed elsewhere on the machine. If you do an OS upgrade or do anything that increments the system Python version, it could cause problems in the environment.
pyenv manages python versions, not virtual environments. venv won't help you grab python versions you don't have, and will happily install your requirements into a venv with the wrong python version.
The premise of controlling python version is that - you're sending the project to your friend / colleague and you want both of you to use the same python version so that you get the same behaviour.
And then the same for your friend the buildserver.
venv only works if you already have the required version of python installed and have it active as the python for your shell (or use a launcher to which you specify the python to use in the command line), since venv just creates a virtual environment based on the version of python it is run with.
So you actually need something that can read a specification of the required python version and use the correct one from the available options (maybe even reaching out and getting it if it isn't already locally available, though I don5 remeber if any of the existing python build tools will do this; docker obviously will, but its not really a python build tool), across different OS flavors (maybe not that last bit, depending on the use case, especially for internal development), and with minimal overhead for the build tool itself.
Not really a complaint - but I am surprised to see this on the front page of HN. This is an enormously popular project that's been around for over a decade that basically anyone who writes Python regularly is aware of.
Only marginally less surprising than seeing a link directly to python.org.
https://asdf-vm.com/ ASDF is better because it works with many more languages, other than only Python, like Rust, Go, Node, etc, and other tools, such as AWS/Google/Firebase/Azure CLIs.
Curious what you like about mise! I've been a strong advocate for asdf for a few years now. Aside from smooshing together tools that are often used together, but not always (direnv, make), I can't see what mise adds to the picture. And I like still having direnv for what direnv does and make for make does, rather than an all-in-one.
Damn. I'm convinced. The asdf CLI was always a bit of a pain ... mostly I don't interact with it that often. But mise seems light years better designed. Thanks!
You should check out mise (https://github.com/jdx/mise), it is very similar to ASDF (multi langs) but written in Rust instead of Bash, and doesnt use the shim technique that ASDF does. My shell startup went from a couple hundred ms to a couple dozen ms.
There has been tons of churn in the Python project management space, and I feel like I've been blissfully unaware of all of it with this workflow. Can't recommend enough.
Why regular Python binaries and not pyenv? I've had enough pyenv snafus and meltdowns that I started looking for alternatives, and it turned out I could just install whatever Python binary I wanted and specify it wherever I wanted. What could be easier?
python -m venv is fine, I've just gotten used to virtualenv.
pyproject.toml is the future and almost everything supports it pretty well; it also has fewer implicit weirdnesses than setup.py or setup.cfg.
pip-tools are really simple, and they let you have hashed dependencies which are super important IMO.
You can specify different dependencies in pyproject.toml and then use --extra with pip-compile to pull them in. I do this with dev and test sections for example. Constraints seem like they work like you'd expect here. pip-compile also has the -c flag if you want to use that.
I guess I don't know what you mean by OS stuff, but maybe this works for that too?
Hmm. So you generate a requirements.txt and a dev-requirements.txt. It sounds like they have overlapping dependencies? I can't get them to be "layered" where the dev-requirements.txt only has the dev dependencies.
This is the sort of tool that experienced devs all use, but nobody teaches or writes it into a tutorial on the larger language as a whole, because it feels like a side tool.
For now I'm not seeing a lot of reasons to use PyEnv over the `venv` module that ships with Python 3.3+ [1]. I'm sure there are some thing PyEnv does that `venv` doesn't, but the fact that `venv` ships with Python greatly simplifies things.
Basically my workflow is:
1. I'll create a Python project with, `mkdir dirname`, `cd dirname`, `git init`, `python3 -m venv .env`. This creates a hidden folder named `.env/` which contains the virtual environment. I'll then usually add `.env/` to my `.gitignore`, and also run `pip install --upgrade pip` and `pip install wheel`.
2. If I'm using an existing project, I'll do `git clone `projname`, `cd dirname`, `python3 -m venv .env`, `pip install -m requirements.txt` (requirements.txt is the idiomatic name for the dependencies list in Python projects).
3. I have the following lines in my `~/.bashrc` (hidden file that contains Bash settings):
# Gets a directory named .env or .venv if it exists in the currend directory or any of its parents
get_env() {
if [ -d "$1/.env" ] ; then
echo "$1/.env"
else
if [ -d "$1/.venv" ] ; then
echo "$1/.venv"
else
if [ -d "$1/.." ] ; then
get_env "$1/.."
fi
fi
fi
}
get_absolute_path() {
python3 -c "import os; print(os.path.realpath('$1'))"
}
on_prompt() {
# Load a virtualenv environment if it exists in a file named .env
env_folder=$(get_env $(pwd))
if [ -d "$env_folder" ] ; then
if [[ $VIRTUAL_ENV != $(get_absolute_path $env_folder) ]] ; then
echo "Activating env '$env_folder'"
source "$env_folder/bin/activate"
fi
else
if [ -d "$VIRTUAL_ENV" ] ; then
deactivate
fi
fi
}
# Call on_prompt() every time the command prompt executes
PROMPT_COMMAND=on_prompt
What this does is when I `cd` or `pushd` into a directory or subdirectory of a directory that contains a `.env/` folder, it loads the virtual environment, and when I leave said directories, it exits the virtual environment.
4. When I install a dependency, I use `pip freeze` to get the dependency string, and I append that line to the file `requirements.txt`. For example, if I do `pip install django`, I get `Django==5.0.1` in my output from `pip freeze`, so I'll append that line to my `requirements.txt`.
5. To upgrade dependencies, I edit the version number in the `requirements.txt` file, and then run `pip install -m requirements.txt`. This makes sure that the requirements file stays up-to-date with my locally installed dependencies.
6. To get valid version numbers for a file, you can do `pip install <package>==`, which is basically asking pip to install an invalid version number. This causes it to list valid version numbers. For example, you can do `pip install django==` to view all available Django versions.
340 comments
[ 4.1 ms ] story [ 1116 ms ] threadSo fast it finally made virtual environments usable for me. But it's not (yet) a full replacement for conda, e.g. it won't install things outside of Python packages
A notable open issue in poetry is we can't currently specify one dependency on torch, and have it grab CPU version on some systems and GPU on others. Does PDM solve that?
However, this works perfectly with Poetry 1.8 and Pytorch 2.2. I suppose the only problem is what PDM also does, where the lock file is platform-dependent. I'm not sure whether Poetry allows you to select a specific lock file, however.
This link shows which package versions come in which Docker tag and is invaluable: https://docs.nvidia.com/deeplearning/frameworks/support-matr...
Nowadays, most DS people only want to do ML at the experimental stage only and get lost when things get on the engineering side of things. But for their defense, nowadays the bare minimum skills require to do programming, containerization, CI/CD, etc. More experienced and swiss army knife SWE/MLE have to educate the willing.
It was already the same 10 years ago with MATLAB dudes not wanting to get dirty with C/C++/ASM SIMD. The history repeats itself, only at a faster pace
There's a very good chance I've installed cuda on my system before this though. And usually cudnn and some other packages because this is part of my standard install. And then I also never run into the issue where a package is looking for nvcc.
We make extensive use of conda/mamba to solve this, and are pretty happy with it, especially with conda-forge.
It's definitely a learning curve, but it turns out every conda user has been bit by the irreproducible tendencies of conda quite often. Nobody uses the conda env file, they just start an env and pip install things into it. They don't realize the base env has stuff, too, and conda envs are hierarchical rather than isolated. I know it's possible to use conda in an isolated and reproducible way, but have yet to meet someone that does so.
So it hasn't been hard to pitch poetry to these folks, and while many complain about the learning curve they appreciate the outcomes.
We're a pytorch shop, and torch mostly just works with pip or poetry these days, as long as you skip the versions the torch maintainers mispackaged. We rarely need anything higher-level that only conda could install.
We really like having more than two dependency groups as this allows us to keep research and production in the same repository. main, dev, research. Then researchers contribute to the core library of a project and keep research and production using the same code for running and evaluating models.
I haven't been using anything CUDA, but the scientific geospatial stack is often a similar mess to install, and it's been handling it really well.
They will try it, and most of them will pay the price for it.
It's a cycle.
The number of possible modes of failure in this situation is huge.
See also: "Why not tell people to "simply" use pyenv, poetry or anaconda"
https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
I'm not saying pyenv is not a useful tool, but it is not a tool for beginners fighting with python packaging problems. It's a specialist tool to normalize your setup.
Very often, I see people tell me they don't have problems with pyenv, but later on have other unrelated problems with their dependencies. Analysis then prove it was because of pyenv, they just didn't know it. The cost is not obvious.
What is the alternative? Every solution that I know of on Linux requires you to build Python on the machine: asdf, official Python downloads, etc.
I'll leave further discussion of the reliability and provenance of said binaries to someone else.
Still, it's part of those new generation of tooling that bringing hope for the next few years.
Using per-distro official installation channels.
E.g. using deadsnakes PPA on Ubuntu, AUR on Arch Linux, etc.
Those are what Rye and hatch use.
Drawbacks: late availability of patch versions, various quirks from how they are built (missing readline, missing some build info that self-compiled C python modules might need.)
Not only does the format need to exist but the service of building and publishing them is needed too.
For linux, official repos are ideal. If you really, really can't (which is different than wanting to), ubuntu deadsnake and red hat epl are the best second plan, while already more finicky.
If you use something more exotic, you chose hardship, and you will have to be up to the task.
Anything else will come with bigger caveats than the people promoting them will really admit to.
The story is of course a bit richer because linux packaging is fun, so I'll complete with:
https://www.bitecode.dev/p/installing-python-the-bare-minimu...
Thete are currently no silver bullet to bootstrap python on linux. I tried all of them with hundred of colleagues and trainees.
I do have hope for astral to come up with one but don't run on rye thinking your life is saved.
Macports and homebrew Pythons are dependencies of other packages. They can be used by you, but they are not meant for you.
This means at some point, they will contain a surprise, and not a good one.
Sample of one, but I never encountered anything even broken in MacPorts. They seem to embrace the BSD ethos of doing everything right (even if at a slower pace, as some packages lag a few releases behind Homebrew).
Unless you need a Python that's not supported by your Linux distribution, you can just use what's available.
On macOS, MacPorts provides compiled versions for 3.2 all the way to 3.13, as well as 2.6 and 2.7. Right now, I have 3.8, 3.9, 3.10, 3.11, 3.12, and a 3.13 development build. The fact it's not Linux (or x86) might cause some frustration.
My understanding of their comment is that they’re talking about getting older Python interpreters to run on more modern OSes, modern enough that they don’t carry the older Python as a system package anymore. Hence, deadsnakes.
That's indeed extremely specific. I'd imagine this pain could be self-inflicted with C-based extensions that were compiled (and can't be recompiled) with structures that don't exist in other versions.
I don't want to imagine what other eldritch horrors await developers working on this application.
I also imagine the scenario you mentioned causing a lock to a patch version. But I see this as just a normal steady state a lot of orgs drift to if not staying on top of version updates
Drop by and send thanks cause he really needs them.
The big advantage for me is that I can match whatever runtime and standard library a target has (and yes, that's needed more times than not, even in this new age of Docker).
Additionally, you can build an _optimized_ Python. I have this set for my builds:
[1]: https://taoofmac.com/space/blog/2015/10/03/1245E.G, I'm on Ubuntu 20.04 on an Dell XPS, a fairly standard machine, I'll get:
There are obviously binaries for this Python, since I can apt install it and I didn't specify the minor version.But by default it downloads the source, and compiles it.
On top of that it will use a shim which comes with its own world of possible pain.
Again, I don't want to bash on pyenv.
But I do want to lower people's expectations.
It's a tool for experts, not beginners.
Really? I regularly use this on Linux exactly because it compiles from source rather than using the system provided Python which often have patches in them that breaks pip's test suite.
I've never encountered it trying to use the system provided Python, but maybe there's a weird quirk in the way I am using it.
$ PYTHON_CONFIGURE_OPTS='--disable-ipv6 --enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' pyenv install 3.12.2 python-build: use openssl@3 from homebrew python-build: use readline from homebrew Downloading Python-3.12.2.tar.xz... -> https://www.python.org/ftp/python/3.12.2/Python-3.12.2.tar.x... Installing Python-3.12.2... python-build: use readline from homebrew python-build: use ncurses from homebrew python-build: use zlib from xcode sdk
BUILD FAILED (OS X 14.4 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/w9/xvxzj68j6kx7m480rnwq6hvh0000gn/T/python-build.20240325114837.43577 Results logged to /var/folders/w9/xvxzj68j6kx7m480rnwq6hvh0000gn/T/python-build.20240325114837.43577.log
Last 10 log lines: ./Include/internal/pycore_interp.h:193:24: error: field has incomplete type 'struct _dtoa_state' struct _dtoa_state dtoa; ^ ./Include/internal/pycore_interp.h:193:12: note: forward declaration of 'struct _dtoa_state' struct _dtoa_state dtoa; ^ 1 error generated. make[2]: ** [Objects/boolobject.o] Error 1 make[1]: ** [profile-gen-stamp] Error 2 make: ** [profile-run-stamp] Error 2
I’m sure he is trying to promote some sort of work flow in that article but I don’t understand which.
> https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
Just curious: what are the downsides of poetry installed with pipx? The article mentions having to install poetry in another venv, but that's hardly an issue with pipx (you just add an 'x' after 'pip'), and installing pipx is as straight-forward as it can be.
Coming from the Homebrew/Ruby ecosystem - Hey @mikemcquaid - installing a entirely separate package manager just to deal with a few projects felt like the wrong thing to do.
Occasionally, I have still needed to compile Python myself in order to get things to work, which isn't guaranteed not to blow up w/ `brew`, but this has become far less common of late.
That is, the pattern seems to be that someone mentions a solution, then a zillion responses as to why it sucks.
Is there any tool or pair that sucks least for most cases and beginners? I get that every case is different, but perhaps there are some useful starting points?
I think his point was that you shouldn't pretend to users that just switching to pyenv is the solution.
It's clearly not because most people successfully use it fine.
The problem of distribution and packaging is often a matter of user expectations vs. the actual problem.
The user expectations is that Python is a high level language and will run the same across different machines regardless of OS and Hardware.
The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. So people can end up moving the the glue of the project first and then expect everything else to just work.
Things are getting better (e.g. https://lukeplant.me.uk/blog/posts/python-packaging-must-be-...), a lot of people have put a lot of work into standards and improving the ecosystem, but it is a hard problem that most other popular languages don't interface with nearly as much.
As someone who has managed Python distributions in a large company and who triages issues on the pip github issue page that my anecdotal experience is things are getting better.
The only hard statistic I can point to is the number of top packages on PyPI that offer wheels has substantially gone up, and is close to 100% in the top 500.
And yeah, in general there are significantly less NodeJS third party packages interfacing with packages that directly depend on OSes and hardware. Python has many third party packages that are older than NodeJS that depend on foreign function interfaces, win32 com apis, directly talking to graphics shaders, etc.
Well, no because it's perfectly possible to successfully use a horribly broken system. I use Python "successfully", it just meant I have spend probably literal weeks of my life fighting pip and virtualenv and relative imports and finding I need flags like `--config-settings editable_mode=compat`.
> The problem of distribution and packaging is often a matter of user expectations vs. the actual problem.
Ha yes, I expect it to work reliably and simply and it doesn't!
> The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting.
That's completely irrelevant to the kind of problems I was talking about. I run into issues with Python failing to compile C libraries relatively rarely! Even compiling Python itself seems to work quite well (maybe not surprising since that's one of the only ways to get a new version on Linux).
It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go:
https://stackoverflow.com/questions/14132789/relative-import...
> https://stackoverflow.com/questions/14132789/relative-import...
I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc.
Though the specific example you linked was largely solved with Python 3, there was a lot of confusion during the 2 to 3 transition because people had to support both behaviors, but most people don't have to think about Python 2 any more.
I can assure you it absolutely was not.
> I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc.
Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad.
In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions.
It's similar to the situation with C/C++ - worse in some ways, better in others (at least there is a de facto package registry in Python). In some ways it's because both languages are very old and predate the idea that packaging should be easy and reliable. That's fine, but please don't pretend that it is easy and reliable now.
The question, as posted, was a confusion about how Python 2 relative importing worked, which was indeed bad. I don't know what you think you are pointing out, you haven't said, and the question *is* about Python 2.
> Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad. > > In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions.
I agree that a lot of the solutions were created when there was not an understanding, or thought out design, on what would be a good packaging solution.
But these ill thought out solutions were exactly because of trying to support this wide range of situations, from working on weird OSes, to integrating with strange build systems.
However, what you seemed to have missed is there is now a first party standard on:
* How package installers (pip, poetry, PDM, etc.) should interact with package builders (setuptools, hatchling, etc.)
* How and where build configuration and project metadata should be stored (pyproject.toml)
Almost every popular Python package tool now supports these standards, meaning they all interact with each other pretty well.
Dropping all legacy configuration, and updating the standards to support edge cases is still a long road, but it is a road being travelled and things are getting better.
The common interface is they all use `docker-compose up` and have their editors hooked into the containers.
And of course you'll have to make sure people equally have good practices, since incorrectly using sudo pip install in docker and not a venv are very common.
So again, one possible solution for a certain context, but I wouldn't sell that to most people. Certainly not to beginners.
People writing their pythonanywhere website won't pop up a container, won't they ?
And btw, JS doesn't have this problem, they use NPM.
Its absolutely, by far, the highest friction alternative on Windows for anyone that doesn't use Docker for other purposes.
And, no, docker isn't just install and run on Windows. Before Docker made a heavy push for paid Docker Desktop for even personal use, it was close to that.
But now the way to get a free usable docker command line is to install WSL and a Linux environment, install docker there, and then invoke docker via wsl. (Which, of course, you will not find via Docker’s own information, which will try to sell you a paid subscription.)
1. Download and install Docker desktop for Windows
2. Restart Windows (guess you don't have to do this with just Python)
3. Run Docker Desktop
4. Say "no" to signing into a Docker account
5. Wait for engine to start, which took a few minutes the first time, a bit annoying
6. Pull and run an image (I tried nginx)
It was weird being asked to log in, but the "no" button was pretty clear. I didn't feel like I was forced to use WSL to avoid paying; maybe they've backtracked from something. Unity was far more convincing that I had to pay.
- Not if you have to deploy the code.
- Not if you have to import the code in your own project.
Basically, not if you want to do anything by running the code. For which I would suggest an installer instead.
It does imply, on linux, to limit yourself to the choices of Python you can install. This constraint is, for most people, preferable than the alternative, even if it gets frustrating to our geeky soul.
For your own projects built from scratch (rather than big multi-dep projects off GitHub), there's a smaller learning curve if you go to the vanilla https://www.python.org/downloads/ , install the latest, and use the included pip to install packages. That'll probably get you very far. It's not like the older days when you needed both Py3 and Py2.
For experts working frequently in Python, tools like Pyenv can make more sense.
But, to add to the list of problems, these Python versions IIRC do not compile with optimisation turned on, so they’re by default quite a bit slower than they need to be.
When the God Of Programming made Python, all other languages were jealous of its elegance, simplicity and intuitive beauty. When the other programming languages went to complain he said..."Wait until you see what package systems I will give them...They will never get an environment properly setup..." :-)
”The CUDA version on your system does not match the CUDA version the realm of mortal men was compiled with”
You will know when you care. These days, doing common tasks, the constraint solver in poetry will often tell you:
“Hey! I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2. Sorry! I give up!”
Now, if you aren’t concerned with using your bare metal for CUDA or fancy new MPS or AMD stuff. Just ignore this and use containers. I’d use podman compose.
However, I use pyenv on every machine. Because it compiles specific versions I actually need, even to create various virtual environments. If compiling python automatically sounds tough, you probably don’t need to anyway.
To describe the problem you’d see. I try to use poetry by default, though I think it became popular before it was PEP-compliant or useful in devops. It is impossible to control the behavior of other package managers, and poetry is/was strict about that. Which means you can’t force deploy in many cases. (Better lately.)
For the problem pyenv helps to solve, I back-up my pyproject.toml setuptools backend with pip and requirements.txt. These days, requirements-cuda.txt and requirements-mps.txt.
The landscape is still a disaster for binary compatibility, but it can be done lol. (I’ve been doing python packaging professionally since prom, which was python 2.6 give or take.)
> I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2
I'm a reasonably advanced python user. I've shipped web apps, Desktop GUIs, cli-tools, and even written cpython extension modules for custom hardware control. I typically target the system `python3` of whatever linux distribution I'm shipping to and I use the system 'python3-virtualenv' for venvs.
But I have never encountered a dependency resolution issue or been forced to use poetry. What am I doing wrong?
It’s always the inclusion of a specific dependency we added for a feature, and based on that dev’s knowledge and experience. It’s often me, but not always.
This doesn’t happen in ecosystems with a base package versioning. This is arguably why anaconda became popular, and why we target base docker images of ubuntu.
Doesn’t work in complex deployments based on money rather than ideals, every time. At least in my career.
Edit: first time I dealt with this, we ended up forking a dependency chain instead of using pip. I lost that war and they ended up reviving a legacy PHP app instead of funding python dev.
For example, I want my own Python apps to work equally well on Debian oldstable (which currently provides Python 3.9) and Arch Linux (currently on 3.11). That means that I’m going to choose the lowest common version (3.9) as the language level for my app or library.
And if I program against a 3.9 language level, I absolutely *refuse* to use any other interpreter than a 3.9 one at development time. (If I used a newer one, then my linter and type checker would helpfully give me false alarms all the time, unaware of the actual language level.)
Hence, I use pyenv and poetry to get exactly the interpreter for the language level I want, and to allow pylint and mypy to be perfectly aware of the language level I’m using at development time, allowing them to give me findings that actually make sense.
Not using python on a system without the necessary build tools while trying to use dependencies with native code that don't have binaries built for your combination of python version and platform?
This used to be a huge problem with Python on Windows, as things that weren't pure Python would very often not have binary packages or have them for a narrow set of python versions, often not the same set as other dependencies. (Not just a windows problem, but it was definitely big on windows.)
Tooling and practices have advanced so that more packages are automatically built against a wider set of targets, so the problem is a lot smaller than it used to be.
Also, the issues the article you link are things that damn near every programmer will eventually need to know. Things like PATH are IMO the basics -- if you don't understand this or how to `$packageManager install` something, you're gonna have a rough time programming in general, regardless of language.
I guess, I'm in the "expert" category. I'm saying it so that people won't be afraid to try it.
> It doesn’t support Windows. That’s game over right there, for half of the community.
What do you mean by this? I use pyenv on windows all the time.
I don't get why you just don't set a minimum version like every other manager. Or if you want to do this crazy thing, don't rely on package maintainers to enforce it. I just ends up in major bloat
‘brew pyenv-sync’ may be useful around this area, too.
This is practically the only remaining annoyance I have with the Python ecosystem (relative imports aside). I use some tools, like Glances [0] whose formula relies on a much newer version (3.12) than the actual package requires (3.8) [1].
So when there's a Python update, all of those update as well. I thought I'd fixed this with pipx, but in a way that's worse, because the venvs it builds depend on a specific version of Python existing, which doesn't work well with brew always wanting to upgrade it.
I want a stable, system-level Python that I don't touch, don't add packages to, and which only exists as a dependency for anything that needs it. If an update would break a package I have installed (due to Python library deprecation, etc.), it should warn me before updating. Otherwise, I don't care, as long as any symlinks are taken care of.
Separately, I want a stable, user-level Python that I can do whatever I want to. Nothing updates it automatically. I can accomplish this by compiling Python and using `make altinstall`, but if there's a better way, I'd love to hear about it.
[0]: https://github.com/Homebrew/homebrew-core/blob/20e744191e74d...
[1]: https://github.com/nicolargo/glances
Just looking at a random formula, am I correct to understand that this will use python 3.10 and NOT 3.12?[0] I understand ones like this[1] where there's a note about the issue with newer versions.
What I'm trying to understand is if the python version is specified by the formula or it will default to the newest version. If it requires it to be specified in the formula then doesn't this make it contingent on the maintainer upgrading it every python version? I didn't verify [0], but it looks like it should work with later versions of python, and it it is still using 3.10 then isn't that essentially the maintainers "fault?" Because that's my concern. I can't see how something like this stays updated when it requires a maintainer to update. Seems better to have a >=3.10 and then do ==3.10 if only 3.10 works (odd) or >=3.10 <3.12 if it works for 10 and 11 but not 12. Especially since formulas are often made by people that are not the package developers themselves and we're just reliant upon someone keeping up. I'd rather break upon a new version than have many old pythons installed. I know there's no perfect solution, but we're talking about failure modes. And fwiw, I'd rather it try to use the system or environment python rather than installing a unique version. It just gets confusing when you need to add dependencies for optional stuff that wasn't included in the formula and is extremely non-obvious to a new user.
[0] https://github.com/Homebrew/homebrew-core/blob/12a0f6bbbeda8...
[1] https://github.com/Homebrew/homebrew-core/blob/12a0f6bbbeda8...
(I am a Homebrew maintainer).
Pyenv allows you to use whatever version of Python you want.
On most Linux distros you probably don't want to change your system python to whatever, because there are a lot of system programs that may not be compatible with a version other than what is on the system by default.
Then once you activate it you can make sure your python
just install the different python versions by downloading the binaries or through your package manager and run python3.<whatever_version_you_want> -m venv and the virtual environment will automatically, forever, use that python version when you activate it.
Many developers, not just me, have a similar setup: we use virtual environments everywhere, and if you aren't in one, `python` doesn't even resolve to a symbol.
If I want to write a quick script with no dependencies, I directly call `python3.xx` on it. Otherwise, I create a virtualenv.
Yes, it's a bit harder for beginners, but from a huge amount of experience helping people who are starting up in programming, people have little issue in following a few more instructions. What demolishes beginners is getting into a bad state where nothing works and you don't know why.
Virtual environments are for your installed dependencies, whilst pyenv is for installing python.
I have a client that uses Python X and another that strictly uses Python X+1.
The virtual environments are so that I can have the project dependencies installed and the pyenv lets different companies have different cadence for their Python upgrades.
I could be completely mistaken and mixing up my Python support utils as I've not had a client request Python for a couple of years.
Also, this really isn't a problem -- just blast out of the old venv, make a new one with a new version, and you're good.
The main scripts are `venv-create.sh`, `venv-delete.sh` and `bootstrap.sh`. `venv-reset.sh` pulls these three scripts together to make reinstalling your venv a single command.
Here's the link if anyone is interested: https://github.com/adap/flower/tree/main/dev
Now I understand I should have been using Pyenv the whole time. But it sure felt unfriendly for a newcomer to the language/ecosystem. Granted my learning approach is to jump in without much preparation, I wish the language designers or the ecosystem provided a better experience when things don't work right.
If I'm keeping the projects under Dropbox, then I'll just add the .venv folder to the ignore list ("attr -s com.dropbox.ignored -V 1 .venv") to reduce the amount of files that need to be synced. If I need to get back to old project, I simply recreate the venv and install dependecies using Poetry.
A good habit would be to add a .pythonversion file on the project folder. Pyenv can pick that up and it is then obvious which version was used for the project.
I do development on Windows, but run all the Python stuff on WSL2 and use VS code for notebooks and everything. One thing I haven't looked at is using pipx to manage Poetry. Now I simply have one Poetry version installed.
Pyenv has worked fine once I managed to collect all the necessary dependencies to Ubuntu. For me it's not just to switch between Python 3.x. I like the fact that you can easily have different patch versions and match exactly the version I would use in production (via containers).
Our flow is similar: pyenv (windows and linux), pipx, poetry.
We’ve also defaulted poetry to utilize the current global version of Python and build the venv within the project folder.
And every time that `.python-version` changes (which I at most do a few times per year per project,) I throw away the `.venv`, do `pyenv install -s` and start over.
Your `poetry run` point still stands, though.
Python really makes things difficult :D
You’re absolutely right.
On second thought, I guess I only need these Poetry installs just once anyway: for creating the venv and then never again. That’s not worth it.
So here’s a new method that does away with these extra Poetry installs:
assuming that a `poetry.toml` exists with `virtualenvs.in-project` set to `true`.Going to adopt this variant from now on, I guess. Thank you for your thoughts!
It also ensures that my requirements.txt is sufficient for the code to run.
Which ones did I miss? Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
We still need a generated lock file with every top level dependency and sub-dependencies locked down to their most precise version commit to version control so that when you build your image today or in 6 months you end up with the same result.
Using pip to freeze your dependencies and writing a tiny shell script to generate a lock file at build time is better than nothing to solve this problem with nothing more than pip. It's what I do in https://github.com/nickjj/docker-flask-example and https://github.com/nickjj/docker-django-example. It's not perfect but it solves 80% with minimal complexity.
Of course this doesn't come free: packaging with Nix may involve nontrivial effort for some projects.
nixpkgs maintainers are frequently the first to notify a project author of incompatibility with new versions of another package.
In Nix lingo, a package successfully building or running against an implicit or unmanaged dependency is called 'impurity'. To help keep builds 'pure', Nix builds everything in a sandbox and does various other tricks to ensure that a package being built can't/doesn't find any dependencies at build time that you don't explicitly tell Nix to include in the build environment.
(This is also increasingly how Linux distros build their packages, to solve the same problem.)
For the most part, if building (and running the test suite during the build) a Python package with Nix succeeds, you can be confident that you've got all the dependencies sorted out— even the ones upstream forgot to tell you about.
> Do you have to find them out and fix them manually or can you generate a lock file and hope for the best?
At install time, you can be confident that Nix will bring along all of the system-level dependencies your Python package needs. You don't have to find and fill any gaps at install time 10 years from now or whatever.
When you're writing your Nix package for the first time, you'll be doing a mix of generating the Nix code that defines your package from some upstream, Python-specific lockfile and making manual corrections when the build fails. Nix doesn't have any magic for figuring out dependencies that are left out of poetry.lock or whatever, or for disambiguating guaranteed-compatible exact versions from requirements.txt.
And that's the whole problem. "The same Python environment" doesn't mean much because it underspecifies the full runtime dependency chain.
No other solution besides fixed OS (containers, nix) could solve this problem.
You don't have to use NixOS to use Nix. You can install it on any Linux distro, or MacOS, and use it as your build system.
But not the same Python runtime, which does not fulfil the request of guaranteeing it works the same later without asterisks.
Just don't uninstall your python binaries and you're fine.
If you want to package everything up into am image or a zip, you can do that too if you want, but in my 10+ year career I've not had much of an issue just using boring venvs.
It seems that there is a way to do this in pip but I don’t think it is widely used: https://stackoverflow.com/questions/19559247/requirements-tx...
It does not.
> There are symlinks in the environments bin directory to the specific runtime.
Precisely. They symlink to a path. Which means that if you have a certain version of Python at one location (let’s say /usr/bin/python3) and later update that (let’s say by upgrading macOS and the Xcode developer tools), the same virtual environment will point to a different version of Python.
> Just don't uninstall your python binaries and you're fine.
That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
The point of the question was to reproduce the same thing without asterisks and you’ve introduced a major one and called it a day.
[0]: https://docs.python.org/3/library/venv.html
> Error: This build of python cannot create venvs without using symlinks
— Then don’t raise your arm.
Like I said in the original comment you replied to and are now ignoring:
> That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
- Doctor, it hurts when I hit my head on the wall
- Then don't hit your head on the wall
You have solutions available, use them
— Hey, so you know how we have this requirement, which came about after years of dealing with and understanding a problem and the available solutions?
— Yes, what about it?
— Well, a random commenter on Hacker News who has zero context of the problem has suggested we use a method we already found inadequate for our specific use case.
— Oh wow, in that case let’s replace the whole system right now.
I'm not ignoring you; I just don't think your use case of "I want to capture the Python from Xcode in my dev env so it's resilient to changes and upgrades" is something anyone wants (or should want) to do. Do you really want to ship on that version? Are you asking your users to install Xcode?
> That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
In other words: when are you forced to use Xcode's Python for development, and why would that be a good idea? I'm earnestly asking; there may be reasons; I'm just not aware of any.
Then you are wrong. Simple as that. I’m describing a very real scenario.
> Do you really want to ship on that version?
Holy moly, is it really that hard to understand the difference between wanting and having to? For the use case, and older version in a consistent place which is easy to install is the best solution.
> Are you asking your users to install Xcode?
Triggering an Xcode CLI tools installation is simple and done graphically. And it’s one step removed from installing Homebrew or pyenv, which both need them (even for the scripted installation, pyenv requires git).
> In other words: when are you forced to use Xcode's Python for development, and why would that be a good idea?
See, for a moment there you understood it’s about users, not just your own dev environment, but then went back. Unfortunately, after this conversation with you two I no longer have the energy to go through it in detail in an uphill explanation. Another time, maybe.
> Then you are wrong. Simple as that. I’m describing a very real scenario.
>> Do you really want to ship on that version?
> Holy moly, is it really that hard to understand the difference between wanting and having to? For the use case, and older version in a consistent place which is easy to install is the best solution.
Tone is hard on the internet, but I'm honestly trying to understand your use case. The way I understand it now is "I want to be able to develop Python programs locally using only Xcode's Python." I'm still not sure why you want to (generally when people ship Python programs they bundle a runtime) but let's set that to the side. For this use case, I don't understand why symlinks don't work for you. Xcode installs to versioned folders so you can have multiple Xcode installs side by side, that way new versions won't overwrite things. I'm obviously not an expert here though; am I missing something?
I never said that. My assertion was that venv does not ensure the same Python runtime.¹ That’s it. I don’t have a problem with that. But I do know of one situation where it could make a difference and “do it another way” is not a reasonable answer.
If we ever meet in person I’ll gladly explain it in detail.
¹ https://news.ycombinator.com/item?id=39814963
It's python3.11, not python3, and the python3 "executable" is, itself a symlink to the particular binary (in this example python3.11). Upgrading just changes the symlink, which wouldn't affect the venv, which isn't using python3, it's using python3.11.
I didn't introduce anything, I explained how the links are to the versioned binaries, which isn't what you're stating is happening.
Edit to add: as someone else also points out, you don't even have to use the symlinked versions, you can use --copies
Not for the example I gave. If you’re seeing Python 3.11, you’re definitely not using the /usr/bin/python3 on macOS with is made available by the Xcode CLI tools. That’s at 3.9.6 even on Sonoma.
> as someone else also points out, you don't even have to use the symlinked versions, you can use --copies
Which, again, doesn’t work for the stated case.
https://news.ycombinator.com/item?id=39818732
No, it is you who are failing to understand. I’m describing to you a real scenario, but it’s not one that bothers me. I don’t need you to come up with a solution and didn’t ask you for it. You need to understand not everyone has the same requirements and tradeoffs you do.
You're supposed to install a specific version of python in a specific place, with a specific name. Say, /usr/local/python-3.10.6
Use pyenv to use that python. Control that by creating a `.python-version` file that says 3.10.6
You now have a project that uses 3.10.6. Unless, of course, somebody installs a different version in that path - at which point you've got bigger issues
Using pyenv to use `/usr/bin/python3` and hoping for the best misses the point
Because that’s not what the conversation is about. It’s about virtual environments.
Forget to install virtualenv? Have fun getting rid of packages in your global system.
Sub-dependencies? You'll have to think of a strategy for it. Most people don't and the reproducibility of their project suffers.
Developer vs production dependencies ? You'll have to think of a strategy for it.
And I think I'm forgetting one, but that's already enough.
Using python and python libraries only from your package manager (like APT) for a specific OS version.
Micromamba as well, lightweight version of conda/miniconda.
Only pdm and poetry generate cross-platform lock files by default as far as I know, but there are a lot of people trying to solve this problem right now.
It's not an easy problem to solve. Python's package management predates package managers from most other programming languages and Python itself predates Linux. There is a lot of baggage so change is very slow.
I think rye hits most of the points pretty well, it ensures both python and package versions.
Nix does, if you want to actually invest the effort into the "without asterisks" part.
Edit: Wow y'all are some sour people for voting down this question. I truly wonder how often people run into this problem compared to just complaining about it.
In my edperience I had many many problems with OS packages vs pip installed ones. There were really strange dependency issues.
In some cases I encountered dependency hell.
Even if somebody said to me that the core means also using virtualized en I would disagree as places over Internet not always explicitly guide you in that route.
NB. The latest incident was Friday when I've discovered that some CI pipeline ran `setup.py install` that down the lane invoked easy_install, which doesn't have a policy of ignoring bizarre versions s.a. X.Y.Zrc1 or X.Y.Zb2 etc. It ran aground when it was trying to install scikit-learn which wanted NumPy >=X.Y.Z, but it already installed X.Y.Zb1, and it didn't realize that this version should be OK (also, it shouldn't have installed non-release versions anyways).
I run into some kind of packaging or dependency problem for basically every non-trivial project. Hence the many "solutions", but somehow most end up worse than the problems they attempt to solve
Standards ensure that going forward language semantics and syntax don't change. Having minimal dependencies ensures program longevity. Package manager cannot solve these problems, no matter how good it is at its job.
Python doesn't have a standard, it's heavily reliant on dependencies which are very plentiful and similarly unregulated. A program written in C that uses only functionality described in some POSIX standard will endure decades unmodified. Even Python helloworld program went stale sometime ago, even though it's just one line.
[1] https://www.nuget.org/
Or the "says it will work with > X," but doesn't.
This is, of course, an infrastructure/maintenance issue as much as a package manager design issue. But in Nix's case, the public 'binary cache' (for Nixpkgs/NixOS) of build outputs includes not only final build outputs but also the source tarballs that go into them. As Nix disallows network access at build time, all dependencies are represented this way, including jar files or source tarballs, or whatever— Nix itself must be the one to fetch your dependencies. Consequently, everything you fetch from the Internet for your build is a kind of intermediary Nix build that can be cached using the usual Nix tools. The Nix community's public cache has a policy of retaining copies of upstream sources forever (there is recently talk of limiting storage of the final built packages to a retention period of only 2 years, but sources will continue to be retained indefinitely. So far the cache reaches back to its inception around a decade ago.)
Taken together, these things mean that when a project disappears entirely from GitHub or Maven Central or whatever, people building against old versions of it with Nix/Nixpkgs don't even notice. Nix just fetches those upstream sources from the public cache without even reaching out to that central repository from which those sources have been removed.
For private use cases where your project and its dependencies won't be mirrored to the public cache of Nixpkgs builds, you can achieve the same effect by running your own cache or paying a hosted service to do that.
For builds outside the Nix universe, you can make special arrangements for each type of package your various builds fetch, and mirroring those repos. Then configure your builds to pull from your mirrors instead of the main/public ones.
For me, the combination of asdf and Poetry has worked quite well recently: I use asdf to pin the Python & Poetry version and then use Poetry to pin everything else.
https://github.com/asdf-community/asdf-python?tab=readme-ov-...
[0] https://tox.wiki [1] https://nox.thea.codes/en/stable/
It’s no panacea, but feels more stable and usable (especially from onboarding new team members PoV) than other tooling I’ve tried
The generated virtual env name is a little wonky. I'm not sure exactly what the scheme is, but it's basically `$(package)-$(some sort of hash?)-$(pyversion)`. I can't speak for all tools, but at least VS Code detects the poetry env and suggests it (and indicates it as a Poetry env) when you go to configure the project interpreter.
There aren't such tools. Python not being a standard and heavily reliant on the OS that runs it and on third-party components that are also not standard leaves you with no choice by to "be at the wheel" all the time. Virtually anything written in Python will go stale in a mater of few years. In other words, you need to constantly update and test as the environment changes just to stand still.
The other one is Docker, which some people hate. I don't mind it, in some cases I prefer it.
requirements.txt is a text file, not a separate tool...
if you want your program to work exactly as intended in the future, there are tools like py2exe and py2app for that.
On my personal machine I don't bother with venvs, so also haven't thought about it in years. But was trying to figure out what the "workon" command did for GP.
this way I have all my envs in ~/virtual and all my projects in ~/projects and just `workon xyz` when I want to be in a certain venv for a given project (which doesn't always map one-to-one)
Edit: Oh I see you didn't. I can't read.
https://mise.jdx.dev/lang/python.html
It has other nice helpers for development environments (tasks, env variables, etc.).
I'm not using things like anaconda for a similar reason I'm not using docker. -- I don't want a separate OS install for each service I'm running on my server. It works perfectly fine.
And then the same for your friend the buildserver.
It's all about reproducibility.
in the "worst" case you can always do a docker
So you actually need something that can read a specification of the required python version and use the correct one from the available options (maybe even reaching out and getting it if it isn't already locally available, though I don5 remeber if any of the existing python build tools will do this; docker obviously will, but its not really a python build tool), across different OS flavors (maybe not that last bit, depending on the use case, especially for internal development), and with minimal overhead for the build tool itself.
Only marginally less surprising than seeing a link directly to python.org.
(Also I like mise better currently: https://github.com/jdx/mise)
Even if you just use mise as an asdf alternative, it has a nicer CLI and interacts with the same plugin ecosystem more smoothly.
(I've found it to be an improvement on direnv, but I still use make)
the comparison to asdf page has more: https://mise.jdx.dev/dev-tools/comparison-to-asdf.html
- Download whatever Python binaries I need from python.org
- virtualenv --python=$PYTHON_VERSION ~/.virtualenvs/$PROJECT_NAME
- pyproject.toml
- pip-compile --generate-hashes
There has been tons of churn in the Python project management space, and I feel like I've been blissfully unaware of all of it with this workflow. Can't recommend enough.
Why regular Python binaries and not pyenv? I've had enough pyenv snafus and meltdowns that I started looking for alternatives, and it turned out I could just install whatever Python binary I wanted and specify it wherever I wanted. What could be easier?
python -m venv is fine, I've just gotten used to virtualenv.
pyproject.toml is the future and almost everything supports it pretty well; it also has fewer implicit weirdnesses than setup.py or setup.cfg.
pip-tools are really simple, and they let you have hashed dependencies which are super important IMO.
It's also my understanding that pip-tools does not let you make layered requirements files with pyproject.toml files, only with .in files
I guess I don't know what you mean by OS stuff, but maybe this works for that too?
For now I'm not seeing a lot of reasons to use PyEnv over the `venv` module that ships with Python 3.3+ [1]. I'm sure there are some thing PyEnv does that `venv` doesn't, but the fact that `venv` ships with Python greatly simplifies things.
Basically my workflow is:
1. I'll create a Python project with, `mkdir dirname`, `cd dirname`, `git init`, `python3 -m venv .env`. This creates a hidden folder named `.env/` which contains the virtual environment. I'll then usually add `.env/` to my `.gitignore`, and also run `pip install --upgrade pip` and `pip install wheel`.
2. If I'm using an existing project, I'll do `git clone `projname`, `cd dirname`, `python3 -m venv .env`, `pip install -m requirements.txt` (requirements.txt is the idiomatic name for the dependencies list in Python projects).
3. I have the following lines in my `~/.bashrc` (hidden file that contains Bash settings):
What this does is when I `cd` or `pushd` into a directory or subdirectory of a directory that contains a `.env/` folder, it loads the virtual environment, and when I leave said directories, it exits the virtual environment.4. When I install a dependency, I use `pip freeze` to get the dependency string, and I append that line to the file `requirements.txt`. For example, if I do `pip install django`, I get `Django==5.0.1` in my output from `pip freeze`, so I'll append that line to my `requirements.txt`.
5. To upgrade dependencies, I edit the version number in the `requirements.txt` file, and then run `pip install -m requirements.txt`. This makes sure that the requirements file stays up-to-date with my locally installed dependencies.
6. To get valid version numbers for a file, you can do `pip install <package>==`, which is basically asking pip to install an invalid version number. This causes it to list valid version numbers. For example, you can do `pip install django==` to view all available Django versions.
[1] https://docs.python.org/3/library/venv.html