>Now you may be saying, "I always install the latest versions, so that would mean Python 3.8.0 was installed last since it's newer than 3.7.5". OK, but what happens when Python 3.7.6 comes out? Your pip command would have gone from using Python 3.8 to Python 3.7.
This seems less like a reason to use `python -m pip` and more a reason to endorse languages with saner versioning and less need to sandbox twenty different versions from one another.
Golang is really good at a lot of the things Python is not (concurrency, package management, tooling, docs, types, good standard lib, easy to distribute...).
Golangs package management is almost objectively a dumpster fire, confirmed by the fact that they’ve made at least 3 whacks at trying different esoteric things. There’s a lot of things that are nice about Go but package management is definitely not one of them.
This isn't true since `go mod` in Go 1.12. It's as good as Cargo and Lein now (both of which are also excellent). I'd recommend Rust and Clojure as well but Golang seems like a good next step for Python devs since it's the most similar.
This is certainly not one of the languages that the author is referring to, but I super appreciate how nodejs sandboxes modules with no fuss. Even within python virtualenvs I run into issues all the time where 2 modules have conflicting versions of a dependency and all hell breaks loose. The tradeoff is hard drive space but f-ing take it, that resource is the least constrained & the best fit to abuse in 2019.
Nah - what you simply need to do is to access the pip binary in the directory of the virtualenv (using PATH) you’re working with and simply stop using the system interpreter.
That being said I would like to inquire which languages _you_ could give as an example that work perfectly well with the system-provided compilers when you are using the latest features.
The same issue occurs with Node and NPM. I don't want to speak for Ruby as I haven't used Ruby extensively but I can't imagine that Gem's and Ruby versions don't have the same issue.
Honestly virtualenv's solve most of these problems. If using a virtualenv seem to low level or a pain, I can't recommend Poetry [0] enough. It uses virtualenv's in the background, and has from my experience a great dependency resolver.
They're complex, because they handle a wide variety of tasks. Maven, for example, supports multiple languages - you want to have a project mixing Scala, Kotlin, Java and Clojure? No problem. Well, maybe some problems, depends on how well you've kept your exposed APIs interoperable.
Want to ship all your app and all it's dependencies in a single file, sweet as, use the shade plugin, or maybe the capsule plugin. etc. Etc.
I've never had a JVM dependency break because I upgraded Maven. I have had that exact issue with Python - we're using Apache Airflow, and anyone who installed the latest version of Pip hit an issue - Pip had changed an API that Python dependencies depended on, so Airflow could no longer be installed by that version of Pip, because it was written for the old API. Have fun googling "TypeError: 'module' object is not callable" to enjoy that rabbit hole.
Not to mention that it was impossible to generate a pipfile.lock for Airflow because its dependencies had rules demanding incompatible versions of dependencies (one wanted <= 7.0.0, the other wanted > 7.0.0) - in Maven I can easily exclude one of the two competing dependencies.
Oh, and then there was the time that a Python dependency used by Airflow required me to set an env var to handle a licensing issue. Can't even do that in Maven, which strikes me as a feature.
pip3 install apache-airflow
RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when you install or upgrade Airflow. To force installing the GPL version set AIRFLOW_GPL_UNIDECODE
Also, your simple requirements.txt didn't mention how you manage virtualenvs.
Lastly, Gradle is being used quite a bit, but Maven is still pretty dominant. It really depends on if you want a purely declarative build tool like Maven, or one you can drop scripts into like Gradle. I prefer purely declarative for reproducible buils.
> I've never had a JVM dependency break because I upgraded Maven. I have had that exact issue with Python - we're using Apache Airflow, and anyone who installed the latest version of Pip hit an issue - Pip had changed an API that Python dependencies depended on, so Airflow could no longer be installed by that version of Pip, because it was written for the old API. Have fun googling "TypeError: 'module' object is not callable" to enjoy that rabbit hole.
Pip explicitly has no API (except for its command line interface), even the Pip namespace implicates this (pip._internal). So if some dependency is using a Pip "API" they're wrong.
Now, I am developing an application that needs to interface with Pip, in this case it may be appropriate to use the internal Pip API if you pin your Pip version in requirements.txt. However a library shouldn't do this ever.
Sure, it's wrong, not going to disagree. But it happened with a "plain ol requirements.txt", hence why I don't rate Python dependency management as highly as you do.
What do you mean "plain ol requirements.txt" in this case? Either you're pinning your dependencies in requirements.txt (in this case, you wouldn't have a problem since requirements.txt is acting like a lockfile) or you're doing it wrong. I mean, there is no problem to depend in a internal API if you're locking the versions, if not, just assume the consequences.
BTW, I don't rate Python dependency management very high: most other languages have better solutions. However it is manageable.
At least half of that is figuring out what the classpath should be. And the worst part, it is all exposed to user, too -- when I was setting up Spark, I had to muck with class paths a lot.
So JVM has 3 big problems: (1) No single way to set up dependency path, each project is different; (2) No way to set up classpath in JVM language, has to use some other one (like bash); (3) Somehow, user has to fix it for many semin-advanced programs. No other ecosystem I know of is this insane! Python, node.js, even C++ are all better!
Okay, so you've changed the context from "I'm a developer" to "I'm an end user" - how many Python end users are having to bugger about with virtualenvs?
Okay, so, speaking for myself, we generally ship fat jars or capsules these Docker days. I can't speak as to why Apache Spark chose to do it that way, I don't run their project. But if I'm guessing I suspect it's so you can provide your own Hadoop dependencies as needed, as there's a bunch of varying distributions that people use (Cloudera, Hortonworks etc.) and they're trying to play nice with the existing Hadoop ecosystem - probably also to make it possible to extend it with any other jars that you want to use (although spark-shell has a nice feature where you provide additional dependencies on the command line, or within the shell, using Maven artifact coordinates)
No, OP case is for developers setting up their development machines. I feel his pain since I work with Clojure, that 99% of time it simply works however in the 1% of time that it breaks its is painful to fix.
BTW, the uberjar equivalent in Python is something like PEX/shiv/zipapps.
looks like a custom one.. and non-relocatable, too -- if I wanted to install "jmxterm" to my home dir, I'd need to manually edit the script.
(and yes, this is totally "I am developer" issue, not just "I am a user" one -- the dev had to write the script, they had to debug it, and dev would be handling bug reports about inability to invoke from command line when installed to other location)
> looks like a custom one.. and non-relocatable, too -- if I wanted to install "jmxterm" to my home dir, I'd need to manually edit the script
Funny how you had to go searching to find that script, when the installation instructions are simply - 1) download uberjar, 2) run uberjar. Also, you had to go searching to find a script 8 years old, in a project with a commit 20 days ago, to attempt to prove your point.
Surely you must realise you're grasping at this stage.
It creates more and worse problems than it solves. People find it alright to write code that only works in a specific combination of language and package versions, and that is a shameful state of affairs.
I solve this whole issue by using pyenv + direnv. I can have a project specific .python-version file and upgrade to a newer python version when necessary. Then I just run python -m venv env once on project creation and have direnv source env/bin/activate and can use pip as usual
The articles arguments against this and for always using python -m pip don't convince me, personally. I never work on windows, so that's a mute point, and b/c I use direnv to automatically activate my env when cd'ing into the directory, I'm never not in the project environment. So basically, its impossible to not be using the correct pip version i.e. I have never once had this problem and I am constantly working on multiple python projects + code bases in various envs and versions.
You just do around twenty medium involved things manually because they’re not done for you by anyone else or any tool.
And then python “just works”.
Does your text editor know that? Your colleague? How about your CI-system? What about the deployment story? When are things “unexpectedly” going to break because not everyone knew all these informally applied requirements?
Admit it: python is worst in class here. Almost no other popular language or platform forces it’s users into taking these absurd, error-prone steps, just to have basic shit “just working”.
Take a look at Node: npm may be a mess for other reasons, but boooy do you have clean per project dependencies. Look at Cargo for something even better. Clojure has Lein. The new dotnet CLI is slick too.
At this Python just plain sucks. It’s about time you guys start at least admitting it, or else it will never be fixed.
You can have your opinion, but to answer some of your questions, I use vscode and it works amazingly with python, so yes it does know what environment and python version you are using and can show source code, do type checking (if you use the "typing" module, and tons of other nice things that just work out of the box).
As far as the CI-system knowing, there is not much difference in installing a particular version of rust/cargo say in a docker container and having a particular version of python installed in a container, so I don't really know what you are getting at.
Could it be better? Sure, I do some programming in rust and IMHO the best build tool I've ever used is cargo. But compared to some of the other stuff you mentioned which I've used on projects as well (npm, lein specifically) I would say python is on the same level, and I prefer python over them but that's probably just my bad experience with package-lock/caching and long jvm load times speaking.
You're getting downvoted but I wanted to let you know that I emphatically agree with you, but we might be the only two.
Whenever a discussion of this comes up, it seems there are always some "just use xyz" patterns, and usually 3 or 4 xyz or some very new thing someone just released and it always seems contradicts python official documentation. For every python packing question, there are 33 answers. I guess the irony is that there is more than one way to do it.
Use poetry, use prose, use pip, use venv, use vidtualenv, use conda, use --user, pipenv, pipfile, pipy,pipx,anaconda,miniconda.
It really is terrible. Npm gets a lot of flack for having vulnerable packages but do you see how many python fanatics push pip freeze? It allows them to ignore security vulnerabilities. I'm scared of upgrading any dependencies in my python projects because any update is virtually guaranteed to break. I can upgrade npm all day long and rarely encounter an issue. The ecosystem is so radically different.
Because it's not really a dependency manager, just a "dumb" installer and uninstaller. It's for this same reason you can specify an incompatible set of dependencies in requirements.txt and pip won't bat an eye.
What about installing from source ? conda does not support it at all. For some esoteric python packages which use C++ or Cython backends, it is sometimes impossible to install those packages and installing from source becomes the only viable option. pip is quite awesome for this.
This is where conda excels because it uses its own toolchain and therefore doesn’t need to make any assumptions about the system. Someone needs to have packaged it, ideally using conda-forge, but it should then just work.
Pip installing all from source is tricky to get working correctly as soon as you require a newer compiler or dependencies. Providing wheels works well but is a pain to set up correctly with good platform coverage.
That said, it doesn’t need to be one or the other. Conda integrates well with pip so you can install a base layer with conda then pip install your more esoteric dependencies.
Conda was explicitly designed with installation from source, especially for C/C++ extension modules, as a first-class feature. It is bar none the easiest and most effective way to solve that aspect of packaging and environment management. Just look into conda-forge feedstocks for extension modules for a ton of examples.
Conda even adds advanced features, like managing multiple compiler-specific build variants in the same environment, that go way beyond anything other Python packaging tools do for this.
On top of this, you can install pip into conda environments and still use pip to manage packages if you want, or mix and match conda & pip with a pip section in conda’s environment.yml. Conda automatically manages pip installation into the activated conda environment.
I certainly understand why non-technical folks would like conda, but to me, it's 1) extremely slow to restore a complete environment and 2) hard to interoperate with, as it uses it's own compiler and toolchain.
Personally, I've come to prefer homebrew python + pip for development on macOS (even though said python is not optimized), and clearlinux + pip for Linux and production use.
Clearlinux actually gives you a pretty simple way to install a highly optimized python and most all common libs (numpy, scipy, ML stuff, even with GPU and Intel MKL support) - then you just run a pip install -r requirements.txt to get the small, pure python stuff that you want on top that foundation.
Pip is a powerful but low-level tool, which probably shouldn't be user-facing. It doesn't have a full resolver, locking mechanism, or environment management. Likewise setuptools.
Haven't used Python in a long time, but last I remember, pipenv was the tool of choice. Frankly, I'm overwhelmed by all these choices. Does Poetry, pipsi, pipx, pipenv, and pip + venv all accomplish similar goals? Is it just a matter of preference?
Yeah, venv's are a must. And to the article's "forgot to activate the venv" complaint, I highly recommend direnv.
On top of activating the venv when (and only when) you're inside your project directory tree, you can use dotenv for loading all relevant env vars (AWS creds, API keys, etc.,) and add your project's root dir to PYTHONPATH with path_add.
Come to think of it, only there was a PIP_DEFAULT_VIRTUALENV_PROJECTDIR variable which would causes pip to look upwards until it found a setup.py file, and then use a .venv directory in that directory by default...
I usually use pip3 install --user. Then just add ~/.local/bin to PATH. It hasn't caused any issues for me thus far. I also use venv, but only in CI to make sure that other people's environments don't get messed up. I personally prefer using pip3 install --user.
Regardless, let's not turn this into yet another argument about why anyone should/shouldn't upgrade to Python 2. I was just trying to confirm that the parent is not doing anything that requires Python 2.
All packages that matter were already ported, some new packages work only with version 3, many popular packages already dropped support for python 2, a lot will drop in December.
Most (all? actually I don't know any from your list that do) packages that you listed are no longer maintained
> “Let's say I have two versions of Python installed, like Python 3.7 and 3.8 (and this is very common for people thanks to Python coming installed on macOS and Linux, let alone you may have installed Python 3.8 to play with it while having previously installed Python 3.7). Now, if you were to type pip in your terminal, which Python interpreter would it install for?”
Yeah but you just kick the can to which python happens to be the system python or activated python referenced by “python”. It’s exactly the same problem.
If I as a user have to be careful of which underlying Python installation is being referenced, then I want to be responsible for activating the conda environment I need and using “regular” commands like plain “pip” after that. The “python -m” idiom is not important for this use case, as it doesn’t actually solve the problem (ensuring the right referenced Python).
> Yeah but you just kick the can to which python happens to be the system python or activated python referenced by “python”. It’s exactly the same problem.
Exactly right. And on any sane Linux system the pip command should get you a version compatible with the system Python, which means this is a pointless extra step. Unless these instructions are supposed to be for Windows, or MacOS, in which case it should probably say so. (And doesn't everyone on Windows use Conda anyway?)
Once you figure out how venv works it's really simple to set up a python project. It's just another quirk with just another programming language, they all have them. People don't complain about how hard it is to learn to program, but learning how to use some pip commands is too difficult for them?
Sure it is a quirk, however it could really be better.
Anyway, I think the fact that people has more patience with learning a language than tools is that learning a language is fun, while tooling is just boring.
Maybe it's just on my installation, but pip comes with versioned executables. That is, I can call "pip3.7" or "pip3.4". What's the benefits of using the -m approach in this case?
Not sure why this is downvoted. Only the uninformed are using (usually multiple, lol) requirements.txt files + raw `pip` and managing venvs themselves nowadays.
There's a huge rift between hobbyist Python users and professional developers. That's okay, but the former group doesn't always like it when the latter group chimes in...
As someone who is stuck on a very old version of python (2.7) on win32 due to Reasons (vendors, basically), how I wish I could even try to install pip; unfortunately, it will only recommend upgrading to a newer version of python when I attempt. All roads seem to lead to this place, a one-size-fits-all dismissive "lol upgrade."
Packaging is one of my least favorite things about the language I love the most.
Can you at least upgrade to latest version of python 2.7? If you're on at least python 2.7.9, pip should be already installed because it's included by default.
Just use [mini]conda. Everytime someone posts about a python package issue or virtual env issue, it's something that conda already had solved before 2014. And yes, pip is fully supported within a conda environment.
I really can't figure out why conda is not ore popular.
It adds many minutes on complex trees. That adds up fast if you’re doing just 60 builds a day.
Lockfiles are not just convenient, they are all but required for anything serious. You can’t have your dependencies break or change under your feet from one build to the next. Your builds need to be reproducible.
I’ll take all that over whatever conda has to offer any day.
> Lockfiles are not just convenient, they are all but required for anything serious. You can’t have your dependencies break or change under your feet from one build to the next. Your builds need to be reproducible.
I see. So because C, C++ and most other build environments don't have lockfiles, there is no way to do reproducible builds. I better tell the Debian people they've been wasting an awful lot of time on their reproducible build project. /s
Seriously, "lockfiles are required for anything serious"? That's ridiculous. But if you insist, a quick google shows e.g. [0] and [1] provide that.
I’ll leave it to someone else to explain to you how a .gitmodules file or a source tarball URL with a hash is a lockfile equivalent, or how “c, c++” isn’t a package manager, or how Debian do intact use several methods or specifying locked dependencies in a file format (a lockfile, if you will).
Most build environments do have lockfiles. And, just to clarify, that doesn’t have to be a specific dedicated file. It has to be something that can be versioned alongside the code, so each build gets the exact same set of dependencies and updates are explicit commits.
This basic principle is a requirement for anything serious (I.e something with customers). I’m sorry that this statement hit a nerve for you, but it’s true. In fact in some industries it’s a legal requirement.
And no, a dodgy third party plugin that hasn’t been updated in a year isn’t a good solution.
You missed the /s sarcasm tag in my post. You can, in fact specify specific versions for conda without any external plugins and people who are serious about their build being reproducible do.
In fact, they also specify a specific gcc for extensions that need it, because relying on the system gcc is not reproducible. How do you do it in pip/vent/pipenv/poetry?
Specifying first-level versions is not the same as a lockfile and does not ensure you have the same tree of dependencies each time, and is no way to ensure that your builds are reproducible. I'm not sure you're clear on exactly what a lockfile is, but the problem it solves is this:
You depend on `cool-package==1.0`. That depends on `another-package` with a loose specifier, i.e `another-package=LATEST`.
Now when `another-package` is updated you suddenly have a different tree of dependencies, because the package resolution is run again and `another-package=LATEST` is installed. Imagine you're in the middle of rolling back (or rolling out) a deploy to your product. Suddently what you've been testing and working with has changed, and `another-package=LATEST` breaks the deploy due to some changes or bugs.
What's worse is that it's now harder to roll back, as a re-build and a re-deploy will still bring in the broken `another-package=LATEST`!
The solution is to lock the tree of dependencies, including all sub-dependencies. This has the advantage of speeding up installation as resolution doesn't need to happen. So, your package tree is locked to:
I'm not sure why you think conda doesn't support that. You might want to look at [0] specifically the section titled "Building identical conda environments", and at the documentation for "conda list --export --md5" and "conda list --explicit" (and if you don't use pip also "conda list --canonical") which all do exactly that, and which can all be fed back to start a new environment.
Also "conda env export", which adds some more documentation; In fact, all defaults list explicit versions of packages. The only way you can list installed packages without it is "conda env export --from-history", which will reproduce the non-version-annotated list.
So, yes, you don't have a "requirements.txt" and a specific lockfile maintained on disk; instead, conda manages it, and if you want it in source control (and you likely do), you need to make sure your local pre-commit hook runs the export. But everything is indeed reproducible, without any additional package needed.
> apt install gcc:4:4.9.2-2
Ok, so your ci (and everyone) must be root / fakeroot; and you somehow have to manage that outside your requirements.txt and lockfile (e.g. in a pre-commit or post-checkout hook). And you can't have more than one at the same time on the same system (so your ci must have different containers/vms to build two executables with conflicting requirements, even if they are part of the same project). And you can't do it on Windows or Mac at all (whether root or brew user or whatever).
From every possible aspect, Conda is much, much more suited for reproducible and well defined builds. It does it differently than managers using lockfiles; But it definitely does it, and more completely.
p.s. conda also has "conda list --revision" which shows you the entire history of version updates. Very useful for troubleshooting, even if not strictly required.
We do, but docker caching is not terribly easy to implement in a project with lots of concurrent builds. At least not for me.
We build images based on their commit hash, caching off the last commit hash. That works but has issues with merges and the first commit to a branch.
We also do it based on the branch name, but Docker has issues around specifying multiple cache from arguments in the CLI. That causes unnecessary invalidations on branches.
That all leads to more rebuilds than I would like.
The most I had was about 50 dependencies (I guess that’s the metric that matters), not huge but not trivial. It took about 15 secs at most to resolve dependencies, and usually much less; less than a minute a week overall.
Whereas I’ve spent a lot more time getting pip packages to install, update, etc before switching to conda.
Conda never broke anything for me - it did downgrade after upstream broke things. Pip, on the other hand, broke things more than once.
Python itself is not a mess, the packaging part of Python is.
However, Python is also one of the oldest programming languages still in use and one of the first to have the concept of installable dependencies (easy_install).
Now it is kinda difficult to fix without breaking something, however I think we eventually will have a solution, like __pypackage__ https://www.python.org/dev/peps/pep-0582/.
112 comments
[ 4.1 ms ] story [ 186 ms ] threadThis seems less like a reason to use `python -m pip` and more a reason to endorse languages with saner versioning and less need to sandbox twenty different versions from one another.
Go doesn't have package management, it's just list of git repositories and revisions that should be checked out and compiled with your code.
In fact it is encouraged to have as little dependencies as possible.
- great concurrency (https://trio.readthedocs.io)
- bad package management (it just is. though Poetry's making a dent.)
- tooling? I'm not sure what you mean here.
- docs are ok not great not terrible
- types are rolling out rapidly across the ecosystem
- stdlib is too big but PyPI is right there so who cares
- ok to distribute? pyinstaller and shiv are alright.
He means linters, debuggers, refactoring libs, and other such things...
That being said I would like to inquire which languages _you_ could give as an example that work perfectly well with the system-provided compilers when you are using the latest features.
Honestly virtualenv's solve most of these problems. If using a virtualenv seem to low level or a pain, I can't recommend Poetry [0] enough. It uses virtualenv's in the background, and has from my experience a great dependency resolver.
[0] https://poetry.eustace.io/
All of those build tools are pretty complicated. I honestly prefer python and virtual environment. Simple requirements.txt file.
Want to ship all your app and all it's dependencies in a single file, sweet as, use the shade plugin, or maybe the capsule plugin. etc. Etc.
The Python equivalent is, well, take your pick: https://docs.python-guide.org/shipping/freezing/#freezing-yo...
> Simple requirements.txt file
I've never had a JVM dependency break because I upgraded Maven. I have had that exact issue with Python - we're using Apache Airflow, and anyone who installed the latest version of Pip hit an issue - Pip had changed an API that Python dependencies depended on, so Airflow could no longer be installed by that version of Pip, because it was written for the old API. Have fun googling "TypeError: 'module' object is not callable" to enjoy that rabbit hole.
Not to mention that it was impossible to generate a pipfile.lock for Airflow because its dependencies had rules demanding incompatible versions of dependencies (one wanted <= 7.0.0, the other wanted > 7.0.0) - in Maven I can easily exclude one of the two competing dependencies.
Oh, and then there was the time that a Python dependency used by Airflow required me to set an env var to handle a licensing issue. Can't even do that in Maven, which strikes me as a feature.
Also, your simple requirements.txt didn't mention how you manage virtualenvs.Lastly, Gradle is being used quite a bit, but Maven is still pretty dominant. It really depends on if you want a purely declarative build tool like Maven, or one you can drop scripts into like Gradle. I prefer purely declarative for reproducible buils.
Pip explicitly has no API (except for its command line interface), even the Pip namespace implicates this (pip._internal). So if some dependency is using a Pip "API" they're wrong.
Now, I am developing an application that needs to interface with Pip, in this case it may be appropriate to use the internal Pip API if you pin your Pip version in requirements.txt. However a library shouldn't do this ever.
BTW, I don't rate Python dependency management very high: most other languages have better solutions. However it is manageable.
https://github.com/apache/spark/blob/121f9338cefbb1c800fabfe...
At least half of that is figuring out what the classpath should be. And the worst part, it is all exposed to user, too -- when I was setting up Spark, I had to muck with class paths a lot.
So JVM has 3 big problems: (1) No single way to set up dependency path, each project is different; (2) No way to set up classpath in JVM language, has to use some other one (like bash); (3) Somehow, user has to fix it for many semin-advanced programs. No other ecosystem I know of is this insane! Python, node.js, even C++ are all better!
Okay, so, speaking for myself, we generally ship fat jars or capsules these Docker days. I can't speak as to why Apache Spark chose to do it that way, I don't run their project. But if I'm guessing I suspect it's so you can provide your own Hadoop dependencies as needed, as there's a bunch of varying distributions that people use (Cloudera, Hortonworks etc.) and they're trying to play nice with the existing Hadoop ecosystem - probably also to make it possible to extend it with any other jars that you want to use (although spark-shell has a nice feature where you provide additional dependencies on the command line, or within the shell, using Maven artifact coordinates)
For an example of what's involved using an uberjar https://docs.cyclopsgroup.org/jmxterm > java -jar jmxterm-1.0.0-uber.jar
That's it.
BTW, the uberjar equivalent in Python is something like PEX/shiv/zipapps.
looks like a custom one.. and non-relocatable, too -- if I wanted to install "jmxterm" to my home dir, I'd need to manually edit the script.
(and yes, this is totally "I am developer" issue, not just "I am a user" one -- the dev had to write the script, they had to debug it, and dev would be handling bug reports about inability to invoke from command line when installed to other location)
Funny how you had to go searching to find that script, when the installation instructions are simply - 1) download uberjar, 2) run uberjar. Also, you had to go searching to find a script 8 years old, in a project with a commit 20 days ago, to attempt to prove your point.
Surely you must realise you're grasping at this stage.
/s
The articles arguments against this and for always using python -m pip don't convince me, personally. I never work on windows, so that's a mute point, and b/c I use direnv to automatically activate my env when cd'ing into the directory, I'm never not in the project environment. So basically, its impossible to not be using the correct pip version i.e. I have never once had this problem and I am constantly working on multiple python projects + code bases in various envs and versions.
And then python “just works”.
Does your text editor know that? Your colleague? How about your CI-system? What about the deployment story? When are things “unexpectedly” going to break because not everyone knew all these informally applied requirements?
Admit it: python is worst in class here. Almost no other popular language or platform forces it’s users into taking these absurd, error-prone steps, just to have basic shit “just working”.
Take a look at Node: npm may be a mess for other reasons, but boooy do you have clean per project dependencies. Look at Cargo for something even better. Clojure has Lein. The new dotnet CLI is slick too.
At this Python just plain sucks. It’s about time you guys start at least admitting it, or else it will never be fixed.
As far as the CI-system knowing, there is not much difference in installing a particular version of rust/cargo say in a docker container and having a particular version of python installed in a container, so I don't really know what you are getting at.
Could it be better? Sure, I do some programming in rust and IMHO the best build tool I've ever used is cargo. But compared to some of the other stuff you mentioned which I've used on projects as well (npm, lein specifically) I would say python is on the same level, and I prefer python over them but that's probably just my bad experience with package-lock/caching and long jvm load times speaking.
Whenever a discussion of this comes up, it seems there are always some "just use xyz" patterns, and usually 3 or 4 xyz or some very new thing someone just released and it always seems contradicts python official documentation. For every python packing question, there are 33 answers. I guess the irony is that there is more than one way to do it.
Use poetry, use prose, use pip, use venv, use vidtualenv, use conda, use --user, pipenv, pipfile, pipy,pipx,anaconda,miniconda.
It really is terrible. Npm gets a lot of flack for having vulnerable packages but do you see how many python fanatics push pip freeze? It allows them to ignore security vulnerabilities. I'm scared of upgrading any dependencies in my python projects because any update is virtually guaranteed to break. I can upgrade npm all day long and rarely encounter an issue. The ecosystem is so radically different.
Your advice: forget whatever brought you to Python and start from scratch.
Pip installing all from source is tricky to get working correctly as soon as you require a newer compiler or dependencies. Providing wheels works well but is a pain to set up correctly with good platform coverage.
That said, it doesn’t need to be one or the other. Conda integrates well with pip so you can install a base layer with conda then pip install your more esoteric dependencies.
Conda even adds advanced features, like managing multiple compiler-specific build variants in the same environment, that go way beyond anything other Python packaging tools do for this.
On top of this, you can install pip into conda environments and still use pip to manage packages if you want, or mix and match conda & pip with a pip section in conda’s environment.yml. Conda automatically manages pip installation into the activated conda environment.
Personally, I've come to prefer homebrew python + pip for development on macOS (even though said python is not optimized), and clearlinux + pip for Linux and production use.
Clearlinux actually gives you a pretty simple way to install a highly optimized python and most all common libs (numpy, scipy, ML stuff, even with GPU and Intel MKL support) - then you just run a pip install -r requirements.txt to get the small, pure python stuff that you want on top that foundation.
For installing global command-line scripts, I like pipx instead of pip. https://github.com/pipxproject/pipx
For development, I like Poetry instead of pip. https://poetry.eustace.io
Poetry is a replacement for pipenv + setuptools.
Pipsi is unmaintained and replaced by pipx.
On top of activating the venv when (and only when) you're inside your project directory tree, you can use dotenv for loading all relevant env vars (AWS creds, API keys, etc.,) and add your project's root dir to PYTHONPATH with path_add.
Come to think of it, only there was a PIP_DEFAULT_VIRTUALENV_PROJECTDIR variable which would causes pip to look upwards until it found a setup.py file, and then use a .venv directory in that directory by default...
Most (all? actually I don't know any from your list that do) packages that you listed are no longer maintained
Except for PYTHONPATH variable, 2 and 3 worlds are completely separate.
Yeah but you just kick the can to which python happens to be the system python or activated python referenced by “python”. It’s exactly the same problem.
If I as a user have to be careful of which underlying Python installation is being referenced, then I want to be responsible for activating the conda environment I need and using “regular” commands like plain “pip” after that. The “python -m” idiom is not important for this use case, as it doesn’t actually solve the problem (ensuring the right referenced Python).
Exactly right. And on any sane Linux system the pip command should get you a version compatible with the system Python, which means this is a pointless extra step. Unless these instructions are supposed to be for Windows, or MacOS, in which case it should probably say so. (And doesn't everyone on Windows use Conda anyway?)
Anyway, I think the fact that people has more patience with learning a language than tools is that learning a language is fun, while tooling is just boring.
https://github.com/pypa/pipenv
Unfortunately there are people who complain that pipenv makes the ecosystem as bad as npm, which frankly is an improvement.
Packaging is one of my least favorite things about the language I love the most.
I really can't figure out why conda is not ore popular.
The lack of a Conda lockfile makes it impractical to use outside of toy research proof of concepts.
And while lockfiles are convenient, they are far from making anything “impractical”. I take Conrad correctness over minor conveniences any day.
Lockfiles are not just convenient, they are all but required for anything serious. You can’t have your dependencies break or change under your feet from one build to the next. Your builds need to be reproducible.
I’ll take all that over whatever conda has to offer any day.
I see. So because C, C++ and most other build environments don't have lockfiles, there is no way to do reproducible builds. I better tell the Debian people they've been wasting an awful lot of time on their reproducible build project. /s
Seriously, "lockfiles are required for anything serious"? That's ridiculous. But if you insist, a quick google shows e.g. [0] and [1] provide that.
[0] https://github.com/Nextdoor/conda_lockfile
[1] https://picky-conda.readthedocs.io/en/latest/index.html
Most build environments do have lockfiles. And, just to clarify, that doesn’t have to be a specific dedicated file. It has to be something that can be versioned alongside the code, so each build gets the exact same set of dependencies and updates are explicit commits.
This basic principle is a requirement for anything serious (I.e something with customers). I’m sorry that this statement hit a nerve for you, but it’s true. In fact in some industries it’s a legal requirement.
And no, a dodgy third party plugin that hasn’t been updated in a year isn’t a good solution.
In fact, they also specify a specific gcc for extensions that need it, because relying on the system gcc is not reproducible. How do you do it in pip/vent/pipenv/poetry?
You depend on `cool-package==1.0`. That depends on `another-package` with a loose specifier, i.e `another-package=LATEST`.
Now when `another-package` is updated you suddenly have a different tree of dependencies, because the package resolution is run again and `another-package=LATEST` is installed. Imagine you're in the middle of rolling back (or rolling out) a deploy to your product. Suddently what you've been testing and working with has changed, and `another-package=LATEST` breaks the deploy due to some changes or bugs.
What's worse is that it's now harder to roll back, as a re-build and a re-deploy will still bring in the broken `another-package=LATEST`!
The solution is to lock the tree of dependencies, including all sub-dependencies. This has the advantage of speeding up installation as resolution doesn't need to happen. So, your package tree is locked to:
That makes any updates to your packages safe, versioned and able to be reverted.The lack of this feature makes Conda a no-go for anything serious.
> specific gcc for extensions that need it, because relying on the system gcc is not reproducible
apt install gcc:4:4.9.2-2
Also "conda env export", which adds some more documentation; In fact, all defaults list explicit versions of packages. The only way you can list installed packages without it is "conda env export --from-history", which will reproduce the non-version-annotated list.
So, yes, you don't have a "requirements.txt" and a specific lockfile maintained on disk; instead, conda manages it, and if you want it in source control (and you likely do), you need to make sure your local pre-commit hook runs the export. But everything is indeed reproducible, without any additional package needed.
> apt install gcc:4:4.9.2-2
Ok, so your ci (and everyone) must be root / fakeroot; and you somehow have to manage that outside your requirements.txt and lockfile (e.g. in a pre-commit or post-checkout hook). And you can't have more than one at the same time on the same system (so your ci must have different containers/vms to build two executables with conflicting requirements, even if they are part of the same project). And you can't do it on Windows or Mac at all (whether root or brew user or whatever).
From every possible aspect, Conda is much, much more suited for reproducible and well defined builds. It does it differently than managers using lockfiles; But it definitely does it, and more completely.
p.s. conda also has "conda list --revision" which shows you the entire history of version updates. Very useful for troubleshooting, even if not strictly required.
[0] https://docs.conda.io/projects/conda/en/latest/user-guide/ta...
That's a very inefficient way to run your CI, with conda and pip alike.
Instead, you could build your environment once in a Docker image and use that as your build image.
Saves a lot of time on your builds, guarantees reproducibility, and will work even when package servers are unavailable.
We build images based on their commit hash, caching off the last commit hash. That works but has issues with merges and the first commit to a branch.
We also do it based on the branch name, but Docker has issues around specifying multiple cache from arguments in the CLI. That causes unnecessary invalidations on branches.
That all leads to more rebuilds than I would like.
Whereas I’ve spent a lot more time getting pip packages to install, update, etc before switching to conda.
Conda never broke anything for me - it did downgrade after upstream broke things. Pip, on the other hand, broke things more than once.
However, Python is also one of the oldest programming languages still in use and one of the first to have the concept of installable dependencies (easy_install).
Now it is kinda difficult to fix without breaking something, however I think we eventually will have a solution, like __pypackage__ https://www.python.org/dev/peps/pep-0582/.