All jokes are only funny when they are resembling to the truth but they are not when it's far from it. So this maybe funny five years ago. But it doesn't any more since Docker[1].
docker pull; docker run, you are golden. Even if you are going to customize the official image, it's much less cumbersome than that xkcd cartoon.
What version of Python you'd be running? And Windows Server 2012 is 5 years old, no? If you are running a five years old OS, perhaps unsuported Docker version 1.7 (released 2015) might not be your only and most import concern right?
Nah, but I was helping the author clean up his Python environment on Friday, which I think prompted this comic. His environment isn't even the messiest I've seen. I helped an astronomer friend with theirs at one point, and it had something like six different "manage your whole Python workflow easily with just this tool!" things installed.
I think what a lot of comments here miss is that while new tools come out that can help make things better, they don't suddenly fix people's existing installs. It seems weird that there isn't more documentation out there to help people get out of these kinds of messes.
Exactly right. There is an irreversible entropy to this: once a certain number of implementations exist, it becomes almost impossible to create a "new standard" that fixes them all, because it has to be backwards compatible with too many of them.
As someone responsible for probably one of your astronomer friend's tools (Anaconda), I have to say that we can only do so much: the nature of Python itself is that we cannot ignore what the user puts into PYTHONPATH, PATH, etc. On Windows, at least, we do create shortcuts in the Start Menu to ensure that we get a well-defined command shell. But if the user forces things like PYTHONPATH, there's not much we can do to un-fsck the setup.
Maybe it would be useful to have a python-safemode (or python-portable) binary that explicitly ignores all environment variables, configuration files, registry settings, dotfiles, etc. etc. and only looks at files that are next to it in the path....
We've been discussing this, R users have asked for the same. The argument is that Anaconda should give them isolation. I think it should be opt-in though, probably. Tricky thing is that to do it you need to muck about in the internals of CPython and R and that'll upset some other people.
Yea macos is kinda a clusterfuck when it comes to dealing with Linux packages. Now you're fighting between system, brew and pip and none of it is fun. At least in Linux you're pretty much dealing with system python packages and user python packages.
- `brew install python3` (which includes pip) and any system dependencies you need: C libraries, etc. For example, `brew install openssl libpq libffi libsass libsodium ossp-uuid` and so on
- Create a virtualenv and install all python stuff in there. Doing `virtualenv -p python3 venv` in the root of your project works. But, if you want to get fancy, there's pyenv, pyvenv, and the like. Use one that works for you.
This seems to be the same as the experience with Linux, except you don't need to install the package manager. In Docker, you also don't need a virtualenv.
Or maybe people are using more difficult to install dependencies than I am? Even if you have to install from source though, it's not a huge deal.
Another possibility is people are using IDE's that want you to do things "their way". I don't know anything about this b/c I use a terminal-based text editor (not trying to start a flamewar).
And the final thought is just that people are not experienced with package managers or general terminal use. You obviously have to be fluent in terminal to use the workflow I outlined above.
I use Docker and like it over virutalenvs, but even without it, if you're just playing around/sandboxing, pip3 --user is your friend. Don't use pip for system packages ever! Let your distros package manager handle that.
Docker doesn't solve the problem, and virtualenv is irrelevant to the discussion. It's just another option -- all other options already exist, will continue to exist, and can interact with your Python setup in subtle ways (not to mention one might need to distribute their python code).
I think the comic should be more aptly be titled "Life before pipenv".
Pipenv is the first (popular) sane tool to make Python accessible to programmers coming from other languages. Sure you can also just use bare virtualenv, but then you have to figure out how it works together with everything displayed in the picture.
I think the intent here is to convince adherents of other tools to switch to pipenv. Having used most of the solutions in this thread, pipenv has been the simplest and the one that "just works" most often.
> Sure you can also just use bare virtualenv, but then you have to figure out how it works together with everything displayed in the picture.
How so? I use virtualenv and don't have any issues. Seems pretty straight forward to me. I haven't moved to pipenv because it forces a folder structure on you, which I hate. The great thing about virtualenv is that you can be anywhere on your system, it doesn't matter, as long as you see the project name in paren's next to your prompt you know you are good.
I don't know the specific problems anymore, but I've set up virtualenv on ~5 separate occasions for my machines. Every time it was a long painful session of trial end error, that ended with a setup that was mostly stable (but also sometimes fell apart when I updated the system/homebrew ?) and also seemed to never be completely shielded off from othe dependencies I had installed globally. The workflow with Pipenv was the first stable, no hassle solution I experienced with Python.
> The great thing about virtualenv is that you can be anywhere on your system
That sounds like an anti-pattern to me. Each project (= directory) should bring its own environment with it and should be completely decoupled from the environment of other projects.
Well, my experience has been drastically different than yours. Virutalenv has been flawless for me.
>That sounds like an anti-pattern to me. Each project (= directory) should bring its own environment with it and should be completely decoupled from the environment of other projects.
Yes they should be decoupled, but why would I want in the same folder? If am building a Django app I don't want the entire instance of python inside my project folder for a few reasons. 1) It clutters it up. 2) The web service (user/group) needs to access the Django folder, why would I put python in a folder that the web service has control over? I much rather have my web application in /svr/www/webapp and my python virtualenv in /opt/python-virtual/webappname
Regarding '1)': It doesn't have to bin _in_ the directory, and AFAIK it isn't with pipenv.
Regarding '2)': That sounds like a deployment issue? Certainly nothing I've experienced on my development machine. With pipenv your still free to generate a requirements.txt and deploy your app with whatever webserver setup you want.
>With pipenv your still free to generate a requirements.txt and deploy your app with whatever webserver setup you want
yeah but then I can't be in the Django web application folder and just type "python manage.py makemigrations", for example. I would have to be the pipenv folder then "python /srv/www/webapp/manage.py makemigrations".
Unless I am missing something, which could be the case.
You would be in the Django web application folder (which has a Pipfile) and run "pipenv run python manage.py makemigrations" or run "pipenv shell" and then "python manage.py makemigrations". There is no "pipenv folder" you need to think about. There is a virtualenv folder somewhere on the system, but activation of that etc. is all managed by the pipenv command and tied to a project.
All of which work fine, the real problem is installing Python with homebrew. If you just install Python from the binary on the website then any of those solutions will work with minimal drama.
Homebrew pulls it directly from the Python.org, but it's also slightly opinionated in what else it installs. I would suggest just reading through the formula to see what it does (it's straighforward).
Likewise, you can quickly see what any Homebrew package is doing with `brew info <package>` and looking at the "From:" link.
It's almost - but not quite - this bad when using a "simple" local development environment. As others have said it's not nearly this bad when you're deploying to sane infrastructure, be it containerized or not.
I've never understood the purpose of virtualenvwrapper. I only use three commands for administering virtual environments: "virtualenv -p python3 ~/envs/foo" to make one, ". ~/envs/foo/activate" to activate it, and "rm -r ~/envs/foo" to delete it. That is really not something that needs further simplification. What am I missing?
Virtualenvwrapper (I hate the name, and as a fish user I actually use virtualfish instead) has as the main advantage in keeping all virtualenvs in one place (by default ~/.virtualenvs), so you don't have to worry about the actual location of your virtualenv. This allows for a number of useful features like activating, listing and deleting available environments with autocomplete, and creating a virtualenv is also quite simplified.
But yes, if you're making sure that you're always careful about passing the location of your environments, you don't really need it.
It always felt like virtualenvwrapper was moving things in the wrong direction. Why should the virtualenv be located in the home directory, instead of in the project directory? The whole point of virtualenv is so that you don't have environment dependencies, and each project can have its own environment.
Treating the environment files as a build artifact does make more sense than having a directory of environments somewhere else.
I'm not sure there are any practical advantages to doing it that way, though. I name my environments after the project, so it's more like the environment's folder is a secondary project folder than part of the environment.
I'm not sure how the location of the environment affects dependencies; the environment is isolated regardless of where it is located. Also, the virtualenv location doesn't have to be in the home directory; it can be anywhere your user has access to, the home is simply most convenient.
Most definitely in automated setups, i.e. in production. (I personally never depend on virtualenvs in production, preferring to have a fully isolated environment either via Docker or even better a dedicated VM.)
But in development, where you manually switch between environments, a centralised setup is great. You don't have to worry about gitignoring the virtualenv directory, or maintaining paths in general -- a common problem with virtualenv in your code directory is that IDEs and linters and similar tools tend to just cut through and parse everything, unless explicitly prevented. With virtualfish/virtualenvwrapper, the process is simply `workon {envname}` and you have everything in place.
Legacy setups from previous decades, laziness, automated installers written by others (so they play by different rules), semi-broken packages with messed-up dependency graphs that require manual treatment. The comic is, of course, exaggerating - or at least I hope no one have things gone that bad.
> every sane programmer uses ... pip in a virtualenvwrapper environment
Check out Pipenv <https://docs.pipenv.org/> - you may like it. It aims to make things saner that bare pip+virtualenv{,wrapper}, and IMHO it really does.
There was a time where I would have strongly related to this. Thankfully, now I've started using pipenv whenever possible and it basically just works(TM).
It combines many features that people manage separately with virtualenv, pip, and any custom scripts. I think it reduces boilerplate and simplifies env management.
It is still under active development, but I think it is stable enough to use for production. I think it's probably better for people learning Python to use too.
pipenv doesn't solve any problems well, especially if you require multiple versions of Python, which is _always_ a problem if you're developing libraries. You still need something like pyenv to manage multiple versions of Python.
If you're using pyenv, it's easy to do virtualenv with the pyenv-virtualenv plugin, at which point you don't need Pipenv because it doesn't solve anything (except locking, see blow) that you can't do with requirements files. Furthermore, if you're testing under multiple versions of Python, you'd use tox to manage your test environments anyway.
_Further_ furthermore, they authors recommend _against_ use pipenv to manage dependencies for anything but development, so it's on you to have a copy of runtime dependencies isolated from Pipfile. And since you have to put your runtime requirements in setup.py anyway, there was never any way around the issue of duplicating requirements to begin with.
The one thing Pipenv does that pip and requirements files can't is lock dependencies down the entire tree... except it's abominably slow at that because it has to download dependencies and walk the entire tree. There's a debate ongoing about using pip's cache to solve the problem but there's no end in sight.
If you're really sensitive to having the entire dependency tree frozen to exact versions of you requirements, just `pip freeze` them occasionally. It's way less hassle.
Packaging in Python is far better than it was a few years ago. There's still a lot to improve but pipenv shouldn't be part of the solution.
> _Further_ furthermore, they authors recommend _against_ use pipenv to manage dependencies for anything but development, so it's on you to have a copy of runtime dependencies isolated from Pipfile.
I think the use-case is slightly different than that.
Pipenv is a tool that enables reproducible builds for python app development and deployment. For example, many web apps, aren't going to be packaged up using setuptools, they are just going to be deployed somewhere (possibly containerized first). And with pipenv you can avoid the need to vendor all your dependancies to ensure reproducible builds.
But its not a tool that solves problems for library packaging and distribution (on say pypi). You still have to use setup.py for that, and shouldn't be pinning your dependancies on exact package versions. But you can still use pipenv to manage your dev environment in those cases, and for reproducible dev environments.
It's funny because within the comment section there are already 2 different opinions on why this is not funny because $xyz is what every sane python dev should use.
Maybe this is obvious, but to informally install executables supplied by python packages, use a separate virtualenv for each one and place a symlink to the venv/bin/theexecutable in a location like ~/bin that you have on your $PATH.
Python environment headaches have been largely solved for me thanks to Miniconda[1]. Not only do the environments isolate dependencies, they can easily use different versions of Python, and can include arbitrary binary packages too. It helps that for my work in biology an extensive number of packages are available from the bioconda[2] channel (with many non-bio packages from conda-forge[3]). Environments can be described via environment files, allowing them to be transferred to collaborators, managed with source control, or included with publications to support reproducible science. If you like virtualenvs, you may want to give Miniconda a try. One current limitation is that environment files cannot specify a source channel for specific packages--they're installed from channels based on the global channel precedence.
I used Conda recently and it was OK, but it's annoying that it can't install stuff directly from PyPI. Unless you're lucky, you still have to use Pip inside the Conda environment to install some packages.
But it's annoying how many packages like qiime make it basically a requirement to use Anaconda/miniconda. I really don't want to pollute my computer with multiple installations of even things like R just to run a package.
What if the right answer to the packaging nightmare that plagues languages with native extensions, is to actually have multiple isolated copies of the interpreter, each with well-known build chain metadata and sane, isolated locations for shared libraries?
* The package formats (source or wheel) do not sufficiently capture assumptions that devs make at release time, in order to produce a usable installable artifact at install time.
* Worse, when different package builders release binaries, there is insufficient information in the binaries metadata in order to know which sets of binaries will actually install AND run properly on any given system.
* Finally, the Python packaging ecosystem suffers from a "Tesla Autopilot" problem: some kinds of things are actually WORSE if they only work 90% of the time. So, many people get by just fine with pip and virtualenv... until they don't. Since package building and dependency-solving are not exactly the sexiest or most fun part of software development, most devs in the modern era don't necessarily take the time to understand the actual roots of the problem, but instead bat around lore and cut-and-paste stackoverflow until things seem to kind of work.
A major part of Python's current success is due to its numerical and data analysis libraries. These, in turn, are successful because they take advantage of deep capabilities in C, C++, and other "native" code. This means that Python packaging inherits the original sins of C (the dynamic linker) and of C++ (no standard binary ABI). Most other languages do not have to solve such a hard problem: Perl, Ruby, Node, etc. all don't go nearly as deep as Python does in terms of leveraging a rich ecosystem of native code extension modules.
Even Java avoids this and lives almost entirely within the JVM runtime - but even then, classpath conflicts show that dependency management in ANY language is a hard problem unless treated holistically and intentionally. Python got its package system bolted-on after the fact. Then several tools came and went as maintainers entered and exited the ecosystem. With Anaconda and conda, we are just now finally at a point where people can reliably install the basic scientific and numerical libraries, across hardware and OSes..... 20 years after I started using the language. :-)
Image alt text is "The Python environmental protection agency wants to seal it in a cement chamber, with pictoral messages to future civilizations warning them about the danger of using sudo to install random Python packages."
"This place is not a place of honor" is the first line of the most basic signage created by a Sandia National Lab project aiming to keep future explorers/archaeologists/looters/settlers unharmed by a nuclear waste dump.
The problem space is very difficult - it'll be something like AD 12000 before the waste is effectively harmless. That's further in the future than all of civilization[1] is behind us. The earliest surviving language specimens are proto-heiroglyphics that are less than 5000 years old.
It's difficult to craft a message that won't be taken like all the myriad curse-promising looter-deterrents guarding e.g. plundered tombs throughout the world. It's especially unconvincing when the hazards we're trying to protect future humans from include things like a hundredfold increased risk of cancer.....after inhabiting the site for twenty years.
It's further difficult to attempt to ensure that the message survives that massive amount of time. The designers came up with a linked system where elements are supposed to reinforce and index each other, with redundancy of information and presentation at multiple levels of complexity, with written and pictographic forms, attempting to avoid overstatement, attempting to ensure that protective structures cannot be usefully scrapped and re-used for new construction, ensure that the communications resist deliberate vandalization, etc.
When I first got mac, I've spent the whole evening trying to install python3+pipenv and get it working properly. Ended up nuking system's python 2.7 and installing it through homebrew.
Homebrew installing Python and installing virtualenv and virtualenvwrapper makes life easy.
All of these methods people are listing that requiring me to do more than manage one file (requirements.txt) is sort of missing the point.
I get that a lot of Node developers are coming to Python to do things, and that's great. But the environments don't need the exact same tools. pipenv seems to fix a problem I never encounter with pip/virtualenv during daily development work.
Man it's frustrating to see this comic and read the comments here, because it's all so true. Python, in its noble quest for backwards compatibility, has accumulated so many different ways of packaging, distributing and installing libraries and apps that it's rivaling Google's chat apps.
Pipenv is the most promising solution today but is still very, very new. It's modeled after yarn and has been officially blessed as "The One True Way" of installing stuff by the Python documentation. It has a way to go still to be as good as yarn (especially in terms of speed). The Python ecosystem has never had proper declarative packages like package.json (setup.cfg can be used to have fully declarative package metadata, but I seem to be the only one using it that way), which is a problem for package managers.
To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. It avoids the need for virtualenv, kind of, but so does pipenv and it does so more reliably and reproducibly (pipenv implements lockfiles).
> To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point.
Package manager/dependency manager is must for any sort of mature mass production coding environment (e.g. cargo for Rust, Maven for Java etc.) Docker doesn't solve all that but at least you are doing it only once and replicate easily and predictably. And a lot of times you can reuse Docker images built by someone else.
If the "built by someone else" docker container contains a bunch of incomprehensible hacks or black magic, then you're really just handing yourself a nightmare down the road when it comes time to update that thing or apply security fixes.
Pipenv is mentioned in the PYPA Python Packaging User Guide tool recommendations:
> Application dependency management
> Use Pipenv to manage library dependencies when developing Python applications. See Managing Application Dependencies for more details on using pipenv.
You're quite right. I skimmed Installing Packages, which mentions it in passing is a tool for managing multiple venvs, and Packaging & Distributing Projects, which doesn't mention it at all.
Was thinking the same thing. When python was getting started it was up against Perl, which toon the opposite approach and there was a myriad of ways to do something in the language.
Package managers for languages were just starting to become a thing back then. (At least I think that is true...). Now days you wouldn’t dream of releasing a new language without some kind of standard package manager...
Dunno where I’m going with this comment but it is definitely interesting to see how much has changed since then. Hell even Microsoft has a package manager for it’s languages...
My feeling is that a good package manager is the exception to the rule. Particularly around managing versions of the language runtime and the versions of packages available within projects using that runtime.
Rust is the exception here in that Cargo is a very nice piece of kit.
In terms of generic runtime version managers, ASDF is very nice cross language version manager for scoping versions to projects and switching between them easily.
> Rust is the exception here in that Cargo is a very nice piece of kit
Also Rustup, which is pretty key to Cargo remaining flexible going forward. There's a huge advantage to having the build/dependency management piece being highly coupled with the language runtime/compiler. Making cargo work with all versions of Rust would mean having to build in backwards compability and make it much more difficult to do projects like the new modules changes.
It's a key insight that there needs to be two tools surrounding a language, not just one. You need good dependency management and good version management. Node also has this right with nvm and npm/yarn.
Just consider urllib, urllib2, urllib3, and requests. Or getopt, optparse, argparse, and docopt. Or countless others.
I appreciate that the Python community continually strives for ever-more obvious and ergonomic ways of accomplishing tasks. While there is an ecosystem complexity cost to the solutions-that-weren't, it's no greater than the churn than I've seen in other popular languages.
The Zen of Python even pokes fun at the "one obvious way" by writing its three em-dashes three different ways in the same document.
Pipenv solves a very real problem, but it has a lot of problems of its own:
- It's very slow: re-locking after updating 1 dependency often takes me ~1 minute.
- It has lots of bugs. To name a few in 11.6.9: clobbers comments in the Pipfile, inconsistently includes dependencies for other build environments in Pipfile.lock, stores the wrong index in Pipfile.lock for packages not on PyPI.
- They release multiple times per day, often breaking things in patch releases.
- Kenneth Reitz is quite unpleasant to deal with in GitHub issues, which I often have to because of the previous 2 issues.
From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
I apologize for not meeting perfect standards of mood at all times. I have bipolar disorder, for what it's worth, and working on a packaging tool is very stress-inducing.
> From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
Sharing your awesome Python program is its Achilles heel. I have 5 computers I work on a weekly basis. I stopped using Python and Haskell due to all of this.
Yes, packaging Python applications for distribution is yet another nightmare situation.
Back in the day, we used one of the various "freeze" applications (cxfreeze, bbfreeze, probably others) and some tools developed in-house to package and distribute them for various platforms.
It has been a while since I actually needed to do this, so I don't know what the current state of the art is.
I've thought about starting a "build an ecosystem for Racket" project where I document gaps in the extended ecosystem and review best-of-breed solutions from other languages as a kind of literature review / review of prior art.
Racket has such a great story for so many things that are painful in other environments. It's an under-appreciated language.
Would you be willing to collaborate on this? Racket was my first programming language and I miss it dearly. An excuse to write code in it would be great!
The only issues that this really could address. Would be nice to have a "Wall of Shame" for these gaps.
A) Racket is so enjoyable everyone builds their own solutions
B) "The view that full-fledged problem solving almost always calls for language design" Which sadly falls onto deaf ears especially people from Python.
> Python, in its noble quest for backwards compatibility, has accumulated so many different ways of packaging, distributing and installing libraries and apps that it's rivaling Google's chat apps.
I'm not sure it's because of backwards compatibility. Perl has amazing backwards compatibility, but doesn't seem to suffer from this.
That's not said to pump up Perl or degrade Python, but because if you misidentify the problem, your proposes solution has a much lower likelihood of fixing it.
(Then again, maybe you mean something different by "backwards compatibility" than what I thought you meant)
Please forgive me for venting my frustration here, but my reaction to this is basically: oh great, yet another blinking tool to add to the list of misery that is the Python packaging experience. May as well check it out now before I have to do it in a hurry...
First of all it's not in Debian unstable. Ok, maybe it's just super new... so I'll install it with pip. A python3 -m pip install pipenv later and let's try it out!
$ cd ~/src/nexsan-exporter/nexsan-exporter
$ pipenv install
Creating a virtualenv for this project…
Using /usr/bin/python3.6m (3.6.5) to create virtualenv…
⠋Running virtualenv with interpreter /usr/bin/python3.6m
Using base prefix '/usr'
New python executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m
Also creating executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python
Installing setuptools, pip, wheel...done.
Virtualenv location: /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG
Installing dependencies from Pipfile.lock (ca72e7)…
0/0 — 00:00:00
To activate this project's virtualenv, run the following:
$ pipenv shell
Cute emoji and nice colours but... ~/.local/share? For a directory that will end up containing arch-specific libraries? Ok, I guess no one gives a shit about this in the modern world, oh well. Wait... "python3.6m"? That isn't the Python interpreter I asked for... but it seems to be a hardlink to the same file as /usr/bin/python3.6 so I guess maybe this is intentional? Anyway let's check out the venv...
So this is the Python folk's what, fourteenth attempt to get this right, and they are still copying the python executable into the virtual environment instead of symlinking it in? This seems to be a regression from venv, which seemed to get this right!
Right, time to install my dependencies... according to 'pipenv' the right command for this is 'pipenv install -e .' Hm, I wonder exactly what the -e option does?
$ pipenv install --help
* no menition of -e in the output *
:sadface:
Oh well, let's just run it blind!
$ pipenv install -e .
Installing -e .…
⠏
Error: An error occurred while installing -e .!
Directory '.' is not installable. File 'setup.py' not found.
Maybe I screwed up and ran this from the wrong directory?
$ ls setup.py
setup.py
Weird, what's going on here?
$ strace -f pipenv install -e . 2>&1 | grep setup\\.py
stat("/home/yrro/src/nexsan-exporter/nexsan-exporter/setup.py", {st_mode=S_IFREG|0644, st_size=1769, ...}) = 0
stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0200) = -1 ENOENT (No such file or directory)
[pid 7365] stat("./setup.py", 0x7fff84176e30) = -1 ENOENT (No such file or directory)
stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0300) = -1 ENOENT (No such file or directory)
write(2, "Directory '.' is not installable"..., 62Directory '.' is not installable. File 'setup.py' not found.
Oh FFS, I give up. I think I'll let pipenv pass me by f...
I wouldn't go that far. I would absolutely love for some tool, and pipenv as as fine as any other, to replace everything else. But right now it doesn't seem to be ready.
Well I filed these as bugs this morning and got mixed feedback; the 'install -e .' but was caused by a stale Pipefile in the parent directory that I did not create, but at least removing it makes things work. Unfortunately upstream does not consider a stale copy of the python interpreter copied to dozens of directories on the filesystem that will never be patched to be an issue, so from my PoV pipenv gets a big raspberry and I'll stick with manually running 'python -m venv' which has gotten this right since Python 3.3!
As a Python dev it looks like I need to do some reading as I am still using virtualenv (it usually works fine but I have do encounter the occasional issue).
Can anyone give me a rundown of the benefits of other systems over virtualenv?
On a side note I hope Python isn't going to turn into a shambles like JavaScript - its certainly starting to look like it as far as package managers go.
JS isn't in shambles; in fact, their package manager situation is excellent: Use yarn, period. It could be better (yarn isn't default yet), but I would love for the Python ecosystem to get to: "Use pipenv and setup.cfg, period".
But as I said, it's too young. If you use pipenv, you still have to interact with pip, so that's two binaries whose commands you have to remember. And urgh, it's so slow. And it's kinda useless for libraries... pipenv doesn't publish stuff, pipfiles aren't appropriate for packages.
But we'll get there. One day. Hopefully it won't be too late.
I've not given you anything other than a warning that it's too young.
Pipenv is promising. That means at some point I'm fairly certain it'll be and deserve to be the standard in Python packaging. In the mean time, it doesn't.
I just started using pipenv for a small personal project, and I like it. The usage is similar to git which I prefer over the weird bash activate mangling that a normal virtualenv does.
There was Zope Buildout for years, that had declarative packaging format, used the cheese shop (biggest issue with conda) and isn't slow as molasses but it was and still is largely ignored by the larger community.
Sorry, idk. I just went that route and then was like well, maybe this time I should simply return and rethink the choice of a language. Didn’t do much credit to an engineer, but honestly, I didn’t feel myself a mess cleaner that day.
Sure, it caused a lot of grief for Debian packagers (because the name "site-packages" was hardcoded everywhere) when it was introduced back in the Python 2.6 era; but I don't think I've ever had to worry about it as a user.
185 comments
[ 5.5 ms ] story [ 229 ms ] threaddocker pull; docker run, you are golden. Even if you are going to customize the official image, it's much less cumbersome than that xkcd cartoon.
[1]: https://hub.docker.com/_/python/
https://www.liquidweb.com/kb/how-to-install-docker-on-centos...
I think what a lot of comments here miss is that while new tools come out that can help make things better, they don't suddenly fix people's existing installs. It seems weird that there isn't more documentation out there to help people get out of these kinds of messes.
As someone responsible for probably one of your astronomer friend's tools (Anaconda), I have to say that we can only do so much: the nature of Python itself is that we cannot ignore what the user puts into PYTHONPATH, PATH, etc. On Windows, at least, we do create shortcuts in the Start Menu to ensure that we get a well-defined command shell. But if the user forces things like PYTHONPATH, there's not much we can do to un-fsck the setup.
Maybe it would be useful to have a python-safemode (or python-portable) binary that explicitly ignores all environment variables, configuration files, registry settings, dotfiles, etc. etc. and only looks at files that are next to it in the path....
- Install brew w/ the one-liner on https://brew.sh
- `brew install python3` (which includes pip) and any system dependencies you need: C libraries, etc. For example, `brew install openssl libpq libffi libsass libsodium ossp-uuid` and so on
- Create a virtualenv and install all python stuff in there. Doing `virtualenv -p python3 venv` in the root of your project works. But, if you want to get fancy, there's pyenv, pyvenv, and the like. Use one that works for you.
This seems to be the same as the experience with Linux, except you don't need to install the package manager. In Docker, you also don't need a virtualenv.
Or maybe people are using more difficult to install dependencies than I am? Even if you have to install from source though, it's not a huge deal.
Another possibility is people are using IDE's that want you to do things "their way". I don't know anything about this b/c I use a terminal-based text editor (not trying to start a flamewar).
And the final thought is just that people are not experienced with package managers or general terminal use. You obviously have to be fluent in terminal to use the workflow I outlined above.
https://packaging.python.org/guides/tool-recommendations/#ap...
Pipenv is the first (popular) sane tool to make Python accessible to programmers coming from other languages. Sure you can also just use bare virtualenv, but then you have to figure out how it works together with everything displayed in the picture.
No amount of adding things to the Python comic is going to simplify it.
> No amount of adding things to the Python comic is going to simplify it.
It doesn't simplify it, but it puts you in a whole new canvas where you don't have to worry about whats in the messed up one.
How so? I use virtualenv and don't have any issues. Seems pretty straight forward to me. I haven't moved to pipenv because it forces a folder structure on you, which I hate. The great thing about virtualenv is that you can be anywhere on your system, it doesn't matter, as long as you see the project name in paren's next to your prompt you know you are good.
> The great thing about virtualenv is that you can be anywhere on your system
That sounds like an anti-pattern to me. Each project (= directory) should bring its own environment with it and should be completely decoupled from the environment of other projects.
>That sounds like an anti-pattern to me. Each project (= directory) should bring its own environment with it and should be completely decoupled from the environment of other projects.
Yes they should be decoupled, but why would I want in the same folder? If am building a Django app I don't want the entire instance of python inside my project folder for a few reasons. 1) It clutters it up. 2) The web service (user/group) needs to access the Django folder, why would I put python in a folder that the web service has control over? I much rather have my web application in /svr/www/webapp and my python virtualenv in /opt/python-virtual/webappname
yeah but then I can't be in the Django web application folder and just type "python manage.py makemigrations", for example. I would have to be the pipenv folder then "python /srv/www/webapp/manage.py makemigrations".
Unless I am missing something, which could be the case.
Homebrew pulls it directly from the Python.org, but it's also slightly opinionated in what else it installs. I would suggest just reading through the formula to see what it does (it's straighforward).
Likewise, you can quickly see what any Homebrew package is doing with `brew info <package>` and looking at the "From:" link.
But yes, if you're making sure that you're always careful about passing the location of your environments, you don't really need it.
Edit: It looks like pipenv, which hopefully will replace virtualenvwrapper and the like, supports storing the environment in the project folder: https://docs.pipenv.org/advanced/#changing-where-pipenv-stor...
But in development, where you manually switch between environments, a centralised setup is great. You don't have to worry about gitignoring the virtualenv directory, or maintaining paths in general -- a common problem with virtualenv in your code directory is that IDEs and linters and similar tools tend to just cut through and parse everything, unless explicitly prevented. With virtualfish/virtualenvwrapper, the process is simply `workon {envname}` and you have everything in place.
Legacy setups from previous decades, laziness, automated installers written by others (so they play by different rules), semi-broken packages with messed-up dependency graphs that require manual treatment. The comic is, of course, exaggerating - or at least I hope no one have things gone that bad.
> every sane programmer uses ... pip in a virtualenvwrapper environment
Check out Pipenv <https://docs.pipenv.org/> - you may like it. It aims to make things saner that bare pip+virtualenv{,wrapper}, and IMHO it really does.
Dongles for days.
I hope it becomes the standard going forward.
https://github.com/pypa/pipenv
It combines many features that people manage separately with virtualenv, pip, and any custom scripts. I think it reduces boilerplate and simplifies env management.
It is still under active development, but I think it is stable enough to use for production. I think it's probably better for people learning Python to use too.
If you're using pyenv, it's easy to do virtualenv with the pyenv-virtualenv plugin, at which point you don't need Pipenv because it doesn't solve anything (except locking, see blow) that you can't do with requirements files. Furthermore, if you're testing under multiple versions of Python, you'd use tox to manage your test environments anyway.
_Further_ furthermore, they authors recommend _against_ use pipenv to manage dependencies for anything but development, so it's on you to have a copy of runtime dependencies isolated from Pipfile. And since you have to put your runtime requirements in setup.py anyway, there was never any way around the issue of duplicating requirements to begin with.
The one thing Pipenv does that pip and requirements files can't is lock dependencies down the entire tree... except it's abominably slow at that because it has to download dependencies and walk the entire tree. There's a debate ongoing about using pip's cache to solve the problem but there's no end in sight.
If you're really sensitive to having the entire dependency tree frozen to exact versions of you requirements, just `pip freeze` them occasionally. It's way less hassle.
Packaging in Python is far better than it was a few years ago. There's still a lot to improve but pipenv shouldn't be part of the solution.
I think the use-case is slightly different than that.
Pipenv is a tool that enables reproducible builds for python app development and deployment. For example, many web apps, aren't going to be packaged up using setuptools, they are just going to be deployed somewhere (possibly containerized first). And with pipenv you can avoid the need to vendor all your dependancies to ensure reproducible builds.
But its not a tool that solves problems for library packaging and distribution (on say pypi). You still have to use setup.py for that, and shouldn't be pinning your dependancies on exact package versions. But you can still use pipenv to manage your dev environment in those cases, and for reproducible dev environments.
1. https://conda.io/miniconda.html
2. https://bioconda.github.io
3. https://conda-forge.org
* Worse, when different package builders release binaries, there is insufficient information in the binaries metadata in order to know which sets of binaries will actually install AND run properly on any given system.
* Finally, the Python packaging ecosystem suffers from a "Tesla Autopilot" problem: some kinds of things are actually WORSE if they only work 90% of the time. So, many people get by just fine with pip and virtualenv... until they don't. Since package building and dependency-solving are not exactly the sexiest or most fun part of software development, most devs in the modern era don't necessarily take the time to understand the actual roots of the problem, but instead bat around lore and cut-and-paste stackoverflow until things seem to kind of work.
A major part of Python's current success is due to its numerical and data analysis libraries. These, in turn, are successful because they take advantage of deep capabilities in C, C++, and other "native" code. This means that Python packaging inherits the original sins of C (the dynamic linker) and of C++ (no standard binary ABI). Most other languages do not have to solve such a hard problem: Perl, Ruby, Node, etc. all don't go nearly as deep as Python does in terms of leveraging a rich ecosystem of native code extension modules.
Even Java avoids this and lives almost entirely within the JVM runtime - but even then, classpath conflicts show that dependency management in ANY language is a hard problem unless treated holistically and intentionally. Python got its package system bolted-on after the fact. Then several tools came and went as maintainers entered and exited the ecosystem. With Anaconda and conda, we are just now finally at a point where people can reliably install the basic scientific and numerical libraries, across hardware and OSes..... 20 years after I started using the language. :-)
https://news.ycombinator.com/item?id=11851871 https://archive.is/MVVU1
Explanation edit:
Image alt text is "The Python environmental protection agency wants to seal it in a cement chamber, with pictoral messages to future civilizations warning them about the danger of using sudo to install random Python packages."
The problem space is very difficult - it'll be something like AD 12000 before the waste is effectively harmless. That's further in the future than all of civilization[1] is behind us. The earliest surviving language specimens are proto-heiroglyphics that are less than 5000 years old.
It's difficult to craft a message that won't be taken like all the myriad curse-promising looter-deterrents guarding e.g. plundered tombs throughout the world. It's especially unconvincing when the hazards we're trying to protect future humans from include things like a hundredfold increased risk of cancer.....after inhabiting the site for twenty years.
It's further difficult to attempt to ensure that the message survives that massive amount of time. The designers came up with a linked system where elements are supposed to reinforce and index each other, with redundancy of information and presentation at multiple levels of complexity, with written and pictographic forms, attempting to avoid overstatement, attempting to ensure that protective structures cannot be usefully scrapped and re-used for new construction, ensure that the communications resist deliberate vandalization, etc.
1: https://en.wikipedia.org/wiki/History_of_the_world#Rise_of_c... claims "Though early 'cities' appeared at Jericho and Catal Huyuk around 6000 BCE,[32] the first civilizations did not emerge until around 3000 BCE in Egypt[33] and Mesopotamia.[34]"
The OS comes with a python runtime, which is different from the runtime you can download from python.org.
You can then also install it from Homebrew or from MacPorts.
I'm probably even missing something.
conda install r-essentials
All of these methods people are listing that requiring me to do more than manage one file (requirements.txt) is sort of missing the point.
I get that a lot of Node developers are coming to Python to do things, and that's great. But the environments don't need the exact same tools. pipenv seems to fix a problem I never encounter with pip/virtualenv during daily development work.
I was just going through an old server of mine. Python 2 and Python 3 dependencies, installed over years, in a bit of a mess.
We may have virtualenv and the like today, but a lot of old stuff was never setup to use it. :p
Pipenv is the most promising solution today but is still very, very new. It's modeled after yarn and has been officially blessed as "The One True Way" of installing stuff by the Python documentation. It has a way to go still to be as good as yarn (especially in terms of speed). The Python ecosystem has never had proper declarative packages like package.json (setup.cfg can be used to have fully declarative package metadata, but I seem to be the only one using it that way), which is a problem for package managers.
To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. It avoids the need for virtualenv, kind of, but so does pipenv and it does so more reliably and reproducibly (pipenv implements lockfiles).
Package manager/dependency manager is must for any sort of mature mass production coding environment (e.g. cargo for Rust, Maven for Java etc.) Docker doesn't solve all that but at least you are doing it only once and replicate easily and predictably. And a lot of times you can reuse Docker images built by someone else.
Oh? I don't see any mention of pipenv in the cpython git repo.
> Application dependency management
> Use Pipenv to manage library dependencies when developing Python applications. See Managing Application Dependencies for more details on using pipenv.
https://packaging.python.org/guides/tool-recommendations/
Package managers for languages were just starting to become a thing back then. (At least I think that is true...). Now days you wouldn’t dream of releasing a new language without some kind of standard package manager...
Dunno where I’m going with this comment but it is definitely interesting to see how much has changed since then. Hell even Microsoft has a package manager for it’s languages...
Isn't this one of the major criticisms of Go?
Rust is the exception here in that Cargo is a very nice piece of kit.
In terms of generic runtime version managers, ASDF is very nice cross language version manager for scoping versions to projects and switching between them easily.
Also Rustup, which is pretty key to Cargo remaining flexible going forward. There's a huge advantage to having the build/dependency management piece being highly coupled with the language runtime/compiler. Making cargo work with all versions of Rust would mean having to build in backwards compability and make it much more difficult to do projects like the new modules changes.
It's a key insight that there needs to be two tools surrounding a language, not just one. You need good dependency management and good version management. Node also has this right with nvm and npm/yarn.
Just consider urllib, urllib2, urllib3, and requests. Or getopt, optparse, argparse, and docopt. Or countless others.
I appreciate that the Python community continually strives for ever-more obvious and ergonomic ways of accomplishing tasks. While there is an ecosystem complexity cost to the solutions-that-weren't, it's no greater than the churn than I've seen in other popular languages.
The Zen of Python even pokes fun at the "one obvious way" by writing its three em-dashes three different ways in the same document.
I did not know this! Thanks :)
- It's very slow: re-locking after updating 1 dependency often takes me ~1 minute.
- It has lots of bugs. To name a few in 11.6.9: clobbers comments in the Pipfile, inconsistently includes dependencies for other build environments in Pipfile.lock, stores the wrong index in Pipfile.lock for packages not on PyPI.
- They release multiple times per day, often breaking things in patch releases.
- Kenneth Reitz is quite unpleasant to deal with in GitHub issues, which I often have to because of the previous 2 issues.
From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
https://github.com/pypa/python-packaging-user-guide/pull/443
He doesn't seem to be a pipenv maintainer:
https://github.com/pypa/pipenv/commits?author=jonparrott
Keep up the good work.
> From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
This is blatantly false.
Racket's exe are so simple and quick to share.
https://docs.racket-lang.org/raco/exe.html(Edited tendon to heel)
Back in the day, we used one of the various "freeze" applications (cxfreeze, bbfreeze, probably others) and some tools developed in-house to package and distribute them for various platforms.
It has been a while since I actually needed to do this, so I don't know what the current state of the art is.
Racket has such a great story for so many things that are painful in other environments. It's an under-appreciated language.
A) Racket is so enjoyable everyone builds their own solutions
B) "The view that full-fledged problem solving almost always calls for language design" Which sadly falls onto deaf ears especially people from Python.
I'm not sure it's because of backwards compatibility. Perl has amazing backwards compatibility, but doesn't seem to suffer from this.
That's not said to pump up Perl or degrade Python, but because if you misidentify the problem, your proposes solution has a much lower likelihood of fixing it.
(Then again, maybe you mean something different by "backwards compatibility" than what I thought you meant)
First of all it's not in Debian unstable. Ok, maybe it's just super new... so I'll install it with pip. A python3 -m pip install pipenv later and let's try it out!
$ cd ~/src/nexsan-exporter/nexsan-exporter $ pipenv install Creating a virtualenv for this project… Using /usr/bin/python3.6m (3.6.5) to create virtualenv… ⠋Running virtualenv with interpreter /usr/bin/python3.6m Using base prefix '/usr' New python executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m Also creating executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python Installing setuptools, pip, wheel...done.
Virtualenv location: /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG Installing dependencies from Pipfile.lock (ca72e7)… 0/0 — 00:00:00 To activate this project's virtualenv, run the following: $ pipenv shell
Cute emoji and nice colours but... ~/.local/share? For a directory that will end up containing arch-specific libraries? Ok, I guess no one gives a shit about this in the modern world, oh well. Wait... "python3.6m"? That isn't the Python interpreter I asked for... but it seems to be a hardlink to the same file as /usr/bin/python3.6 so I guess maybe this is intentional? Anyway let's check out the venv...
$ ls -l ~/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3* lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3 -> python3.6m lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6 -> python3.6m -rwxr-xr-x 1 yrro yrro 4576440 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m
So this is the Python folk's what, fourteenth attempt to get this right, and they are still copying the python executable into the virtual environment instead of symlinking it in? This seems to be a regression from venv, which seemed to get this right!
Right, time to install my dependencies... according to 'pipenv' the right command for this is 'pipenv install -e .' Hm, I wonder exactly what the -e option does?
$ pipenv install --help
* no menition of -e in the output *
:sadface:
Oh well, let's just run it blind!
$ pipenv install -e . Installing -e .… ⠏ Error: An error occurred while installing -e .! Directory '.' is not installable. File 'setup.py' not found.
Maybe I screwed up and ran this from the wrong directory?
$ ls setup.py setup.py
Weird, what's going on here?
$ strace -f pipenv install -e . 2>&1 | grep setup\\.py stat("/home/yrro/src/nexsan-exporter/nexsan-exporter/setup.py", {st_mode=S_IFREG|0644, st_size=1769, ...}) = 0 stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0200) = -1 ENOENT (No such file or directory) [pid 7365] stat("./setup.py", 0x7fff84176e30) = -1 ENOENT (No such file or directory) stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0300) = -1 ENOENT (No such file or directory) write(2, "Directory '.' is not installable"..., 62Directory '.' is not installable. File 'setup.py' not found.
Oh FFS, I give up. I think I'll let pipenv pass me by f...
I agree it's frustrating. I'm planning to donate some of my time to the project to help the situation. I hope others can too.
Can anyone give me a rundown of the benefits of other systems over virtualenv?
On a side note I hope Python isn't going to turn into a shambles like JavaScript - its certainly starting to look like it as far as package managers go.
But as I said, it's too young. If you use pipenv, you still have to interact with pip, so that's two binaries whose commands you have to remember. And urgh, it's so slow. And it's kinda useless for libraries... pipenv doesn't publish stuff, pipfiles aren't appropriate for packages.
But we'll get there. One day. Hopefully it won't be too late.
Pipenv is promising. That means at some point I'm fairly certain it'll be and deserve to be the standard in Python packaging. In the mean time, it doesn't.
I am teaching adult beginners and it would be best to avoid the messy parts in the beginning.
I thought I could get by with just pip and briefly touching on virtualenv.
I sin myself by avoiding all the mess by cloning a full bare bones Lubuntu virtualization whenever I need a clean project.
Theoretically Docker should suffice but it is yet another layer of complexity.
It’s so tempting to just apt install everything until it’s too late.
Sure, it caused a lot of grief for Debian packagers (because the name "site-packages" was hardcoded everywhere) when it was introduced back in the Python 2.6 era; but I don't think I've ever had to worry about it as a user.
It's not just official Debian packages that someone needs to maintain...