I do full time scientific python, and this post has some painfully wrong ideas.
>The official oletools install instructions talk about using either pip or setup.py. As a general rule, we're very strongly against installing anything system-wide except through Ubuntu's own package management system, and the environment our Python program runs in doesn't really have a home directory to use pip's --user option, so the obvious and simple pip invocations are out.
No! The obvious and simple pip invocation is as follows:
and then edit your code in `/path/to/my/installable/project/project-name/*.py` to your heart's content and tell your IDE or editor that `${HOME}/myvenv/bin/python` is the python to use. Activation is strictly optional, but if you like that, `source ${HOME}/bin/activate` (or use direnv to do it for you).
I would also recommend
export PIP_REQUIRE_VIRTUALENV=1
I divorce what $USER thinks `python` means from the system python as soon as I can with `pyenv global 3.8.5`: https://github.com/pyenv/pyenv
Of course never, ever `sudo pip ... `.
And don't touch $PYTHONPATH unless someone has forced you to use Caffe, who should be shamed on a regular basis for being Bad At Packages.
This takes a while but end result is a well enough optimized python. I also replace standard asyncio loop with uvloop to speed things up a little for async tasks.
Would love to hear your thoughts on pipenv as a real-world user of python environments. I leave it on the table that your current solution just works and you have never considered pipenv. ( I am not associated with pipenv. I've seemingly organically gravitated to it lately. )
my hot take: poetry won, and PEP517/518 build systems are going to be the way forward. Poetry might not last forever, but I think`pyproject.toml` has legs.
Pipenv was a valiant effort that moved the field along heaps, but I haven't used it in my daily life for two years.
> No! The obvious and simple pip invocation is as follows:
> [complex commands to set up a venv follows]
"All you have to do is... recompile your kernel... check your version dependencies... maybe do that once or twice. It's so simple, I don't know why everyone doesn't do it!"
Ubergeek.tv's classic "Switch to Linux" spoof ad of 2001:
I generalized a bit for completeness and to emphasize that contrary to the article's point, you can do all of this in $HOME and never install anything outside your home directory, which was a concern of the author as a usage policy for their server.
python -m venv myvenv
myvenv/bin/pip install ...
works just as well, i just needed a concrete path to talk about in the rest of the text.
Compared to "just pip" it's one whole extra line, so I don't think it's amazingly godawful at all.
You can certainly put a directory wherever you like and put the code there. In fact, that's exactly what the author did, as he describes in the article.
What you can't do is just magically have that directory be the "home" directory of a user and have the $HOME environment variable always point to it. You might be able to set things up to do that in principle, but it might not be worth the trouble as compared to other solutions. In fact, that's pretty much the position the author takes in the article for his use case.
> the position the author takes is "i've never heard of `venv` so i'll do these other terrible 5 hacks instead".
No, it's "I don't need the full power of venv so I'll just put the code in a specific directory now that I've figured out how to tell pip to do that when it's not a user's home directory". Using pip install's "--target" option hardly qualifies as a "terrible hack". The designers put it there because there were valid use cases for it. Setting environment variables before running a program is hardly a "terrible hack" either; it's one of the most common uses for shell scripts.
> $HOME or not is a red herring
If all you meant by it is "put the code in any directory you like", then, as I said, that's exactly what the author did.
Using virtualenvs is standard practice when developing with Python. What the OP did will work with every standard Python installation out there.
Most Python developers have virtualenv creation done for them via their IDE or scripts that vastly simplify the process so that you don't need to bother with the details in the OP.
You really don’t want to do that... they can break silently when the system updates Python. Use safe system wide installation for deployment, such as docker or traditional system packages. Alternatively, you can look at one of the many bunglers/static linkers like PyInstaller.
That's what cool kids use today; just dump complete dependency tree of every project into random location on filesystem. That's not how Debian OS works though, and that's what author, a sysadmin, talks about very clearly. So I don't get why you propose exactly what he dismissed in the 2nd paragraph.
> and the environment our Python program runs in doesn't really have a home directory to use pip's --user option
from the perspective of a developer, debian makes absolutely idiotic decisions about how python is packaged and managed by apt, so i opt out completely!
Really? I do `apt-get install foo` and later I in a Python script I write `import foo`. This Python script can be put ANYWHERE on the system. I don't need to mess up with virtualenvs and I have one version across all scripts on the system that's regularly updated on system upgrades (after being tested by community in bleeding edge versions of the OS). To each his own I guess.
apt doesn't have my stack or its dependencies at proper versions, and cannot in general meet the version-locking of dependencies i and my dependencies require.
Which is a very different use case from being a sysadmin for a network used by a large university, which is the use case of the author of the subject post. So it's not surprising that the "obvious and simple" solution that works for you doesn't work for him.
Also, the post discusses the option of using a virtual environment and explains why that option was not taken. So it's not like your "obvious and simple" solution was not considered. It just didn't turn out to be the right solution for this author's use case. (For one thing, the author explicitly notes that there is no "home" directory for this use case in which a venv can be easily set up.)
Virtual environments and installing to $HOME are different things. He says using virtual environments would be a lot more to learn but he just reinvented them.
> Virtual environments and installing to $HOME are different things.
I know that, and you know that, but I'm not sure the poster I was responding to knows that. In any case, for the use case of the author of the article, there is no "home directory" to use, so installing to $HOME was not an option.
> He says using virtual environments would be a lot more to learn but he just reinvented them.
He invented a sort of poor man's venv that just meets the particular needs of his use case. Sometimes that's a better solution than adopting a standard tool that requires more work and understanding to use.
There is no real direction in the article. I did not see any takeaway, or some kind of proposition or nice trick?
It can pretty much be summed up as "I don't know much of Python and admittedly don't want to spend time on it, so I did a bunch of hacks to download my dependencies and it worked".
> Nor do we want to learn about how to build and maintain Python virtual environments
I think the author would have better spent his time learning the basic usage of virtualenv/venv than writing this blog post.
> the author would have better spent his time learning the basic usage of virtualenv/venv than writing this blog post
Chris has a very low threshold for writing blog posts: if he does something he writes it up.
He also has it set up in a way that's very low overhead for him.
I like that he writes these sorts of snapshot blog posts; they're a much better view into what people are actually doing than
"I'm going to introduce this shiny new thing" posts.
> I think the author would have better spent his time learning the basic usage of virtualenv/venv than writing this blog post.
I don't. Virtualenv is a hack to workaround the mess of Python paths that they won't actually fix. I'd put it in a similar bucket as Docker (hacking around the mess of binary software distribution on Linux).
I don't want to spend my time learning workarounds.
But, it's a "hack workaround" that a professional consensus has converged upon. Pythonistas have developed supporting tools & help-texts, & many assume the use of silo'd virtual environments as 'basic hygiene' for serious work. The approach is also highly analogous to other "alternate virtual root" solutions across OSes/containers/user-customization schemes.
So instead of that, you'd rather... make you own custom hack workarounds? That will be idiosyncratic to your installations/projects? That will hit unique errors/tradeoffs the standard convention has already worked around? That will be harder for others to understand?
This is a solved problem and the solution is virtualenv.
Learning how to use venv's would have likely taken less time than the tinkering described in this post. If you're going to regularly develop with python and use 2nd/3rd party code, it's worth knowing.
It depends, binaries are just data. You reference that data with a path on the filesystem. In a programming language it would be assigning a value to a variable. For static values, the compiler/interpreter can optimize and do deduplication. And you expect it to do so because you don't want a global variable holding the value 1 to do the optimization yourself.
Relying on abstractions to simplify the architecture is a sign something might be right. Simply because simple things are less error-prone IMHO.
If you've just now taken the venv plunge wait until you learn about conda. Python is more or less a shitshow when it comes for packaging and it's the reason I'm drifting away from it little by little
The author uses Debian OS packages and he's talking about installing one unpackaged dependency. He's relying on what's packaged in OS and is tested with other packages. He clearly doesn't want to get random dump of dependencies on every installation (given you leave version empty in requirements.txt). Basically, he's relying on Debian for QA of packages. I do the same. I don't need to track versions of gazillion of dependencies for every little Python thing I write. I just install it in the OS and use it from anywhere else.
> Nor do we want to learn about how to build and maintain Python virtual environments
This is a shame, because this is one thing that Python gets mostly right. You don't have to fuss around with the shell wrapper, etc. For 90% of cases you can probably do:
You also lose tested versions from OS. You will either have to do your own QA of working dependency versions and manually keep them updated over time (e.g. security fixes, or stick with what you freeze forever), or just get dump of last versions every time you deploy the virtualenv.
Each package owner probably knows much better about the dependencies than the "OS". Save for a handful of Python packages that have extra-Python OS dependencies, relying upon the explicit versions called out by the package owner is generally more reliable.
With Python the future is here but it is not evenly distributed. In particular, it's been depressing to see that "any answer at all" has so often won out against "a correct answer".
Java has been a safe place to deploy code to because of its xenophobia. You don't expect Java to be linked into endless C libraries, and Java often doesn't come pre-installed thanks to license issues. (Maybe Larry Ellison is rich because you read "INSTALL JAVA TO EXPERIENCE THE POWER OF JAVA". The consequence is that you got a clean copy of Java and that your clean copy of Java won't break all the crapplets that came with your OS that expected an ancient Java that came with the OS.
Python got a boon from being shipped w/ most Linux distros, but it was also a setback... On Linux "python" became a synonym for "python 2" because otherwise you'd break the system scripts. (Today it drives me nuts that Microsoft added "python" to the past on Windows because Windows was the one operating system where you didn't have to hide the system Python to get a good Python)
"venv" is genius but conda got so much mindshare that many people never realized "venv" could do everything "conda" does except for corrupting internal data structures that force a periodic reinstall.
The consequence is that "best practices" haven't been institutionalized in most places, you have to continuously fight bad practices, and people like the poster are always inventing non-solutions to the problem venv solves.
(e.g. for instance, what about the person who uses a Docker container to keep python environments apart as opposed to one who uses venv?)
For conda it’s an unfair characterisation. Conda is language agnostic and can be (and is) used to ship anything. Packages like GCC/Clang/R will never be accessible for venv. Personally I find it magical to be able to ship complex packages on conda-forge that “just work” on Linux/macOS/Windows/ARM/POWER without requiring users to have special privileges.
Here's a specific example. In a previous job I had to work with Tensorflow models of various vintages that depended on different versions of CUDA.
Conda (out of the box) did nothing to help with that problem. I think an early version of Conda might have installed one early version of CUDA, but the elephant in the room as that I would need to install as many versions of CUDA as necessary.
NVIDIA uses a lot of intimidation to fool people as to the nature of the drivers. Specifically, CUDA/CuDNN and such don't depend on kernel drivers or Windows installers, but just require that some huge directories full of userspace libraries be on your path. Thus you can have CUDA version A linked into one conda environment, and CUDA version B linked into another conda environment and it all worked.
That is, it worked when I made my own conda packages (conda wasn't allowed to distribute such package by NVIDIA) and it was astonishingly slow because conda stores packages in BZIP2 files that take forever to unpack and because conda wastes more time with it's symlink machine.
In the end it was possible to solve the problem w/ conda but if I hadn't been distracted by conda I probably would have solved it just as quickly by unpacking the userspace drivers once and just setting the path appropriately.
You’re describing the consequence, not the cause, after making a pretty strong claim about a package manager a lot of people use daily. Want to pony up that cause?
poetry, pipx, and pyenv are pretty much all you need to be happy with python project management and sandboxing. They address the issues in the article and are far more ergonomic than plain venv wrangling.
57 comments
[ 4.1 ms ] story [ 118 ms ] thread>The official oletools install instructions talk about using either pip or setup.py. As a general rule, we're very strongly against installing anything system-wide except through Ubuntu's own package management system, and the environment our Python program runs in doesn't really have a home directory to use pip's --user option, so the obvious and simple pip invocations are out.
No! The obvious and simple pip invocation is as follows:
and then edit your code in `/path/to/my/installable/project/project-name/*.py` to your heart's content and tell your IDE or editor that `${HOME}/myvenv/bin/python` is the python to use. Activation is strictly optional, but if you like that, `source ${HOME}/bin/activate` (or use direnv to do it for you).I would also recommend
I divorce what $USER thinks `python` means from the system python as soon as I can with `pyenv global 3.8.5`: https://github.com/pyenv/pyenvOf course never, ever `sudo pip ... `.
And don't touch $PYTHONPATH unless someone has forced you to use Caffe, who should be shamed on a regular basis for being Bad At Packages.
Then I build python on target machine.
This takes a while but end result is a well enough optimized python. I also replace standard asyncio loop with uvloop to speed things up a little for async tasks.Pipenv was a valiant effort that moved the field along heaps, but I haven't used it in my daily life for two years.
https://github.com/python-poetry/poetry
Won't/can't fix issues:
https://github.com/pypa/pipenv/issues/1783
https://github.com/pypa/pipenv/issues/408
> [complex commands to set up a venv follows]
"All you have to do is... recompile your kernel... check your version dependencies... maybe do that once or twice. It's so simple, I don't know why everyone doesn't do it!"
Ubergeek.tv's classic "Switch to Linux" spoof ad of 2001:
https://www.youtube.com/watch?v=Xtah_05BOe8
Compared to "just pip" it's one whole extra line, so I don't think it's amazingly godawful at all.
For newbies, writing one line is impossible.
Which the author explicitly said does not exist for the use case he is discussing:
"the environment our Python program runs in doesn't really have a home directory"
You can certainly put a directory wherever you like and put the code there. In fact, that's exactly what the author did, as he describes in the article.
What you can't do is just magically have that directory be the "home" directory of a user and have the $HOME environment variable always point to it. You might be able to set things up to do that in principle, but it might not be worth the trouble as compared to other solutions. In fact, that's pretty much the position the author takes in the article for his use case.
$HOME or not is a red herring, i apologize if it appears more salient than it really was meant to be.
No, it's "I don't need the full power of venv so I'll just put the code in a specific directory now that I've figured out how to tell pip to do that when it's not a user's home directory". Using pip install's "--target" option hardly qualifies as a "terrible hack". The designers put it there because there were valid use cases for it. Setting environment variables before running a program is hardly a "terrible hack" either; it's one of the most common uses for shell scripts.
> $HOME or not is a red herring
If all you meant by it is "put the code in any directory you like", then, as I said, that's exactly what the author did.
Most Python developers have virtualenv creation done for them via their IDE or scripts that vastly simplify the process so that you don't need to bother with the details in the OP.
development != deployment.
The whole point of a container is to isolate dependencies.
The whole point of a virtualenv is also to isolate dependencies.
If using a container, why bother with a virtualenv at all?
> and the environment our Python program runs in doesn't really have a home directory to use pip's --user option
from the perspective of a developer, debian makes absolutely idiotic decisions about how python is packaged and managed by apt, so i opt out completely!
And --system-site-packages is a baaaaad idea.
Which is a very different use case from being a sysadmin for a network used by a large university, which is the use case of the author of the subject post. So it's not surprising that the "obvious and simple" solution that works for you doesn't work for him.
Also, the post discusses the option of using a virtual environment and explains why that option was not taken. So it's not like your "obvious and simple" solution was not considered. It just didn't turn out to be the right solution for this author's use case. (For one thing, the author explicitly notes that there is no "home" directory for this use case in which a venv can be easily set up.)
I know that, and you know that, but I'm not sure the poster I was responding to knows that. In any case, for the use case of the author of the article, there is no "home directory" to use, so installing to $HOME was not an option.
> He says using virtual environments would be a lot more to learn but he just reinvented them.
He invented a sort of poor man's venv that just meets the particular needs of his use case. Sometimes that's a better solution than adopting a standard tool that requires more work and understanding to use.
It can pretty much be summed up as "I don't know much of Python and admittedly don't want to spend time on it, so I did a bunch of hacks to download my dependencies and it worked".
> Nor do we want to learn about how to build and maintain Python virtual environments
I think the author would have better spent his time learning the basic usage of virtualenv/venv than writing this blog post.
Chris has a very low threshold for writing blog posts: if he does something he writes it up. He also has it set up in a way that's very low overhead for him.
I like that he writes these sorts of snapshot blog posts; they're a much better view into what people are actually doing than "I'm going to introduce this shiny new thing" posts.
I don't. Virtualenv is a hack to workaround the mess of Python paths that they won't actually fix. I'd put it in a similar bucket as Docker (hacking around the mess of binary software distribution on Linux).
I don't want to spend my time learning workarounds.
So instead of that, you'd rather... make you own custom hack workarounds? That will be idiosyncratic to your installations/projects? That will hit unique errors/tradeoffs the standard convention has already worked around? That will be harder for others to understand?
OK, to each their own!
Databases do it, Git does it, ZFS does it.
Ok the first two examples aren't filesystems but they can store files, so not totally irrelevant?
Relying on abstractions to simplify the architecture is a sign something might be right. Simply because simple things are less error-prone IMHO.
... oh, wait.
https://pypi.org/project/Basket/
This is a shame, because this is one thing that Python gets mostly right. You don't have to fuss around with the shell wrapper, etc. For 90% of cases you can probably do:
Java has been a safe place to deploy code to because of its xenophobia. You don't expect Java to be linked into endless C libraries, and Java often doesn't come pre-installed thanks to license issues. (Maybe Larry Ellison is rich because you read "INSTALL JAVA TO EXPERIENCE THE POWER OF JAVA". The consequence is that you got a clean copy of Java and that your clean copy of Java won't break all the crapplets that came with your OS that expected an ancient Java that came with the OS.
Python got a boon from being shipped w/ most Linux distros, but it was also a setback... On Linux "python" became a synonym for "python 2" because otherwise you'd break the system scripts. (Today it drives me nuts that Microsoft added "python" to the past on Windows because Windows was the one operating system where you didn't have to hide the system Python to get a good Python)
"venv" is genius but conda got so much mindshare that many people never realized "venv" could do everything "conda" does except for corrupting internal data structures that force a periodic reinstall.
The consequence is that "best practices" haven't been institutionalized in most places, you have to continuously fight bad practices, and people like the poster are always inventing non-solutions to the problem venv solves.
(e.g. for instance, what about the person who uses a Docker container to keep python environments apart as opposed to one who uses venv?)
Here's a specific example. In a previous job I had to work with Tensorflow models of various vintages that depended on different versions of CUDA.
Conda (out of the box) did nothing to help with that problem. I think an early version of Conda might have installed one early version of CUDA, but the elephant in the room as that I would need to install as many versions of CUDA as necessary.
NVIDIA uses a lot of intimidation to fool people as to the nature of the drivers. Specifically, CUDA/CuDNN and such don't depend on kernel drivers or Windows installers, but just require that some huge directories full of userspace libraries be on your path. Thus you can have CUDA version A linked into one conda environment, and CUDA version B linked into another conda environment and it all worked.
That is, it worked when I made my own conda packages (conda wasn't allowed to distribute such package by NVIDIA) and it was astonishingly slow because conda stores packages in BZIP2 files that take forever to unpack and because conda wastes more time with it's symlink machine.
In the end it was possible to solve the problem w/ conda but if I hadn't been distracted by conda I probably would have solved it just as quickly by unpacking the userspace drivers once and just setting the path appropriately.
Hold up, what? I have used both for years, how did you corrupt anything?
If he asks nicely on IRC, he might even get someone else to do it for him.