Ask HN: How do you handle/maintain local Python environments?
1 - Python shipped with OS X/Ubuntu
2 - brew/apt install python
3 - Anaconda
4 - Getting Python from https://www.python.org/downloads/
And that's before getting into how you get numpy et al installed. What's the general consensus on which to use? It seems like the OS X default is compiled with Clang while brew's version is with GCC. I've been working through this book [1] and found this thread [2]. I really want to make sure I'm using fast/optimized linear algebra libraries, is there an easy way to make sure? I use Python for learning data science/bioinformatics, learning MicroPython for embedded, and general automation stuff - is it possible to have one environment that performs well for all of these?
[1] https://www.amazon.com/Python-Data-Analysis-Wrangling-IPython/dp/1449319793
[2] https://www.reddit.com/r/Python/comments/46r8u0/numpylinalgsolve_is_6x_faster_on_my_mac_than_on/
99 comments
[ 5.3 ms ] story [ 185 ms ] thread1. Pick Python binary / Official Docker Image
2. Use virtualenv for Developer Boxes
3. Well defined Pipfile / requirements.txt
4. Avoid most binary package building. Prefer those with readymade wheels. But if it has to be built, I prefer to make static libraries, use them to make static ELFs / modules, as opposed to /path/to/some.so, and bundle them / ship them as wheels for both development / production.
Again, you can set up simple Dockerfiles / scripts to do this for you.
These reasons:
- I can't use one version of a binary distro for production, because fixes keep coming.
- Every company / people I work with uses different *nix distro, so the initial setup can look different, Esp when using bundled pythons.
- Irrespective of that, the final outcome should be normalized and sane. This is where virtualenv, tox - these kinds of things help.
- Source standardization is quite easy with Pipfile, so I pretty much rely on that mechanism.
- I have enough experience building libraries (shared and dynamic) and quite aware of ELF and PE issues that sometimes, custom solutions are needed. I avoid them as much, but sometimes they are needed. Why?
Maybe you don't have an Intel CPU (so CPUID based checks break) , or maybe you want to run TF under ROCm and not CUDA, or you're trying to compile a new shared library for RPi because nobody builds armv7l wheels unless they use a Pi, Or you need to link against OpenSSL/libxml for that wsgi based project - and keep runtime/image sizes small on a container.
In these specific cases, more often than not, stay away from anyone who promises you that distro X would / has always worked for them.
- pip + requirements.txt I find is more than acceptable
- wheels > source when it comes to installing / distributing packages
It’s simple. It’s easy to debug. I genuinely don’t understand the point of all this other nonsense like anaconda.
Unfortunately when it comes to data-science-Python it seems like the patients are running the asylum. The APIs and developer ergonomics of tools like tensorflow, pandas etc... are so bizarre to people who are running production, vanilla Python.
pandas.read_table(filepath_or_buffer: Union[str, pathlib.Path, IO[~AnyStr]], sep='t', delimiter=None, header='infer', names=None, index_col=None, usecols=None, squeeze=False, prefix=None, mangle_dupe_cols=True, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skipinitialspace=False, skiprows=None, skipfooter=0, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=False, skip_blank_lines=True, parse_dates=False, infer_datetime_format=False, keep_date_col=False, date_parser=None, dayfirst=False, cache_dates=True, iterator=False, chunksize=None, compression='infer', thousands=None, decimal=b'.', lineterminator=None, quotechar='"', quoting=0, doublequote=True, escapechar=None, comment=None, encoding=None, dialect=None, error_bad_lines=True, warn_bad_lines=True, delim_whitespace=False, low_memory=True, memory_map=False, float_precision=None)
I use R happily enough, and MATLAB too for that matter. Maybe, some-day, I'll see the pandas-light.
I am not a big fan of (the over-use of) optional arguments either: I prefer code I'm reading to be more explicit, which is more pythonic imo.
I disagree with the code is thus doing too much, however.
At least in this specific and some other cases, the method shown there is a wrapper to do a major task of converting data formats from one form to another.
These optional, well-documented arguments show the defaults used, and are modifiable in, as you said, specific use-cases.
Given how utterly powerful it is, I think that's OK.
Not sure a large list of optional params (with good documentation) is a bad thing though.
I'm not a data scientist so I have little experience with these tools, except for Pandas, which I won't use unless I have to.
For data science/numerical computation, all batteries included. It also has fast optimized linear algebra (MKL) plus extras like dask (paralellization), numba, out of the box. No fuss no muss. No need to fiddle with anything.
Everything else is a "pip install" or "conda install" away. Virtual envs? Has it. Run different Python versions on the same machine via conda environments? Has it. Web dev with Django etc.? All there. Need to containerize? miniconda.
The only downside? It's quite big and takes a while to install. But it's a one time cost.
Precompiled MKL is really nice. Conda and conda-forge now build for aarch64. There are very few wheels for aarch64 on PyPI. Conda can install things like Qt (IPython-qt, spyder,) and NodeJS (JupyterLab extensions).
If I want to switch python versions for a given condaenv (instead of just creating a new condaenv for a different CPython/PyPy version), I can just run e.g. `conda install -y python=3.7` and it'll reinstall everything in the depgraph that depended on the previous python version.
I always just install miniconda instead of the whole anaconda distribution. I always create condaenvs (and avoid installing anything in the root condaenv) so that I can `conda-env export -f environment.yml` and clean that up.
BinderHub ( https://mybinder.org/ ) creates docker containers from {git repos, Zenodo, FigShare,} and launches them in free cloud instances also running JupyterLab by building containers with repo2docker (with REES (Reproducible Execution Environment Specification)). This means that all I have to do is add an environment.yml to my git repo in order to get Binder support so that people can just click on the badge in the README to launch JupyterLab with all of the dependencies installed.
REES supports a number of dependency specifications: requirements.txt, Pipfile.lock, environment.yml, aptSources, postBuild. With an environment.yml, I can install the necessary CPython/PyPy version and everything else.
...
In my dotfiles, I have a setup_miniconda.sh script that installs miniconda into per-CPython-version CONDA_ROOT and then creates a CONDA_ENVS_PATH for the condaenvs. It may be overkill because I could just specify a different python version for all of the conda envs in one CONDA_ENVS_PATH, but it keeps things relatively organized and easily diffable: CONDA_ROOT="~/-wrk/-conda37" CONDA_ENVS_PATH="~/-wrk/-ce37"
I run `_setup_conda 37; workon_conda|wec dotfiles` to work on the ~/-wrk/-ce37/dotfiles condaenv and set _WRD=~/-wrk/-ce37/dotfiles/src/dotfiles.
Similarly, for virtualenvwrapper virtualenvs, I run `WORKON_HOME=~/-wrk/-ve37 workon|we dotfiles` to set all of the venv cdaliases; i.e. then _WRD="~/-wrk/-ve37/dotfiles/src/dotfiles" and I can just type `cdwrd|cdw` to cd to the working directory. (Some of the other cdaliases are: {cdwrk, cdve|cdce, cdvirtualenv|cdv, cdsrc|cds}. So far, I have implemented cdalias support for bash, IPython, and vim)
One nice thing about defining _WRD is I can run `makew <tab>` and `gitw` to `cd $_WRD; make <tab>` and `git -C $_WRD` without having to change directory and then `cd -` to return to where I was.
So, for development, I use a combination of virtualenvwrapper, pipsi, conda, and some shell scripts in my dotfiles that I should get around to releasing and maintaining someday. https://westurner.github.io/dotfiles/venv
For publishing projects, I like environment.yml because of the REES support.
1 Setup a production Python environment on Windows Server 2016
2 Write a README on how to set up a prod Python env on Windows Server 2016 (pretty short)
3 Tell my Win 10 coworkers to refer to the README to setup their own Python env
4 Minimize the number of "conda install" commands needed to setup a production Python env, which in our case are mostly the Oracle/cx_oracle and MS SQL Server dependencies.
Simply using pipenv and pyenv is enough for me (brew install pipenv pyenv). You don't ever have to think about this, or worry you're doing it wrong.
Every project has an isolated environment in any python version you want, installed on demand. You get a lockfile for reproducibility (but this can be skipped) and the scripts section of the pipfile is very useful for repetitive commands. It's super simple to configure the environment to become active when you move into a project directory.
It's not all roses, pipenv has some downsides which I hope the new release will fix.
When I need a temporary environment to mess around in, then I use virtualfish by running `vf tmp`: https://virtualfish.readthedocs.io/en/latest/
1. https://github.com/menpo/menpo/blob/master/setup.py#L17
For a while I was encouraging people to use conda especially on windows, but with WSL I found it to be mostly unnecessary.
I do still leverage pip install --user for tools that are stable enough and I use frequently.
It takes way too long to make a Pipenv.lock file, and I need to be in the project folder to activate the env?! Terrible user experience.
I develop using pipenv and then deploy installing the dependencies directly in the folder. It works for AWS lambda and containers, although lambda layers are a little more tricky to release:
`pipenv lock -r > requirements.txt`
`pipenv run pip install -r requirements.txt -t .`
I used to have a Makefile to do the install dependencies in directory, test locally, zip the directory up, deploy to AWS jobs - but SAM is a really good replacement for that workflow. It uses docker containers too, and the other big thing it makes trivial if you need it is running an API gateway locally so you can test the Lambda as an HTTP API instead of just hacking events in as standard input/function inputs.
The CLI used to be slow, but since 4.6 the performance is reasonable -- still slower than some non-Python package managers, but in my experience no longer so slow as to be aggravating (< 10s install now, whereas previously >30-60s in the past just solving the environment)
> their unofficial builds are opaque
I'm not entirely sure what builds you are referring to, but conda-forge [1] is completely open source and community-driven, and Anaconda recipes are in a github organization called AnacondaRecipes [2]. If you are talking specifically about the Python interpreter builds for the Anaconda distribution, I believe the build recipes are here [3] (although I don't know that for a fact).
> my experience the very few packages that are not available on pypi are rarely worth installing
Depending on your field, platform, and dependencies, that is certainly believable. However, several projects such as Apache Arrow and Rapids have dropped support for PyPI wheels [4] because the engineering cost of wheel-building on linux is prohibitive, and conda is preferable on windows for a variety of reasons.
> usually the worst kind of "research code"[1]
That's an oddly-specific complaint. Anyway, given the variety of environments where people run `setup.py`, that kind of thing is usually explained by "someone put in a PR to make it build somewhere." It might be surprising if you've never built or packaged python extensions with native dependencies before, but I'm not sure it's a particular indicator of quality.
[1] https://conda-forge.org/ [2] https://github.com/AnacondaRecipes [3] https://github.com/AnacondaRecipes/python-feedstock [4] https://medium.com/rapids-ai/rapids-0-7-release-drops-pip-pa...
By opaque I mean: https://github.com/AnacondaRecipes/tensorflow_recipes/tree/m...
What do these do. Why are they adding an alternative gettime implementation for MacOS? Why are they patching a bunch of test files to make them pass?
> That's an oddly-specific complaint.
It's unfortunately one we ran into recently while migrating a legacy system off conda, but it's not the first and I doubt it will be the last.
The gettime patch is to support macOS SDK < 10.12, as it says in the patch itself. From a quick look, some of the test patches appear to be for Python 2 compatibility. Every linux distribution and system integrator has a patch list just like that for complicated software. If a particular patch concerns you, I would suggest opening an issue on their bug tracker to ask for an explanation.
> It's unfortunately one we ran into recently while migrating a legacy system off conda, but it's not the first and I doubt it will be the last.
Unclear what that has to do with conda. The same package is available on PyPI (source only), and the line you highlighted almost certainly exists for some other build target than conda, where the compilers will automatically find numpy includes in CONDA_PREFIX without any extra effort.
If you’re on a Mac, just use the brew installation (brew install). If you’re on some type of prod/containerized setup, use apt’s python (apt-get install).
I would not recommend building Python from source unless you _really_ know what you’re doing on that level, as you can unintentionally shoot yourself in the foot quite a bit. From there just using a virtualenv should be pretty straightforward.
In this way, you’re letting the package managers (written by much smarter people than you and I) do the heavy lifting.
Don’t use the system Python, brew-installed Python, it the official Python installer, if you can avoid it (you’ll want your Python to be up to date and self-contained in your home folder as much as possible)
https://github.com/direnv/direnv/wiki/Python#pyenv
(FWIW: there are tools to create Nix expressions from requirements files, so it isn’t necessarily all manual work.)
https://github.com/asdf-vm/asdf
In case of Python, I use asdf to install the needed interpreter version and I use pipenv to create/handle the virtual environments. It's a perfect combo.
Code from an older Python 3 version should always work in the newer Python 3 version though (at least for 3.5 -> 3.7).
https://github.com/pyenv/pyenv
Linux: Typically using the package manager, or the AUR if I'm not using the latest version. At the moment, I'm using 3.6 as my "universal" version since that's what ships with current Ubuntu.
Windows: Good ol' fashioned executable installers, though there's really no reason I don't use chocolatey other than sheer force of habit.
macOS: I don't do enough work on macOS to have a strong opinion, though generally I just use the python.org installer. I don't think there's that much of a difference from the Homebrew version, but I could be wrong on that front. As an aside: IIRC, the "default" python shipped with macOS is still 2.X, and they're going to remove it outright in 10.15. So I wouldn't rely on that too heavily.
As for other tooling, IME pipenv and poetry make dealing with multiple versions of Python installed side-by-side much easier. I have a slight preference for poetry for a variety of reasons, but both projects are worth checking out.
Finally, at the end of the day, the suggestions in here to "just use Docker" aren't unreasonable. Performance between numpy on e.g. 3.6 vs 3.7 or clang vs GCC likely aren't that significant, but if you create a Docker environment that you can also use for deployment you can be sure you're using packages compiled/linked against the same things.
If all of this sounds like an unreasonable amount of effort for your purposes... probably just use Anaconda. It's got a good reputation for data science applications for a good reason, namely removing much of this insanity from the list of things you have to worry about.
Every couple months I nuke all my environments and my install and start fresh out of general frustration. It’s still the least bad for a scientific workflow IMO (having different environments but all globally accessible rather than stuck in specific directories is nice).
I wouldn't use the python shipped with osx because it tends to be out of date rather quickly, and it doesn't ship with libreadline.
Usually the python shipped with Ubuntu is good if it has the version you want.
If you compile it yourself from source you are in for a bad time unless you know how to install all the dependencies. Make sure you turn on optimizations when running ./configure or it will be significantly less fast.
Once you have python installed, install pip. Try the following until one works (unless you use anaconda, then you won't use pip most of the time, and I think you can just `conda install pip` if it's not there by default):
$ sudo pip install -U pip (for python3: sudo pip3 install -U pip)
$ sudo easy_install pip (easy_install-3 or something like that for python3)
(after, if those don't work, that you need to get it from somewhere else, most likely by installing python-pip via apt or whatever the package is in brew or whatever you are using)
When you want to install something globally, do NOT do `sudo pip install ...`, this can break things in a way that is hard to repair, do `pip install --user ...`, and make sure the path that --user installs are installed in is added to your $PATH. I think it's always ~/.local/bin/
If you are working on a project, always use a virtualenv, google for more details on that.
- brew python for my global environment.
- Create a virtual enviroment for each project (python -m venv).
- Use pip install after activating it (source venv/bin/activate)
If you need to work with different versions of python replace brew python with pyenv.
[1]https://code.visualstudio.com/docs/python/environments
Docs: https://code.visualstudio.com/docs/remote/containers
workon [project name] and deactivate.
Pycharm sees these and they work.
replace pyenv with asdf [0]. asdf is like pyenv, but it works for all major programming languages.
[0] https://github.com/asdf-vm/asdf
The fact that it can do multiple languages and environments is lovely.
My current technique for that is a pseudo-wrapper script that activates each 'env' and runs associated commands with it to set it up, e.g. bash exports, $PATH, etc.
Now I just need to fit asdf into my workflow.
Separately I also use anaconda if I need all the scientific packages + Jupyter etc.
And, beyond numpy/scipy (where the conda versions might be better optimize), often pip-install packages from PyPI into those conda environments.