this is cool, ive been burned by trying to move a virtualenv before and discovering everything broke. is it possible to merge this into virtualenv? i think it would be much easier to use that way.
Would be useful for packaging up a virtualenv in an RPM (though I avoid doing that for various reasons), if there was an option to fix the paths without actually performing the move.
I looked at that but didn't use it because I wanted to avoid adding yet another packaging tool into our toolbox. Instead I use virtualenv-tools and fpm. We use FPM for packaging needs and it works well enough.
Last time I did this, I used pip with --download on the build server, and --no-index & --find-links on the deployment target into a virtualenv in /usr/share/<package_name>. It was a pain to get the pre- & post-actions right.
This still required internet access btw, but at least the deployments didn't fail when a host serving packages was unavailable.
The machine you are installing from may not have access to internet. Then you'll need a setup to supply pip the packages. If virtualenv was truly relocatable, you can install once, push all to git and just fetch wherever you need.
Albeit less featureful than pew or virtualenvwrapper, pyenv[0] is great also.
It looks like pew uses virtualenv-clone[1] internally. However a casual test indicated quite a few shebangs unmodified and in the case of venv, the activation scripts as well.
> a better wrapper around virtual environments compared to virtualenv wrapper
This comment alone tells me there's something wrong about python's modularisation. And I'm saying this as a python programmer. Why didn't Python 3 clean up this mess? Or am I missing something there?
I would agree to that yes, as are most (all) languages.
It would be very nice if the Python version was something specified in the project manifest. Don't know what is the popular build tool in python, but something like specifying the node.js version in the package.json file would be great.
I think the reason it never works like that is that the language is written before the package manager and VM/compiler.
In Clojure it's the other way around - the JVM and the package managers were there before the language. The Clojure compiler is a library that you include in your project.clj file like any other lib. Hence you can specify language version per project. That's a feature I would like to have in other programming environments.
No, my point is rather: Why do we need virtualenv at all (and worse, more wrappers around it)? Why should, say, pip packages have any problems with each other as long as they aren't imported directly in a python sourcefile. It seems like versioning of packages isn't solved correctly at the interpreter level, that's why we have the virtualenv crutch.
Why should, say, pip packages have any problems with each other as long as they aren't imported directly in a python sourcefile.
Are you suggesting that package versions should be explicitly provided as part of import statements? Or that import statements should be mangled depending on the versions specified in setup.py?
There is Python, then there is the tooling around Python. Virtual environments constitute tooling around Python. Sometimes aspects of the tooling become indispensable and they are adopted as part of the standard library -- see venv.
You may not realise it, but Python usage patterns have changed significantly over the last 6-8 years and they are still changing. With such change, it is hard to create any tooling that is concrete and stable over the long term. By that same reason not a lot of tooling can be adopted by the standard library.
I can see that reasoning given how Python started, but since Python3 broke backwards compatibilities anyways it should IMO have solved the whole packaging, versioning and isolation problem once and for all through standard utilities delivered with every Python distro.
If there are no agreed upon proposals, then why implement solutions if their utility would still be inadequate. It is not enough to demand perfect solutions, there must be perfect proposals.
Well, sometimes you need an architect rather than a committee. This is one such time.
Looking at the whole thing superficially (i.e. just throwing something at the wall here), this is how I'd have done it:
* packages are by default always installed in user space.
* package manager is built into import system. Importing a package that's not in the user's environment yet will lead to something like pip install at script startup.
* import statements understand versions. By default the latest version of a package is loaded. If something else is needed, the programmer can specify the specific one.
* for most third party packages that are publicised in pypy it's recommended to bind all imports to specific versions of dependencies, so it can be tested in this configuration at release.
> package manager is built into import system.
> Importing a package that's not in the user's
> environment yet will lead to something like
> pip install at script startup
Here's a nifty little recipe I came up with to do just this by exploiting the import hooks introduced in py3.4
Seems to me that virtualenv was a quick n dirty solution that hardcoded a lot of things in the subtree; but that proved extremely useful at the time so it became a standard.
Dependency management isn't really a solved problem in any language ecosystem or operating system AFAIK. I'd love an example of the best case.
I feel like this is why things like Docker exist.
For a language like python, I'm not sure this is something you would really want to try to solve at the language level. Let's say you solved it for pure python code. You'd still have the problem of interfacing with native libraries and the OS itself. For instance, NumPy and SciPy are typically kind of a pain due to Fortran dependencies.
I think Docker is a pretty nice solution. I feel like for any language, any time this becomes painful, I assume I should either be using Docker or AWS+Chef. Disposable VMs essentially.
In a sense, virtualenv is a lot like a VM. It provides a reliable execution environment on demand without forcing you to have an actual different machine. And you don't have to deal with virtualization. But. . . it has its limitations. Because, if this is what really matters to you, you might really be wanting a VM.
Languages that run in VMs already (like Java) solve this problem a bit better at the language level. But Java dependency management turns out to be a nightmare anyway. This is probably just one of those hard problems. But if you really want reliable environments for just your app, I think it's going to be better in the long term to think about containerization and virtual machines instead of virtual environments.
What have you seen go wrong with Java packaging? I've had very, very few issues with it. The fact that you can package compiled blobs in JARs means you don't need to rely on the OS package manager. You can get into some hairy class loader situations with dependencies on conflicting versions, but most other environments don't even let you try and resolve those issues.
This should be called virtualenv-rename right, otherwise what does virtualenv --relocatable do?
Python/Java using the file system to map hierarchies of classes is such a mistake when you start using a language like Elixir; you can move any .ex files anywhere you like within your project and it just works.
The script is most akin to the Unix `mv` command, in fact it uses it, so that's why I called it virtualenv-mv. I figured this would be most easily understood.
Just out of curiosity, have you used `--relocatable` much? It basically attempts to rewrite full paths into relative paths. Whereas with `virtualenv-mv`, there should be no indication that the environment has been moved. As far as `--relocatable` is concerned... I think even the author(s) regret having included it, if that gives you any indication of how useful it is.
Why would you rename a virtualenv? They should be named "env" and you should change the prompt that pops when you enter it to be something like "(VENV: projectname)".
You might want to move our copy it to another directory. I had this problem recently when I wanted to use an env that had a bunch of packages that take a long time to compile, but since it didn't work I had to reinstall them in a new env.
30 comments
[ 4.5 ms ] story [ 77.0 ms ] threadinstallation via pip ought to take advantage of wheels and such, no?
This still required internet access btw, but at least the deployments didn't fail when a host serving packages was unavailable.
It also supports renaming.
It looks like pew uses virtualenv-clone[1] internally. However a casual test indicated quite a few shebangs unmodified and in the case of venv, the activation scripts as well.
[0]: https://github.com/yyuu/pyenv
[1]: https://pypi.python.org/pypi/virtualenv-clone
This comment alone tells me there's something wrong about python's modularisation. And I'm saying this as a python programmer. Why didn't Python 3 clean up this mess? Or am I missing something there?
It would be very nice if the Python version was something specified in the project manifest. Don't know what is the popular build tool in python, but something like specifying the node.js version in the package.json file would be great.
I think the reason it never works like that is that the language is written before the package manager and VM/compiler.
In Clojure it's the other way around - the JVM and the package managers were there before the language. The Clojure compiler is a library that you include in your project.clj file like any other lib. Hence you can specify language version per project. That's a feature I would like to have in other programming environments.
Are you suggesting that package versions should be explicitly provided as part of import statements? Or that import statements should be mangled depending on the versions specified in setup.py?
Yep, see also my suggestion in the other subthread.
You may not realise it, but Python usage patterns have changed significantly over the last 6-8 years and they are still changing. With such change, it is hard to create any tooling that is concrete and stable over the long term. By that same reason not a lot of tooling can be adopted by the standard library.
Looking at the whole thing superficially (i.e. just throwing something at the wall here), this is how I'd have done it:
* packages are by default always installed in user space.
* package manager is built into import system. Importing a package that's not in the user's environment yet will lead to something like pip install at script startup.
* import statements understand versions. By default the latest version of a package is loaded. If something else is needed, the programmer can specify the specific one.
* for most third party packages that are publicised in pypy it's recommended to bind all imports to specific versions of dependencies, so it can be tested in this configuration at release.
I feel like this is why things like Docker exist.
For a language like python, I'm not sure this is something you would really want to try to solve at the language level. Let's say you solved it for pure python code. You'd still have the problem of interfacing with native libraries and the OS itself. For instance, NumPy and SciPy are typically kind of a pain due to Fortran dependencies.
I think Docker is a pretty nice solution. I feel like for any language, any time this becomes painful, I assume I should either be using Docker or AWS+Chef. Disposable VMs essentially.
In a sense, virtualenv is a lot like a VM. It provides a reliable execution environment on demand without forcing you to have an actual different machine. And you don't have to deal with virtualization. But. . . it has its limitations. Because, if this is what really matters to you, you might really be wanting a VM.
Languages that run in VMs already (like Java) solve this problem a bit better at the language level. But Java dependency management turns out to be a nightmare anyway. This is probably just one of those hard problems. But if you really want reliable environments for just your app, I think it's going to be better in the long term to think about containerization and virtual machines instead of virtual environments.
Python/Java using the file system to map hierarchies of classes is such a mistake when you start using a language like Elixir; you can move any .ex files anywhere you like within your project and it just works.
Just out of curiosity, have you used `--relocatable` much? It basically attempts to rewrite full paths into relative paths. Whereas with `virtualenv-mv`, there should be no indication that the environment has been moved. As far as `--relocatable` is concerned... I think even the author(s) regret having included it, if that gives you any indication of how useful it is.