Since moving to Nix, I've been forced to setup virtual envs for my more interesting projects (oh sure, I install the common libraries at the system level), and though I resisted this workflow a lot initially (I like to keep a tight ship), I've come to embrace its practicality when working on projects that have libs with breaking changes due to rapid releases.
it's saved me a world of pain a number of times to have different versions of the same libs installed in different envs
On most Linux distros, it's a requirement for a lot of the desktop functionality. Why would having a default Python environment you can count on be a annoyance?
This is what I love about Nix. It has actually completely solved dependency hell for me. It's always a little painful to move an existing project to Nix but I have never regretted it.
That is in fact also possible, and you could use it to make this magicimport system get called automatically whenever you try to import a non-existent library.
D has import std; which just imports the whole standard library in one go (for writing). You might be able to write your own (not sure if python has public imports)
As someone who has been coding for 20 years, "magic" is a negative adjective for me. I don't want magic code. I want code that is simple and works in obvious ways.
> You can even specify a version number
Sorry to be sarcastic here but wow, you mean I can even make sure that my code will not break at an arbitrary point in the future, when my dependency inevitably introduces a breaking change? Fantastic.
I understand that installing dependencies is a bit of a hassle. But really, once you have done it twice it takes seconds. And I would not want a mechanism that fetches dependencies as a side effect of my app running. I want to know what's going on when my app runs. The fewer moving parts, the better. Often, it's hard work to achieve this simplicity. But, at least in my experience, it pays off.
This is a toy/neat experiment that someone made and they explicitly say you should not use it for production. This sort of criticism is inappropriate (or at best, poorly framed). This is Hacker News, and this is a Hack.
If I made a bot that drives cars around in GTA would you make a post ridiculing how dangerous my lane detection algorithm would be for a self-driving car in snow?
Furthermore, the point OP makes about L5 cars being presumably achievable while "L5 software" is not is a 100% valid point - why should we not aspire to that level of consistency and effectiveness in our systems? Maybe achieving this is actually possible and projects like this inspire the approaches that actually deliver those capabilities in a production-mature form.
You're basically saying that I am not allowed to give my opinion on this project, because it happens to be critical. I don't know how we can have interesting discussions in this forum if criticism is not allowed.
At the end of the day, I think you're saying it's a fun toy and I shouldn't worry, whereas my point was that I wouldn't even want to use it as a toy, and I am afraid it will lead especially less experienced programmers who do use it in production down a path that hurts them in the long run.
To comment further, I believe you take offense at the snarkiness of my original comment. That's fair. I think I could have phrased it in a more respectful way. I'm sorry.
Nope, your comment is poor because it’s off-topic, not critical. You saw the project and launched into a mild rant about “kids these days who use libraries for everything” when on-topic comments would be “I hope nobody actually uses this ;) Although in all seriousness based on my experience this kind of thinking is unironically already present in the software engineering community”. Your original comment fails to tie itself back to the project and it fails to demonstrate that you understood it was not meant for use in production. As such, it deserves as much disdain as a “Mozilla should fire its CEO” comment should get on a post about SpiderMonkey.
It's not at all self-evident that something like this is bad to inexperienced or naive developers, even if the readme has the obligatory "this is just for personal projects, don't use it in production" disclaimer. The comment is perfectly on-topic.
Honestly, even a “this a bad idea to use in production” would be an on-topic comment, and as I said the comment in and of itself is not poor it is just misplaced and such a lead could help it transition between the two topics. The primary issue is that the comment is not “don’t use this in production”, it’s just straight up “importing all the dependencies is what’s wrong in software today” which is no longer connected to the post.
People are allowed to point out when that comment is appropriate for a different context. They're allowed to do that with imperfect language too.
You're allowed to acknowledge you're wrong, and move on. You're also allowed to push back and explain why you think it's fine for this context.
The only person who has control over you is you. This is a web forum. A Random Stranger on the Internet calling a comment inappropriate shouldn't lead one to believe that "criticism is not allowed" in a particular forum.
And I think it's a nice toy. I haven't seen people break for using toys. I've mostly seen people break when they only do things one way, and habituate to believe other ways are wrong or bad. Toys are for playing, and great ways to discover which things work and which ones don't. Even IOCCC didn't result in broken programmers. Java sometimes did.
So I don't see either your comment or their response as ideal, but that's okay. I'd estimated 25% of my comments are poor, and you'd probably estimate me at 50% since I probably don't always realize my comments are poor. That's okay. Poor comments are parts of conversations. Pointing out they're poor is part of conversation too. Continuing to talk (and sometimes make poor comments) is too.
I'm with you 100% on this. Someone posted a project that is clearly a bad idea to actually use and your criticisms are perfectly justified and on-topic (not to mention correct).
I've also seen too many things of this nature in production to buy into the "it's just a toy, it's self-evidently not for real use" argument.
This. The Node.js community is a good example of where this has happened - lots of bad practices with fancy logos making it into the top 50 most depended upon packages (e.g. left-pad).
The top-level comment is just being critical. It's not inappropriate. People just don't like having their feelings hurt.
While I agree with this (in prod or development headed to prod), I'd love to be able to hand this to some of my colleagues who are either from Ruby or Lisp backgrounds or who do data science for use in Jupyter notebooks. Being able to just start Jupyter, start coding some small experiment and not worry about imports would be useful. Further, it would help my colleagues in data science who require a step by step walkthrough whenever we need to generate requirements.txt.
In my personal case, I have loads of dead code directories for various half-remembered Python ideas. It'd be convenient to have a lighter-weight "experiments" dir using something like this.
I can't say I really like the idea of a "magic" import, but I'm sure some people might find it handy. It just sounds like a recipe for having things break later in mysterious ways but if it's just a one-time use sort of thing I can see the appeal.
Personally I've been really happy with with pyenv[0].You can install multiple different versions of Python side-by-side and it does a great job managing virtualenvs.
pipx[1] is also pretty good for managing virtualenvs to install applications in but IMHO less so for doing actual development. You can use it to seamlessly install venv'd Python apps or spin up apps in ephemeral environments.
The downside of such an approach is that it hides the inevitable complexity of dependency management. Things like version constraints, licensing and platform specifica are important. If developers don't see them, they won't learn about them.
For instance I encountered a python wheel lately that "conveniently" shipped a precompiled .so file. Not only was that a security problem but also did that file not work properly under Ubuntu 18.04.
this is a good example of a "simple versus easy" trade off [1] , i.e. this is something that is easier to get started at some expense of simplicity: it is over-complicated & hides complexity behind a simple interface, but not in a way that is reliable - so eventually things will break and then you'll be strictly more confused than if you did things in a less easy but simpler way - learning a bit about dependency management & setting up a reproducible environment for running your python script/app.
that said, it is an amusing hack to run the virtualenv'd python in a subprocess and then extract the environment vars and inject them into the already running python process.
I'm disappointed to see so many negative HN comments about this project. I think it is really cool!
You shouldn't use this in a production web service--and the author says as much in the README--I think there are some great uses for this code:
1. Demonstrating how the Python import system works. There is a lot of nuance at hand that other programming languages (like C) don't have an analogy for.
2. Creating truly self-contained Jupyter notebooks. Right now, if I want to send someone a working Jupyter notebook, I have to send them a requirements.txt (or, better yet, a Dockerfile) for them to install in a virtualenv. This is too bad, because specifying the dependencies in the notebook itself is better for teaching purposes.
When I got my start on Linux with Ubuntu (Breezy Badger IIRC) I compiled many (most?) of the utilities I wanted from source. When `make` didn’t “JustWorkTM”, it give very clear reasons why, and even as a noob I could usually correct the issue. Python , Docker, and NodeJS etc still leave much to be desired in that regard.
> This only occurs in development (using serve or watch), however in production (using build) autoinstall is disabled to prevent unwanted side-effects on deployment.
> You can disable this feature using --no-autoinstall.
It was a joke rather than a serious tool. Caught import errors and runtime errors and would install pip packages going back from most recent to oldest, until the script under test worked without error. At least, that was the idea —- I never really finished it.
Looks cool, I have a small non-open source solution for this purpose that caries this out just using different syntax. The focus there was getting jupyter notebooks and dashboards that also detail their runtime environment requirements.
Compared to R, pythons imports syntax and handling is a god-sent when sharing code, but we still saw a lot of people get stuck after cloning a notebook or dashboard trying to figure out all the dependencies they needed to set up to run it.
Now they just copy the notebook/dashboard and first run configures everything For them referencing both internal and external dependencies.
It’s a small but important difference especially for juniors.
In the future I would love for python just to support something like “import pandas==1.1.3@pypi” and having that automatically do what every python developer who works with virtual envs would do if they saw the “==1.1.3@pupi” in a comment after the import. But then obviously also supporting installs from other sources than pypi.
What happens if someones code accidentally (or purposely) uses a malicious module [0] in their code? Seems like a great way to spread some malware... It would be cool if there was some sort of dependency validation system integrated into this.
> The ONLY package it will automatically install to your system (and in user-space, only) is virtualenv itself
I don't understand why it even uses virtualenv, when venv is a) part of the standard library, and b) the recommended way to create virtual environments in Python.
While the import from __future__ indicates that it's supposed to support Python 2, the shebang line is explicitly Python 3...
One big reason is that ubuntu splits out venv into its own package basically "breaking" python. People then don't know about the venv module and continues using the virtualenv, both due to this preceding venv, and because that is usually the word used to describe the concept.
48 comments
[ 2.9 ms ] story [ 107 ms ] threadit's saved me a world of pain a number of times to have different versions of the same libs installed in different envs
https://docs.python.org/3/reference/import.html#import-hooks
Would definitely recommend Poetry for anything larger, it's been a great experience.
For example if I do glob.glob("*.txt") it's rather obvious that I also wanted to "import glob".
> Parse it, find all not initialized variables.
> Search imports, suitable for found variables. Import them.
Genius. (Evil genius mabye...)
If it finds multiple functions / objects with the same name, it gives you choices. For me on larger projects it shows choices ~80% of the time.
I wonder how can this work correctly on anything larger than the provided example.
> You can even specify a version number
Sorry to be sarcastic here but wow, you mean I can even make sure that my code will not break at an arbitrary point in the future, when my dependency inevitably introduces a breaking change? Fantastic.
I understand that installing dependencies is a bit of a hassle. But really, once you have done it twice it takes seconds. And I would not want a mechanism that fetches dependencies as a side effect of my app running. I want to know what's going on when my app runs. The fewer moving parts, the better. Often, it's hard work to achieve this simplicity. But, at least in my experience, it pays off.
If I made a bot that drives cars around in GTA would you make a post ridiculing how dangerous my lane detection algorithm would be for a self-driving car in snow?
Furthermore, the point OP makes about L5 cars being presumably achievable while "L5 software" is not is a 100% valid point - why should we not aspire to that level of consistency and effectiveness in our systems? Maybe achieving this is actually possible and projects like this inspire the approaches that actually deliver those capabilities in a production-mature form.
At the end of the day, I think you're saying it's a fun toy and I shouldn't worry, whereas my point was that I wouldn't even want to use it as a toy, and I am afraid it will lead especially less experienced programmers who do use it in production down a path that hurts them in the long run.
People are allowed to point out when that comment is appropriate for a different context. They're allowed to do that with imperfect language too.
You're allowed to acknowledge you're wrong, and move on. You're also allowed to push back and explain why you think it's fine for this context.
The only person who has control over you is you. This is a web forum. A Random Stranger on the Internet calling a comment inappropriate shouldn't lead one to believe that "criticism is not allowed" in a particular forum.
And I think it's a nice toy. I haven't seen people break for using toys. I've mostly seen people break when they only do things one way, and habituate to believe other ways are wrong or bad. Toys are for playing, and great ways to discover which things work and which ones don't. Even IOCCC didn't result in broken programmers. Java sometimes did.
So I don't see either your comment or their response as ideal, but that's okay. I'd estimated 25% of my comments are poor, and you'd probably estimate me at 50% since I probably don't always realize my comments are poor. That's okay. Poor comments are parts of conversations. Pointing out they're poor is part of conversation too. Continuing to talk (and sometimes make poor comments) is too.
I've also seen too many things of this nature in production to buy into the "it's just a toy, it's self-evidently not for real use" argument.
Specifically stating to not use in production will not stop people from doing just that!
Requirement files aren't magic, neither are virtual envs. Every slightly professional IDE supports that. No need to create anti-patterns.
The top-level comment is just being critical. It's not inappropriate. People just don't like having their feelings hurt.
In my personal case, I have loads of dead code directories for various half-remembered Python ideas. It'd be convenient to have a lighter-weight "experiments" dir using something like this.
[0]: https://python-poetry.org/
Personally I've been really happy with with pyenv[0].You can install multiple different versions of Python side-by-side and it does a great job managing virtualenvs.
pipx[1] is also pretty good for managing virtualenvs to install applications in but IMHO less so for doing actual development. You can use it to seamlessly install venv'd Python apps or spin up apps in ephemeral environments.
[0]: https://github.com/pyenv/pyenv [1]: https://github.com/pipxproject/pipx
For instance I encountered a python wheel lately that "conveniently" shipped a precompiled .so file. Not only was that a security problem but also did that file not work properly under Ubuntu 18.04.
that said, it is an amusing hack to run the virtualenv'd python in a subprocess and then extract the environment vars and inject them into the already running python process.
some of the code could be cleaned up a bit -- e.g. it could be using https://docs.python.org/3/library/tempfile.html#tempfile.Tem... to just ask for a uniquely named temp dir, some of the subprocess invocations dont appear to have error handling, etc.
for my python hobby project these days i only depend upon packages i can install using apt in debian stable. containers for isolation help too.
[1] https://www.infoq.com/presentations/Simple-Made-Easy/
You shouldn't use this in a production web service--and the author says as much in the README--I think there are some great uses for this code:
1. Demonstrating how the Python import system works. There is a lot of nuance at hand that other programming languages (like C) don't have an analogy for.
2. Creating truly self-contained Jupyter notebooks. Right now, if I want to send someone a working Jupyter notebook, I have to send them a requirements.txt (or, better yet, a Dockerfile) for them to install in a virtualenv. This is too bad, because specifying the dependencies in the notebook itself is better for teaching purposes.
When I got my start on Linux with Ubuntu (Breezy Badger IIRC) I compiled many (most?) of the utilities I wanted from source. When `make` didn’t “JustWorkTM”, it give very clear reasons why, and even as a noob I could usually correct the issue. Python , Docker, and NodeJS etc still leave much to be desired in that regard.
[0]: https://en.parceljs.org/hmr.html#automagically-installed-dep...
> You can disable this feature using --no-autoinstall.
Seems fine to me.
I wrote a script that would try and automatically determine missing packages and their version numbers a while a go: https://gist.github.com/JosephRedfern/ef52916e70497bc9631d6e...
It was a joke rather than a serious tool. Caught import errors and runtime errors and would install pip packages going back from most recent to oldest, until the script under test worked without error. At least, that was the idea —- I never really finished it.
Famous last words.
Compared to R, pythons imports syntax and handling is a god-sent when sharing code, but we still saw a lot of people get stuck after cloning a notebook or dashboard trying to figure out all the dependencies they needed to set up to run it.
Now they just copy the notebook/dashboard and first run configures everything For them referencing both internal and external dependencies.
It’s a small but important difference especially for juniors.
In the future I would love for python just to support something like “import pandas==1.1.3@pypi” and having that automatically do what every python developer who works with virtual envs would do if they saw the “==1.1.3@pupi” in a comment after the import. But then obviously also supporting installs from other sources than pypi.
[0](https://www.zdnet.com/article/two-malicious-python-libraries...)
I don't understand why it even uses virtualenv, when venv is a) part of the standard library, and b) the recommended way to create virtual environments in Python.
While the import from __future__ indicates that it's supposed to support Python 2, the shebang line is explicitly Python 3...
For example, one could replace a python shebang with fades' one and mark dependencies using # fades comments:
https://github.com/PyAr/fades