It'd have been really nice to have that PEP in as it'd have helped me not have to write local imports everywhere.
As it is, top-level imports IMHO are only meant to be used for modules required to be used in the startup, everything else should be a local import -- getting everyone convinced of that is the main issue though as it really goes against the regular coding of most Python modules (but the time saved to start up apps I work on does definitely make it worth it).
Gonna call this an antipattern. Do you need all those modules imported in every script ? Well then you save nothing on loadup time, the time will be spent regardless. Does every script not need those imports ? Well they shouldn't be importing those things and this small set of top level imports should be curated into a better, more fine grained list (and if you want to write tools, you can certainly identify these patterns using tooling similar to that which you wrote for LazyImports).
Interviewed with HRT awhile back. While I didn't get past the final round, their Python internals interview (which I did pass) was an absolute blast to prepare for, and required a really deep dive into implementation specific details of CPython around things like exactly how collisions are handled in dict, details about memory management, etc. Pretty much had to spend a few weeks in the CPython source to prep, and was, for me, worth the interview just to really learn what's going on.
For most teams I would be pretty skeptical of a internal Python fork, but the Python devs at HRT really know their stuff.
Its a finance firm - i.e scam firm. "We have a fancy trading algorithm that statistically is never going to outperform just buying VOO and holding it, but the thing is if you get lucky, it could".
Scammers are not tech people. And its pretty from their post.
> In Python, imports occur at runtime. For each imported name, the interpreter must find, load, and evaluate the contents of a corresponding module. This process gets dramatically slower for large modules, modules on distributed file systems, modules with slow side-effects (code that runs during evaluation), modules with many transitive imports, and C/C++ extension modules with many library dependencies.
As they should.
The idea that when you type something in the code and then the interpreter just doesn't execute it is how you end up with Java like services, where you have dependency injection chains that are so massive that when the first time everything has to get lazily injected the code takes a massive amount of time to run. Then you have to go figure out where is the initialization code that slows everything down, and start figuring out how to modify your code to make that load first, which leads to a mess.
If your python module takes a long time to load, this is a module problem. There is a reason why you can import submodules of modules directly, and overall the __init__.py in the module shouldn't import all the submodules by default. Structure your modules so they don't do massive initialization routines and problem solved.
Furthermore, because of pythons dynamic nature, you can do run time imports, including imports in functions. In use, whether you import something up at the top and it gets lazily loaded or you import something right when you have to use it has absolutely no difference other than code syntax, and the latter is actually better because you can see what is going on rather than the lazy loading being hidden away in the interpreter.
Or if you really care, you can implement lazy work process inside the modules, so when you import them and use them the first time it works exactly like lazy imports.
To basically spend time building a new interpreter with lazy loading just to be able to have all your import statements up at the top just screams that those devs prefer ideology over practicality.
I also interviewed with them a couple of years ago, for their Database SRE (AKA DBRE) role. It was going quite well until I discovered that they required you to live within commuting distance of an office, and unfortunately I had just moved away from one. They didn’t require in-person, supposedly, but needed the option, I guess?
I was quite impressed by the interviews, mostly for their pragmatism and skill-fitting. The programming interview wasn’t LC, it was “can you use a language (preferably Python) to parse a CSV and get useful information out of it,” because that’s the skill level the team needs. On the other hand, the Linux and DB interviews were quite in-depth, because again, the team needs those skills.
10/10 would interview again if I’m ever near an office again.
> This process gets dramatically slower for … modules on distributed file systems, modules with slow side-effects
Oh no. Look I'm not saying you're holding it wrong, it's perfectly valid to host your modules on what is presumably NFS as well as having modules with side effects but what if you didn't.
I've been down this road with NFS (and SMB if it matters) and pain is the only thing that awaits you. It seems like they're feeling it. Storing what is spiritually executable code on shared storage was a never ending source of bugs and mysterious performance issues.
Not sure why you're throwing shade on people making a living. Go lobby to your representative if you think the financial market should be changed instead of belittling folks doing a job.
Libraries for this have always existed, triggering import on first access. The problem was, they would break linters. But that's not an issue anymore with typing.TYPE_CHECKING.
A PEP is very much welcome, but using lazy import libraries is a fairly common, very old, method of speeding things up. My pre PEP 690 code looks like this:
import typing
from lazy import LazyImport
member = LazyImport('my_module.subpackage', 'member')
member1, member2, = LazyImport('my_module', 'member1', 'member2')
if typing.TYPE_CHECKING:
# normal import, for linter/IDE/navigation.
from my_module.subpackage import member
from my_module import member1, member2
Well if you use argparse or one of the many argparse wrappers for a moderately complex CLI you end up lazyfing the CLI parser itself because just fully populating the argparse data structures can easily take half a second or more, so with other startup costs you easily end up with "program --help" taking >1s and any CLI parsing error also taking >1s.
I wonder how much can be saved by using a local file system for imports though. In my testing just a mere presense of a home directory on NFS already dramatically slows down imports (by ~10x) due to Python searching for modules in home directory too by default.
to prevent this, set PYTHONNOUSERSITE=1 will prevent searching for modules in ~/.local/ (for convenience, try calling python through a wrapper in your project, say bin/run-python, and there you can set all the python-specific environment variables you need, set at the time of execution and not have to worry about setting them in the user's shell etc)
While I see the usefulness of lazy imports, it always seemed a bit backward to me for the importer to ask for lazy import, especially if you make it an import keyword rather than a Python flag. Instead I'd expect the modules to declare (and maybe enforce) that they don't have side effects, that way you know they can be lazily imported, and it opens the door for more optimizations, like declaring the module immutable. That links to the performance barrier of Python due to its dynamic nature as discussed in https://news.ycombinator.com/item?id=44809387
Of course that doesn't solve the overhead of finding the modules, but that could be optimized without lazy import, for example by having a way to pre-compute the module locations at install time.
Hmm it strikes me that is they really wanted to go this lazy route, they could've implemented an import hook, instead of creating and maintaining an entire fork.
> we support the Steering Council in their rejection of PEP 690—the implicit lazy imports are not a good fit for upstream due to the same, subtle bugs we encountered during our migration. However, as time permits, we hope to propose a revised lazy imports PEP that introduces an explicit lazy keyword, e.g. lazy import foo or lazy from foo import bar. This approach will satisfy migration and compatibility concerns, allow users to opt-in gradually, and enable all Python users to reap the speed benefits of lazy imports in a safe way.
We need something like the ancient unexec from Emacs to dump out Python images. More generally, we need something like that for generic checkpointing, maybe based on CRIU.
24 comments
[ 1.9 ms ] story [ 62.5 ms ] threadAlso maybe, if this approach could yield stats on if some import was needed or not ?
As it is, top-level imports IMHO are only meant to be used for modules required to be used in the startup, everything else should be a local import -- getting everyone convinced of that is the main issue though as it really goes against the regular coding of most Python modules (but the time saved to start up apps I work on does definitely make it worth it).
For most teams I would be pretty skeptical of a internal Python fork, but the Python devs at HRT really know their stuff.
Its a finance firm - i.e scam firm. "We have a fancy trading algorithm that statistically is never going to outperform just buying VOO and holding it, but the thing is if you get lucky, it could".
Scammers are not tech people. And its pretty from their post.
> In Python, imports occur at runtime. For each imported name, the interpreter must find, load, and evaluate the contents of a corresponding module. This process gets dramatically slower for large modules, modules on distributed file systems, modules with slow side-effects (code that runs during evaluation), modules with many transitive imports, and C/C++ extension modules with many library dependencies.
As they should.
The idea that when you type something in the code and then the interpreter just doesn't execute it is how you end up with Java like services, where you have dependency injection chains that are so massive that when the first time everything has to get lazily injected the code takes a massive amount of time to run. Then you have to go figure out where is the initialization code that slows everything down, and start figuring out how to modify your code to make that load first, which leads to a mess.
If your python module takes a long time to load, this is a module problem. There is a reason why you can import submodules of modules directly, and overall the __init__.py in the module shouldn't import all the submodules by default. Structure your modules so they don't do massive initialization routines and problem solved.
Furthermore, because of pythons dynamic nature, you can do run time imports, including imports in functions. In use, whether you import something up at the top and it gets lazily loaded or you import something right when you have to use it has absolutely no difference other than code syntax, and the latter is actually better because you can see what is going on rather than the lazy loading being hidden away in the interpreter.
Or if you really care, you can implement lazy work process inside the modules, so when you import them and use them the first time it works exactly like lazy imports.
To basically spend time building a new interpreter with lazy loading just to be able to have all your import statements up at the top just screams that those devs prefer ideology over practicality.
I was quite impressed by the interviews, mostly for their pragmatism and skill-fitting. The programming interview wasn’t LC, it was “can you use a language (preferably Python) to parse a CSV and get useful information out of it,” because that’s the skill level the team needs. On the other hand, the Linux and DB interviews were quite in-depth, because again, the team needs those skills.
10/10 would interview again if I’m ever near an office again.
Oh no. Look I'm not saying you're holding it wrong, it's perfectly valid to host your modules on what is presumably NFS as well as having modules with side effects but what if you didn't.
I've been down this road with NFS (and SMB if it matters) and pain is the only thing that awaits you. It seems like they're feeling it. Storing what is spiritually executable code on shared storage was a never ending source of bugs and mysterious performance issues.
A PEP is very much welcome, but using lazy import libraries is a fairly common, very old, method of speeding things up. My pre PEP 690 code looks like this:
I'd say if you see
Its probably 99% safe to pull that from a quick run over of the AST (and caching that for the later import if you want to be fancy)Of course, should one be doing a star import in a proper codebase?
Of course that doesn't solve the overhead of finding the modules, but that could be optimized without lazy import, for example by having a way to pre-compute the module locations at install time.
> monorepo
> vast proliferation of imports
> large modules
> distributed file system
> side-effects
> many transitive imports
This sounds like a very optional problem to have.