Love this. My https://llm.datasette.io/ CLI tool supports plugins, and people were complaining about really slow start times even for commands like "llm --help" - it turned out there were popular plugins that did things like import pytorch at the base level, so the entire startup was blocked on heavy imports.
Note that this is global to the entire process, so for example if you make an import of Numpy lazy this way, then so are the imports of all the sub-modules. Meaning that large parts of Numpy might not be imported at all if they aren't needed, but pauses for importing individual modules might be distributed unpredictably across the runtime.
Edit: from further experimentation, it appears that if the source does something like `import foo.bar.baz` then `foo` and `foo.bar` will still be eagerly loaded, and only `foo.bar.baz` itself is deferred. This might be part of what the PEP meant by "mostly". But it might also be possible to improve my implementation to fix that.
> Has ongoing performance overhead on every attribute access.
I would have expected so, but in my testing it seems like the lazy load does some kind of magic to replace the proxy with the real thing. I haven't properly dug into it, though. It appears this point is removed in the live version (https://peps.python.org/pep-0810).
> Doesn’t work well with from ... import statements.
Hmm. The PEP doesn't seem to explain how reification works in this case. Per the above it's a solved problem for modules; I guess for the from-imports it could be made to work essentially the same way. Presumably this involves the proxy holding a reference to the namespace where the import occurred. That probably has a lot to do with restricting the syntax to top level. (Which is the opposite of how we've seen soft keywords used before!)
> Requires verbose setup code for each lazy import.
> Less clear and standard than dedicated syntax.
If you want to use it in a fine-grained way, then sure.
Is it another potential solution (until PEP 810 is accepted) to override the NameError exception, decide if it was triggered by an unloaded package from a list, and then running again that line of code? I understand the inefficiency of this solution (e.g. the same line could trigger NameError several times and you need to run it again until all modules are loaded) but this is a good brainstorming thread.
That sounds very unpleasant. However nicely you wrapped it up, you'd still be referring to the code for that process everywhere that the NameError could occur.
If a tool has different capabilities that use different imports, why load all of them if only a subset is required?
As a simple example, a tool that can generate output in various formats (e.g., json, csv, xml, ...) should only import the appropriate modules to handle the output format after having determined which ones will be used in this invocations.
I think really the problem is that packages like pytorch take so long to import. In my work I've tried a few packages (not AI stuff) that do a lot of work on import. It's actually quite detrimental because I have to setup environment variables to pass things that should be arguments of a setup function in. All things considered a python module shouldn't take any noticeable time to import
Parse the command line and do things like "--help" without doing the imports.
Only do imports when you know you need them -- or as an easy approximation, only if the easy command line options have been handled and there's still something to do.
Just FWIW, a trick that I'm planning to use for PAPER: first I make separate actual commands — `paper-foo`, `paper-bar` etc. that are each implemented as separate top-level scripts that can import only what they need. Later, the implementation of `paper foo` has the main `paper` script dynamically look up `paper-foo`. (Even a `subprocess.call` would work there but I'd like to avoid that overhead)
It's not just help - the plugins need to be imported so the root level CLI tool knows what to do if you type "llm subcommand ..." where that subcommand is defined by a plugin.
It doesn't know which plugin defines a subcommand until it imports the plugin's module.
I'm happy with the solution I have now, which is to encourage plugin authors not to import PyTorch or other heavy dependencies at the root level of their plugin code.
> It doesn't know which plugin defines a subcommand until it imports the plugin's module.
That might be considered a design mistake -- one that should be easy to migrate away from.
You won't need to do anything, of course, if the lazy import becomes available on common Python installs some day in the future. That might take years, though.
Lazy imports have been proposed before, and were rejected most recently back in 2022: https://discuss.python.org/t/pep-690-lazy-imports-again/1966.... If I recall correctly, lazy imports are a feature supported in Cinder, Meta's version of CPython, and the PEP was driven by folks that worked on Cinder. Last time, a lot of the discussion centered around questions like: Should this be opt-in or opt-out? At what level? Should it be a build-flag for CPython itself? Etc. The linked post suggests that the Steering Council ultimately rejected it because of the complexity it would introduce to have two divergent "modes" of importing.
I hope this proposal succeeds. I would love to use this feature.
I also hope this proposal succeeds, but I'm not optimistic. This will break tons of code and introduce a slew of footguns. Import statements fundamentally have side effects, and when and how these side effects are applied will cause mysterious breakages that will keep people up for many nights.
This is not fearmongering. There is a reason why the only flavor of Python with lazy imports comes from Meta, which is one of the most well-resourced companies in the world.
Too many people in this thread hold the view of "importing {pandas, numpy, my weird module that is more tangled than an eight-player game of Twister} takes too long and I will gladly support anything that makes them faster". I would be willing to bet a large sum of money that most people who hold this opinion are unable to describe how Python's import system works, let alone describe how to implement lazy imports.
PEP 690 describes a number of drawbacks. For example, lazy imports break code that uses decorators to add functions to a central registry. This behavior is crucial for Dash, a popular library for building frontends that has been around for more than a decade. At import-time, Dash uses decorators to bind a JavaScript-based interface to callbacks written in Python. If these imports were made lazy, Dash would break. Frontends used by thousands, if not millions of people, would immediately become unresponsive.
You may cry, "But lazy imports are opt-in! Developers can choose to opt-out of lazy imports if it doesn't work for them." What if these imports were transitive? What if our frontend needed to be completely initialized before starting a critical process, else it would cause a production outage? What if you were a maintainer of a library that was used by millions of people? How could you be sure that adding lazy imports wouldn't break any code downstream? Many people made this argument for type hints, which is sensible because type hints have no effect on runtime behavior*. This is not true for lazy imports; import statements exist in essentially every nontrivial Python program, and changing them to be lazy will fundamentally alter runtime behavior.
This is before we even get to the rest of the issues the PEP describes, which are even weirder and crazier than this. This is a far more difficult undertaking than many people realize.
---
* You can make a program change its behavior based on type annotations, but you'd need to explicitly call into typing APIs to do this. Discussion about this is beyond the scope of this post.
Product manager for Dash here At Plotly we're actually pretty excited about the potential for lazy loaded imports, as it could help out a lot with the import performance of Plotly.py.
As this comment mentions Dash apps would not support lazy loaded imports until the underlying Dash library changes how it loads in callbacks and component libraries (the two features which would be most impacted here), but that doesn't mean there's no path to success. We've been discussing some ways we could resolve this internally and if this PEP is accepted we'd certainly go further to see if we can fully support lazy loaded imports (of both the Dash library itself/Dash component libraries and for relative imports in Dash apps).
They are not entitled to hold the opinion that their imports takes too long, if they dont know the inner workings of pythons import system? Do you listen to yourself?
Right now in python, you can move import statement inside a function. Lazy imports at top level are not needed. All lazy imports do is make you think less about what you are writing. If you like that, then just vibe code all of your stuff, and leave the language spec alone.
Ie, with lazy, the import happens at the site of usage. Since clearly this is code that could already be written, it only breaks things in the sense that someone could already write broken code. Since it is opt in, if using it breaks some code, then people will notice that and choose not to rewrite that code using it.
I don't see how. It adds a new, entirely optional syntax using a soft keyword. The semantics of existing code do not change. Yes, yes, you anticipated the objection:
> What if these imports were transitive? ... How could you be sure that adding lazy imports wouldn't break any code downstream?
I would need to see concrete examples of how this would be a realistic risk in principle. (My gut reaction is that top-level code in libraries shouldn't be doing the kinds of things that would be problematic here, in the first place. In my experience, the main thing they do at top level is just eagerly importing everything else for convenience, or to establish compatibility aliases.)
But if it were, clearly that's a breaking change, and the library bumps the major version and clients do their usual dependency version management. As you note, type hints work similarly. And "explicitly calling into typing APIs" is more common than you might think; https://pypistats.org/packages/pydantic exists pretty much to do exactly this. It didn't cause major problems.
> Import statements fundamentally have side effects, and when and how these side effects are applied will cause mysterious breakages that will keep people up for many nights.
They do have side effects that can be arbitrarily complex. But someone who opts in to changing import timing and encounters a difficult bug can just roll back the changes. It shouldn't cause extended debugging sessions unless someone really needs the benefits of the deferral. And people in that situation will have been hand-rolling their own workarounds anyway.
> Too many people in this thread hold the view of "importing {pandas, numpy, my weird module that is more tangled than an eight-player game of Twister} takes too long and I will gladly support anything that makes them faster".
I don't think they're under the impression that this necessarily makes things faster. Maybe I haven't seen the same comments you have.
Deferring imports absolutely would allow, for example, pip to do trivial tasks faster — because it could avoid importing unnecessary things at all. As things currently stand, a huge fraction of the vendored codebase will get imported pretty much no matter what. It's analogous to tree shaking, but implicitly, at runtime and without actually removing code.
Yes, this could be deferred to explicitly chosen times to get more or less the same benefit. It would also be more work.
> Libraries such as PyTorch, Numba, NumPy, and SciPy, among others, did not seamlessly align with the deferred module loading approach. These libraries often rely on import side effects and other patterns that do not play well with Lazy Imports. The order in which Python imports could change or be postponed, often led to side effects failing to register classes, functions, and operations correctly. This required painstaking troubleshooting to identify and address import cycles and discrepancies.
This isn't precisely the scenario I described above, but it is a concrete example of how deferred imports can cause issues that are difficult to debug.
Regarding performance benefits:
> At Meta, the quest for faster model training has yielded an exciting milestone: the adoption of Lazy Imports and the Python Cinder runtime. ... we’ve been able to significantly improve our model training times, as well as our overall developer experience (DevX) by adopting Lazy Imports and the Python Cinder runtime.
Some of these worries make sense, but wouldn’t it be relatively trivial to pass a flag to the interpreter or something similar in order to force all imports to evaluate, as in the current behavior? But to be a bit cheeky if some of these issues cause serious production outages for you it might be time to consider moving on from a scripting language altogether.
The issue is that some imports can be made lazy and some cannot. A binaristic all-or-nothing approach does not address the issue. (I also think that there is zero basis to claim that adding such a flag is trivial, since there’s no reference implementation of this flavor of lazy imports.)
What if we have a program where one feature works only when lazy imports are enabled and one feature only when lazy imports are disabled?
This is not a contrived concern. Let’s say I’m a maintainer of an open-source library and I choose to use lazy imports in my library. Because I’m volunteering my time, I don’t test whether my code works with eager imports.
Now, let’s say someone comes and builds an application on top of this library. It doesn’t work with lazy imports for some unknown reason. If they reach for a “force all imports” flag, their application might break in another mysterious way because the code they depend on is not built to work with eager imports. And even if my dependency doesn’t break, what about all the other packages the application may depend on?
The only solution here would be for the maintainer to ensure that their code works with both lazy and eager imports. However, this imposes a high maintenance cost and is part of the reason why PEP 690 was rejected. (And if your proposed solution was “don’t use libraries made by random strangers on the Internet”, boy do I have news for you...)
My point is that many things _will_ break if migrated to lazy imports. Whether they should have been written in Python in the first place is a separate question that isn’t relevant to this discussion.
Maybe the package that requires lazy can somehow declare that requirement, so another package that tries to force not lazy will fail early and realize it needs to replace this dependency with something compatible or change its ways. It definitely adds complexity, though.
Or check at runtime if it's running with the lazy import feature active. Then instead of breaking in mysterious ways in production it would crash on startup, during development.
Theoretically the implementation may use the approach "as lazy as possible". Traverse lazy imports until you encounter a regular one.
I doubt it will make much difference, but at least it gives an option.
From merely browsing through a few comments, people have mostly positive opinions regarding this proposal. Then why did it fail many times, but not this time? What drives the success behind this PEP?
It's been explained many times before why this is not possible: the library doesn't actually have a version number. The distribution of source code on PyPI has a version number, but the name of this is not connected to the name of any module or package you import in the source code. The distribution can validly define zero or more modules (packages are a subset of modules, represented using the same type in the Python type system).
You got three other responses before me all pointing at uv. They are all wrong, because uv did not introduce this functionality to the Python ecosystem. It is a standard defined by https://peps.python.org/pep-0723/, implemented by multiple other tools, notably pipx.
> It's been explained many times before why this is not possible: the library doesn't actually have a version number. The distribution of source code on PyPI has a version number, but the name of this is not connected to the name of any module or package you import in the source code.
You're making the common mistake of conflating how things currently work with how things could work if the responsible group agrees to change how things work. Something being the way it is right now is not the same as something else being "not possible".
No, changing this breaks the world. A huge fraction of PyPI becomes completely invalid overnight, and the rest fails the expected version checks. Not to mention that the language is fundamentally designed around the expectation that modules are singleton. I've written about this at length before but I can't easily find it right now (I have way too many bookmarks and not nearly enough idea how to organize them).
Yes, you absolutely can create a language that has syntax otherwise identical to Python (or at least damn close) which implements a feature like this. No, you cannot just replace Python with it. If the Python ecosystem just accepted that clearly better things were clearly better, and started using them promptly, we wouldn't have https://pypi.org/project/six/ making it onto https://pypistats.org/top (see also https://sethmlarson.dev/winning-a-bet-about-six-the-python-2...).
The hard part is making the change. Adding an escape hatch so older code still works is easy in comparison.
Nobody is claiming this is a trivial problem to solve but its also not an impossible problem. Other languages have managed to figure out how to achieve this and still maintain backwards compatibility.
Note that you will be expected to have familiarized yourself generally with previous failed proposals of this sort, and proactively considered all the reasonably obvious corner cases.
I’m not going to spend an entire weekend drafting a proposal instead of spending time with my kids, just to win “internet points”.
If you want examples then just look at one of the other languages that have implemented compiler / runtime dependency version checks.
Even Go has better dependency resolution than Python, and Go is often the HN poster child for how not to do things.
The crux of the matter is this is a solvable problem. The real issue isn’t that it’s technically impossible, is that it’s not annoying enough of a day to day problem for people who are in a position to influence this change. I’m not that person and don’t aspire to be that person (I have plenty of other projects on my plate as it is)
In spite of the 'You're welcome to bring' this does not actually sound like an encouragement but more of a veiled statement that some non-technical reason will be found to shoot down the proposal if it were to be made so you might as well not bother.
It's an allusion to the fact that there is a very long history establishing that the problem is not as simple as it sounds, even if you get past the most basic issues, and it's hard to explain it all in a single coherent post.
Nobody suggested it was easy. We were just arguing against your claim that it’s impossible
Saying something is possible isn’t the same as saying something is easy.
If you were to have said “it’s a different problem to solve because of…” then you wouldn’t have had any pushback. But you didn’t. You said it “this is not possible”. And that’s the part that people were disputing.
No, the point is that most people in this thread do not appreciate the complexity of implementing lazy imports. If you disagree, your energy is better spent talking to a CPython core developer about implementation details of making baseless assertions from an ivory tower.
There are many people here who think enabling lazy imports is as simple as flipping a light switch. They have no idea what they're talking about.
And actually people do appreciate the complexities of changes like this. We were responding to a specific comment that that said “it’s impossible”. Saying something is “possible” isn’t the same as saying “it’s easy”.
> Not to mention that the language is fundamentally designed around the expectation that modules are singleton.
Modules being singletons is not a problem in itself I think? This could work like having two versions of the same library in two modules named like library_1_23 and library_1_24. In my program I could hypothetically have imports like `import library_1_23 as library` in one file, and `import library_1_24 as library` in another file. Both versions would be singletons. Then writing `import library==1.23` could be working like syntax sugar for `import library_1_23 as library`.
Of course, having two different versions of a library running in the same program could be a nightmare, so all of that may not be a good idea at all, but maybe not because of module singletons.
I know I'm missing something but wouldn't it be possible to just throw an import error when that happens? Would it even break anything? If I try:
import numpy==2.1
And let's say numpy didn't expose a version number in a standard (which could be agreed upon in a PEP) field, then it would just throw an import exception. It wouldn't break any old code. And only packages with that explicit field would support the pinned version import.
And it wouldn't involve trying to extract and parse versions from older packages with some super spotty heuristics.
But it would make new code impossible to use with older versions of python, and older packages, but that's already the case.
> And let's say numpy didn't expose a version number in a standard (which could be agreed upon in a PEP) field, then it would just throw an import exception. It wouldn't break any old code. And only packages with that explicit field would support the pinned version import.
Yes, this part actually is as simple as you imagine. But that means in practical terms that you can't use the feature until at best the next release of Numpy. If you want to say for example that you need at least version 2 (breaking changes, after all), well, there are already 18 published packages that meet that requirement but are unable to communicate that in the new syntax. This can to my understanding be fixed with post releases, but it's a serious burden for maintainers and most projects are just not going to do that sort of thing (it bloats PyPI, too).
And more importantly, that's only one of many problems that need to be solved. And by far the simplest of them.
You could absolutely have this be part of the language in any regard. The question then becomes how does one implement it in a reasonable way. I think every package should have a __version__ property you should be able to call, then you could have versioned imports.
In fact there's already many packages already defining __version__ at a package level.
Edit: What they are solving with UV is at the moment of standing up an environment, but you're more concerned about code-level protection, where are they're more concerned about environment setup protection for versioning.
> In fact there's already many packages already defining __version__ at a package level.
This only helps for those that do, and it hasn't been any kind of standard the entire time. But more importantly, that helps only the tiniest possible bit with resolving the "import a specific version" syntax. All it solves is letting the file-based import system know whether it found the right folder for the requested (or worse: "a compatible") version of the importable package. It doesn't solve finding the right one if this one is wrong; it doesn't determine how the different versions of the same package are positioned relative to each other in the environment (so that "finding the right one" can work properly); it doesn't solve provisioning the right version. And most importantly, it doesn't solve what happens when there are multiple requests for different versions of the same module at runtime, which incidentally could happen arbitrarily far apart in time, and also the semantics of the code may depend on the same object being used to represent the module in both places.
> It's been explained many times before why this is not possible: the library doesn't actually have a version number.
That sounds like it is absolutely fixable to me, but more of a matter of not having the will to fix it based on some kind of traditionalism. I've used python, a lot. But it is stuff like this that is just maddeningly broken for no good reason at all that has turned me away from it. So as long as I have any alternative I will avoid python because I've seen way too many accidents on account of stuff like this and many lost nights of debugging only to find out that an easily avoidable issue became - once again - the source of much headscratching.
> a matter of not having the will to fix it based on some kind of traditionalism
Do you know what happens when Python does summon the will to fix obviously broken things? The Python 2->3 migration happens. (Perl 6 didn't manage any better, either.) Now "Python 3 is the brand" and the idea of version 4 can only ever be entertained as a joke.
Yes, good point. Compared to how the Erlang community has handled decades of change Python does not exactly deserve the beauty prize. The lack of forethought - not to be confused with a lot of hot air - on some of these decisions is impressive. I think that the ability to track developments in near realtime is in conflict with that though. If you want your language to be everything to everybody then there will be some broken bones along the way.
If vereioned imports were added to the language versioned library support obviously would have to become part of the language as well.
However it isn't trivial. First problem coming to my mind:
module a importing first somelib>=1.2.0 and then b and b then requiring somelib>1.2.1 and both being available, will it be the same or will I have a mess from combining?
> It's been explained many times before why this is not possible: the library doesn't actually have a version number.
Not possible? Come on.
Almost everyone already uses one of a small handfull of conventional ways to specify it, eg `__version__` attribute. It's long overdue that this be standardized so library versions can reliably be introspected at runtime.
Allowing multiple versions to be installed side-by-side and imported explicitly would be a massive improvement.
I believe the charitable interpretation is that it is not possible without breaking an enormous amount of legacy code. Which does feel close enough to “not possible”.
Some situations could be improved by allowing multiple library versions, but this would introduce new headaches elsewhere. I certainly do not want my program to have N copies of numpy, PyTorch, etc because some intermediate library claims to have just-so dependency tree.
What do you do today to resolve a dependency conflict when an intermediate library has a just-so dependency tree?
The charitable interpretation of this proposed feature is that it would handle this case exactly as well as the current situation, if the situation isn't improved by the feature.
This feature says nothing about the automatic installation of libraries.
This feature is absolutely not about supporting multiple simultaneous versions of a library at runtime.
In the situation you describe, there would have to be a dependency resolution, just like there is when installing the deps for a program today. It would be good enough for me if "first import wins".
> What do you do today to resolve a dependency conflict when an intermediate library has a just-so dependency tree?
When an installer resolves dependency conflicts, the project code isn't running. The installer is free to discover new constraints on the fly, and to backtrack. It is in effect all being done "statically", in the sense of being ahead of the time that any other system cares about it being complete and correct.
Python `import` statements on the other hand execute during the program's runtime, at arbitrary separation, with other code intervening.
> This feature says nothing about the automatic installation of libraries.
It doesn't have to. The runtime problems still occur.
I guess I'll have to reproduce the basic problem description from memory again. If you have modules A and B in your project that require conflicting versions of C, you need a way to load both at runtime. But the standard import mechanism already hard-codes the assumptions that i) imports are cached in a key-value store; ii) the values are singleton and client code absolutely may rely on this for correctness; iii) "C" is enough information for lookup. And the ecosystem is further built around the assumption that iv) this system is documented and stable and can be interacted with in many clever ways for metaprogramming. Changing any of this would be incredibly disruptive.
> This feature is absolutely not about supporting multiple simultaneous versions of a library at runtime.
> and support having multiple simultaneous versions of any Python library installed.
Which would really be the only reason for the feature. For the cases where a single version of the third-party code satisfies the entire codebase, the existing packaging mechanisms all work fine. (Plus they properly distinguish between import names and distribution names.)
> and support having multiple simultaneous versions of any Python library installed.
Installed. Not loaded.
The reason is to do away with virtual environments.
I just want to say `import numpy@2.3.x as np` in my code. If 2.3.2 is installed, it gets loaded as the singleton runtime library. If it's not installed, load the closest numpy available and print a warning to stderr. If a transient dependency in the runtime tree wants an incompatible numpy, tough luck, the best you get is a warning message on stderr.
You already have the A, B, C dependency resolution problem you describe today. And if it's not caught at the time of installing your dependencies, you see the failure at runtime.
You'd have to invent a different way, within existing Python syntax, to communicate the version, but you can do this today with sys.path and sys.meta_path hacks.
But virtual environments are quite simply not a big deal. Installed libraries can be hard-linked and maybe even symlinked between environments and this can be set up very quickly. A virtual environment is defined by the pyvenv.cfg marker file; you don't need to use or even have activation scripts, and you especially don't (generally) need a separate copy of pip for each one, even if you do use pip.
On the flip side, allowing multiple versions of a library in a virtual environment has very little effect on package resolution; it just allows success in cases of conflict, but normally there aren't conflicts (because you're typically making a separate environment for a single "root" package, and it's supposed to be possible to use that package in Python as it actually exists, without hacks). The installer still has to scrounge up metadata (and discover it recursively) and check constraints.
NO!
I don't want my source code filled with this crap.
I don't want to lose multiple hours debugging why something did go wrong because I am using three versions of numpy and seven of torch at the same time and there was a mixup
I should be able to do "python foo.py" and everything should just work. foo.py should define what it wants and python should fetch it and provide it to foo. I should be able to do "pyc foo.py; ./foo" and everything should just work, dependencies balled up and statically included like Rust or Go. Even NodeJS can turn an entire project into one file to execute. That's what a modern language should look and work like.
The moment I see "--this --that" just to run the default version of something you've lost me. This is 2025.
I don't hate it but I don't love it. It sounds like everyone will start writing `lazy` before essentially every single import, with rare exceptions where eager importing is actually needed. That makes Python code visually noisier. And with no plan to ever change the default, the noise will stay forever.
I would have preferred a system where modules opt in to being lazy-loaded, with no extra syntax on the import side. That would simplify things since only large libraries would have to care about laziness. To be fair, in such a design, the interpreter would have to eagerly look up imports on the filesystem to decide whether they should be lazy-loaded. And there are probably other downsides I'm not thinking of.
If everyone starts favoring lazy imports with not much fuss then it means that lazy should have been the default behavior and eager is the keyword we're missing. This isn't the first time Python revisits this paradigm. Many constructs that used to eagerly produce lists in v2 were turned into generators in v3 with next to no problems.
We heard that about types, the walrus, asyncio, dataclasses and so much more. But it didn't happen, if people don't need something (and many don't know it exists or what it does), it's unlikely they use it.
In fact, half of the community basically uses only a modernized set of python 2.4 features and that's one of the beauties of the language. You don't need a lot to be productive, and if you want more, you can optionally reach for it.
It has worked very well for the last 2 decades and it will likely work again.
People said the same about Perl and its “there’s more than one way to do things” ethos, which gained much criticism.
Same is true for C++.
In this specific case, I think a lazy load directive isn’t a bad addition. But one does need to be careful about adding new language features just because you have an active community.
Perhaps people won't use it. But I for one want it to be used, and I will certainly be using it in my own code if the PEP is accepted. Startup time is very important to me, more important than the cost of making the code noisier. I just wish I didn't have to make that tradeoff in the first place.
The PEP includes the ability to enable (or disable) lazy imports globally via a command-line flag or environment variable, in addition to the import syntax.
> I would gladly take a command line flag that I can pass to python that makes all module loading lazy.
oh, you want a "break my libraries" flag? :D
seriously, in theory lazy imports may be "transparent" for common use cases, but I've saw too many modules rely on the side effects of the importing, that I understand why they needed to make this a "double opt in" feature
You can do it today with a few lines of code, although the implementation I show is not particularly robust (since it works by metaprogramming the import system, of course other code could interfere): https://news.ycombinator.com/item?id=45467489
I don't think this makes sense to be on the module side, the caller is the one with the information as to whether the module can or needs to be lazily loaded. There's nothing really for the module being imported to decide, every module can be lazily loaded. Even if it has side effects the caller may want to defer those as well.
I think side-effects are exactly the problem, you can't have the runtime default to lazy-loading all modules without breaking code that e.g. relies on side effects running before thread creation or forking.
I don't follow your reasoning. `pyproject.toml` has nothing to do with what happens at runtime. It's about building and packaging the code. It also doesn't say anything related to the modules that will be imported at runtime. It deals in the names of distributions, which are completely independent of `import` statements.
What I mean is that the `lazy` keyword need not be specified in the source Python files. Instead you can mark off a bunch of modules identified by regex as lazy. Perhaps `pyproject.toml` isn't the best place and it should be an independent python file.
If anyone's interested I've implemented a fairly user friendly lazy import mechanism in the form of context managers (auto_proxy_import/init) at https://pypi.org/project/lazyimp/ that I use fairly heavily. Syntactically it's just wrapping otherwise unmodified import statements in a with block, so tools 'just work' and it can be easily disabled or told to import eagerly for debugging. It's powered primarily by swapping out the frame's f_builtins in a cext (as it needs more power than importlib hooks provide), but has a lame attempt at a threadsafe pure python version, and a super dumb global hook version.
I was skeptical and cautious with it at first but I've since moved large chunks of my codebase to it - it's caused surprisingly few problems (honestly none besides forgetting to handle some import-time registration in some modules) and the speed boost is addictive.
I love the feature but I really dislike using the word lazy as a new language keyword. It just feels off somehow. I think maybe defer might be a better word. It is at least keeps the grammar right because it would be lazily.
lazily import package.foo
vs
defer import package.foo
Also the grammar is super weird for from imports.
lazy from package import foo
vs.
from package defer import foo.
One thing the PEP doesn't really talk about, and that I find very annoying is that many python linters will complain if you don't put all of your imports at the top of the file, so you get lint warnings if you do the most obvious way to implement lazy imports.
And that is actually a problem for more than just performance. In some cases, importing at the top might actually just fail. For example if you need a platform specific library, but only if it is running on that platform.
> The standard library provides the LazyLoader class to solve some of these inefficiency problems. It permits imports at the module level to work mostly like inline imports do.
The use of these sorts of Python import internals is highly non-obvious. The Stack Overflow Q&A I found about it (https://stackoverflow.com/questions/42703908/) doesn't result in an especially nice-looking UX.
So here's a proof of concept in existing Python for getting all imports to be lazy automatically, with no special syntax for the caller:
import sys
import threading # needed for python 3.13, at least at the REPL, because reasons
from importlib.util import LazyLoader # this has to be eagerly imported!
class LazyPathFinder(sys.meta_path[-1]): # <class '_frozen_importlib_external.PathFinder'>
@classmethod
def find_spec(cls, fullname, path=None, target=None):
base = super().find_spec(fullname, path, target)
base.loader = LazyLoader(base.loader)
return base
sys.meta_path[-1] = LazyPathFinder
We've replaced the "meta path finder" (which implements the logic "when the module isn't in sys.modules, look on sys.path for source code and/or bytecode, including bytecode in __pycache__ subfolders, and create a 'spec' for it") with our own wrapper. The "loader" attached to the resulting spec is replaced with an importlib.util.LazyLoader instance, which wraps the base PathFinder's provided loader. When an import statement actually imports the module, the name will actually get bound to a <class 'importlib.util._LazyModule'> instance, rather than an ordinary module. Attempting to access any attribute of this instance will trigger the normal module loading procedure — which even replaces the global name.
Now we can do:
import this # nothing shows up
print(type(this)) # <class 'importlib.util._LazyModule'>
rot13 = this.s # the module is loaded, printing the Zen
print(type(this)) # <class 'module'>
That said, I don't know what the PEP means by "mostly" here.
Can you explain why does "threading" needs to be loaded? The rest seems decently straightforward, but what initialization from threading is required and why only in 3.13+?
I didn't investigate deeply (I just happened to stumble on the issue while figuring out the implementation), but it might be needed in older versions too. The loading process instantiates a `threading.RLock`, but from a source code file that doesn't import that (because it's in a file that has to bootstrap the import process). I'm not entirely sure how `threading` is supposed to get imported normally. The other complicating factor is that I was testing this at the REPL, and 3.13 has a new REPL implementation which might change how those initial libraries are brought in.
This is the wrong syntax, comparable to how "u" strings were the wrong syntax and "b" strings are the right syntax.
They make that, what should be the default, a special case. Soon, every new code will use "lazy". The long term effect of such changes is a verbose language syntax.
They should have had a period where one, if they want lazy imports, has to do "from __future__ import lazy_import". After that period, lazy imports become the default. For old-style immediate imports, introduce a syntax: "import now foo" and "from foo import now bar".
All which authors of old code would have to do is run a provided fix script in the root directory of their code.
I think your assessment of what's "the right/wrong" syntax is fair. But the transition you describe takes a long time, even now that the community has figured out a "deprecation cycle" process that seems satisfactory (i.e. won't lead to another Python 3.0 situation).
> All which authors of old code would have to do is run a provided fix script in the root directory of their code.
As I recall, `lib2to3` didn't do a lot to ease tensions. And `six` is still absurdly popular, mainly thanks to `python-dateutil` still attempting to support 2.7.
Honestly, I don't care for 2 to 3 or six (which, btw, has just 1k stars on GitHub, compare to e.g. 90k stars for FastAPI). Someone who bases their code on 2.7 in 2025 must expect to run into trouble.
If people do not run such upgrade scripts, it must be documented better.
Python is free. In order to also stay elegant, they should say to their users: "We expect from you to run an upgrade script on your code once per Python upgrade"
Agree, they really did their homework, listed edge cases, made practical compromises, chose not to overdo it, reworked it again and again quite a bit and compared it to real life experience.
It's really beautiful work, especially since touching the back bone (the import system) of a language as popular as Python with such a diverse community is super dangerous surgery.
Yeah, I think this is one of the cleanest PEPs to come around in quite a while, at least from the userspace perspective. Interested to see what happens after the traditional syntax bikeshedding ritual has been completed.
Hopefully they learned lessons from why PEP-690 was rejected. I've spent quite a while trying to build this stuff for our codebase and it's never worked well enough to use.
I think they're understating the thread safety risks here. The import is going to wind up happening at a random nondeterministic time, in who knows what thread holding who knows what locks (aside from the importer lock).
Previously, if you had some thread hazardous code at module import time, it was highly likely to only run during the single threaded process startup phase, so it was likely harmless. Lazy loading is going to unearth these errors in the most inconvenient way (as Heisenbugs)
(Function level import can trigger this as well, but the top of a function is at least a slightly more deterministic place for imports to happen, and an explicit line of syntax triggering the import/bug)
We tend to prefer explicit top-level imports specifically because they reveal dependency problems as soon as the program starts, rather than potentially hours or days later when a specific code path is executed.
As a counterpoint, having all the imports automatically deferred would instantly dramatically speed up pip for short tasks.
$ time pip install --disable-pip-version-check
ERROR: You must give at least one requirement to install (see "pip help install")
real 0m0.399s
user 0m0.360s
sys 0m0.041s
Almost all of this time is spent importing (and later unloading) ultimately useless vendored code. From my testing (hacking the wrapper script to output some diagnostics), literally about 500 modules get imported in total (on top of the baseline for a default Python process), including almost a hundred modules related to Requests and its dependencies, even though no web request was necessary for this command.
I think this makes a ton of sense in the very specific narrow use case of python CLI tools. For a web app or other long-lived process, startup time is typically not of extreme concern, and having more simplicity and legibility to the import process seems better.
That's not to say this PEP should not be accepted. One could always apply a no-lazy-imports style rule or disable it via global lazy import control.
There are tons of great CLI tools written in Python, but I would wager that the vast majority of Python code, no matter how you count it, isn't in a CLI tool. Certainly it's a worthy goal to speed them up especially given how frequently things like `pip` are run both by humans who have better things to do than wait on computers, and by automation which does things at giant scale. But most Python project and programmers who are not making CLI tools or other startup-latency sensitive stuff may be better served by the old default import style.
> But most Python project and programmers who are not making CLI tools or other startup-latency sensitive stuff may be better served by the old default import style.
That may well be the case, and Python's development processes would definitely tend to assume the same. That has a lot to do with why PEP 690 was rejected, and why this proposal is opt-in even though many people are concerned that a lot of projects will "opt in everywhere" and create a lot of noise.
Believe it or not, the process is actually very conservative. People complaining about the "churn" caused by deprecations and removals seem not to have any concept of how few suggestions actually get implemented, and how many are rejected (including ones that perennially occur to many new users). A browse through "ideas" forum where new pre-PEP ideas are commonly pitched (https://discuss.python.org/c/ideas/6) gives one the impression of a leisurely stroll through a graveyard.
And many people (including myself) can tell you that you'll often be put through a run-around: if your idea is good and can be implemented, then surely it falls on you to make and publicize (!) a third-party package, to prove the demand and the community support for your specific implementation; but if you somehow get there, now it's trivial for people to install support, so it doesn't need to be in the standard library (cf. Requests).
Right, Not sure why but a lot of code which claude generates also comes with local dependencies vs globally declared import statements. Don't promote that pattern because
- It reduces visibility into a module’s dependencies.
- It increases the risk of introducing circular dependencies later on.
I haven't fully digested the PEP but perhaps there would be a command-line flag or external tool for dependency validation, a bit like how there are external tools for type annotations?
My current team at my current company (see bio if you're really interested), though I should say I'm not authorized to speak on behalf of my employer, so I should really say something more like "I".
Lazy imports are a great way to create runtime errors far into the operation of a long lived service. Yes, it gives the superficial benefit of 'fast startup', but that upside is negated by the downside of not being sure that once something runs it will run to completion due to a failed import much further down the line. It also allows for some interesting edge cases with the items that are going to be imported no longer being what is on the tin at the time the program is started.
That's fine, because this is still a genuine problem in need of a solution. It's not just about startup time for the sake of it (not that this is even a superficial concern - python startup time with large dependencies quickly gets awful). Large projects can have hefty dependencies that not every user will use. And bundling it all for everyone can sometimes be intractable. The work arounds people use already have the sort of problems you're talking about, on top of being diabolical and hacky. Not having to duplicate and hide imports in functions alone would be a big improvement. It's not like it isn't being proposed as an optional language feature.
An automated test mitigates the risk you describe, and is well worth the tradeoff for fast startup.
I don't consider startup time "superficial" at all; I work in a Django monolith where this problem resulted in each and every management command, test invokation, and container reload incurring a 10-15sec penalty because of just a handful of heavy-to-import libraries used by certain tasks/views. Deferring these made a massive difference.
I don't like the idea of introducing a new keyword. We need a backward compatible solution. I feel like Python needs some kind of universal annotation syntax such as in go (comments) or in Rust (macros). New keyword means all parsers, lsps, editors should be updated.
I’m pretty sure there will be new keywords in Python in the future that only solve one thing.
Right now all the imports are getting resolved at runtime example in a code like below
from file1 import function1
When you write this, the entire file1 module is executed right away, which may trigger side effects.
If lazy imports suddenly defer execution, those side effects won’t run until much later (or not at all, if the code path isn’t hit). That shift in timing could easily break existing code that depends on import-time behavior.
To avoid using lazy, this there is also a proposal of adding the modules you want to load lazily to a global `__lazy_modules__` variable.
I was talking about syntax. I'm pretty sure there will be new features that will require a new keyword or syntax given the speed of Python growth. It can be universal, for example same as decorator, but it can be applied anywhere.
from lazy import make_lazy
from package import module @make_lazy @local @nogil
Let's say this syntax gets introduced in Python 3.16. The @nogil feature can be introduced in 3.17. If such code is running in Python 3.16, the @nogil marker will be ignored.
The problem with new keywords is that you have to stick to the newest Python version every time a new keyword is added. Older Python versions will give a syntax error. It's a big problem for libraries. You need to wait for 3-5 years before adding it to a library. There are a lot of people who still use Python 3.8 from 2019.
I think they mean backwards-compatible syntax-wise, rather than actually allowing this feature to be used on existing code. If I’m understanding correctly they would prefer for the Python grammar to stay the same (hence the comment about updating parsers and IDEs).
But I don’t think I really agree, the extensible annotation syntaxes they mention always feel clunky and awkward to me. For a first-party language feature (especially used as often as this will be), I think dedicated syntax seems right.
Async or not in modules is a huge pain in the ass for web-dev, but thankfully a lot of Python still isn't web dev. As a Data Scientist you can live your life peacefully without ever worrying about this. But I can see why web devs like it, even though personally I really don't want to see anymore Javascript sneaking inty my Python. Especially when there is already support for IO bound concurrency elsewhere in the language. If I want to do JS syntax, I'll fucking use JS. And I really don't want to see Python go the C++ route where it just wants to be everything and do everything so that you end up with so many possible approaches to the same problem that two devs can't read each others code anymore.
I'm relatively new to Python - how does one do concurrent IO without async/await?
My main complaint, though, about Python async is - because it is opt-in I never know if I forgot a sync IO call somewhere that will block my worker. JS makes everything async by default and there is effectively no chance of blocking.
Async was a soft keyword for many, many years in order to maintain compat. And before that, asyncio used yield to be even more compatible.
They took a decade to solidify that. At some point, you have to balance evolution and stability. For a language as popular as Python, you can not break the world every week, but you can't stay put for 5 years.
They thought about backward compatibility, and offer an alternative syntax that use no keyword for library that want to activate it yet stay compact with old version. It's already in the spec.
I'm a fan because it's something you can explicitly turn on and off. For my Docker based app, I really want to verify the completeness of imports. Preferably, at build and test time. In fact, most of the time I will likely disable lazy loading outright. But, I would really appreciate a faster loading CLI tool.
However, there is a pattern in python to raise an error if, say, pandas doesn't have an excel library installed, which is fine. In the future, will maintainers opt to include a bunch of unused libraries since they won't negatively impact startup time? (Think pandas including 3-4 excel parsers by default, since it will only be loaded when called). It's a much better UX, but, now if you opt out of lazy loading, your code will take longer to load than without it.
This is needed, but I don't like new keywords. What I would love, for many reasons, is if we could decorate statements. Then things like:
import expensive_module
could be:
@lazy
import expensive_module
or you could do:
@retry(3)
x = failure_prone_call(y)
lazy is needed, but maybe there is a more basic change that could give more power with more organic syntax, and not create a new keyword that is special purpose (and extending an already special purpose keyword)
As a side note, It would be great to have a 'preview' for HN comments. I updated the above because I forgot to add line breaks. Sigh. I bet 'preview' would stop a lot of not well thought out comments too.
It would be interesting if instead you added a syntax whereby a module could declare that it supported lazy importing. Maybe even after running some code with side effects that couldn't be done lazily. For one thing, this would have a much broader performance impact, since it would benefit all users of the library, not just those who explicitly tagged their imports as lazy. For another, it would minimize breakage, since a module author knows best whether, and which parts of, their module can be lazily loaded.
On the other hand, it would create confusion for users of a library when the performance hit of importing a library was delayed to the site of usage. They might not expect, for example, a lag to occur there. I don't think it would cause outright breakage, but people might not like the way it behaved.
I wonder if this proposal suffers an because of Python's extremely generous support period and perhaps the ship has sailed.
- lazy imports are a hugely impactful feature
- lazy imports are already possible without syntax
This means any libraries that get large benefit from lazy imports already use import statements within functions. They can't really use the new feature since 3.14 EoL is _2030_, forever from now. The __lazy_modules__ syntax preserves compatibility only but being eager, not performance - libraries that need lazy imports can't use it until 2030.
This means that the primary target for a long time is CLI authors, which can have a more strict python target and is mentioned many times in the PEP, and libraries that don't have broad Python version support (meaning not just works but works well), which indicates they are probably not major libraries.
Unless the feature gets backported to 2+ versions, it feels not so compelling. But given how it modifies the interpreter to a reasonable degree, I wonder if even any backport is on the table.
In at least the scientific python environment, there's "SPEC0" in which a lot of the de facto core libraries have basically agreed to support the last three versions of Python, no more.
For other libraries they can of course choose as they want, but generally I don't think it's so common for libraries to be as generous with the support length as cpython.
Ugh...I like the idea, but I wish lazy imports were the default. Python allows side effects in the top level though so that would be a breaking change.
Soooo instead now we're going to be in a situation where you're going to be writing "lazy import ..." 99% of the time: unless you're a barbarian, you basically never have side effects at the top level.
Given all the problems people are mentioning, it seems like this proposal is on the wrong side. There should be an easy way for a module to declare itself to be lazy loaded. The module author, not the user, is the one who knows whether lazy loading will break stuff.
# foo.py
def __getattr__(name):
# clean up the lazy loader before loading
# this way it's cleaned up if the implementation doesn't replace it,
# and not scrubbed if it does
global __getattr__
del __getattr__
import sys
self = sys.modules[__name__]
from . import _foo
# "star-import" by adding names that show up in __dir__
self.__dict__.update(**{k: getattr(_foo, k) for k in _foo.__dir__()})
# On future attribute lookups, everything will be in place.
# But this time, we need to delegate to the normal lookup explicitly
return getattr(self, name)
It's also simpler. It could even be a package level thing similar to typed.py marker. I don't want to pepper literally all my modules with loads of explicit lazy keywords.
I like the approach of ES6 where you pull in bindings that are generally lazily resolved. That is IMO the approach that should be the general strategy for Python.
The import/require schism was necessary precisely of all the issues with module side effects the python community is rediscovering now. I suppose they won’t get around a similar solution eventually; but judging from years of discussions in the JS world, this is going to be dragging on for a while.
This seems unnecessary if we can run imports from functions, right? This feels like a layer of indirection and a potential source for confusion. Rather than implicitly importing a library when the variable is first used, why don't you just explicitly do it?
Edit
> A somewhat common way to delay imports is to move the imports into functions (inline imports), but this practice requires more work to implement and maintain, and can be subverted by a single inadvertent top-level import. Additionally, it obfuscates the full set of dependencies for a module. Analysis of the Python standard library shows that approximately 17% of all imports outside tests (nearly 3500 total imports across 730 files) are already placed inside functions or methods specifically to defer their execution. This demonstrates that developers are already manually implementing lazy imports in performance-sensitive code, but doing so requires scattering imports throughout the codebase and makes the full dependency graph harder to understand at a glance.
I think this is a really weak foundation for this language feature. We don't need it.
On the contrary, I think "everyone is already doing this in their own way, so we will offer it as a language feature, which brings these extra benefits" is one of the stronger arguments for introducing a language feature.
I don’t want lazy imports. That’s just makes performance shitty later and harder to debug. It’s a hacky workaround.
What I want is for imports to not suck and be slow. I’ve had projects where it was faster to compile and run C++ than launch and start a Python CLI. It’s so bad.
Can you show me a profile that demonstrate the slowness is because imports are running large amounts of arbitrary Python code? Maybe they shouldn't do that. Maybe libraries should be pushed to not run expensive arbitrary code execution at import time.
This is a great compromise given how much would break if this was the default. Making this the default would be better in the long-run, require taming the abuse of side-effects in imports too, win-win.
If one could dream, modules should have to explicitly declare whether they have side-effects or not, with a marker at the top of the module. Not declaring this and trying anything except declaring a function or class should lead to type-checker errors. Modules declaring this pure-marker then automatically become lazy. Others should require explicit "import_with_side_effects" keyword.
All of this would be impossible today, given how much the Python relies on metaprogramming. Even standard library exposes functions to create classes on the fly like Enum and dataclasses, that are difficult to assert as either pure or impure. With more and more of the ecosystem embracing typed Python, this metaprogramming is reducing into a less dynamic subset. Type checkers and LSPs must have at least some awareness of these modules, without executing them as plain python-code.
This would be a huge deal for Python startup time *if* it was applied to all the standard library packages recursively. Right now importing asyncio brings in half the standard library through transitive imports.
$ time python -c '' # baseline
real 0m0.020s
user 0m0.015s
sys 0m0.005s
$ time python -c 'import sys; old = len(sys.modules); import asyncio; print(len(sys.modules) - old)'
104
real 0m0.076s
user 0m0.067s
sys 0m0.009s
For comparison, with the (seemingly optimized) Numpy included with my system:
$ time python -c 'import sys; old = len(sys.modules); import numpy; print(len(sys.modules) - old)'
185
real 0m0.124s
user 0m0.098s
sys 0m0.026s
Remember mercurial? Me neither. But what I remeber is this article I've read about all the hacks they had to do to achieve reasonable startup time for CLI in python. And the no #1 cause was loading the whole world you don't ever need. As I recall they somehow monkeypatched the interpreter to ignore imports and just remember their existence until they were actually needed, at which point the import happened. So all the dead paths were just skipped.
All my code which uses import probing would fail, such as fallbacks:
try:
import module
except ImportError:
import slow_module as module
Conditional support testing would also break, like having tests which only run if module2 is available:
try:
import module2
except ImportError:
def if_has_module2(f):
return unittest.skip("module2 not available")(f)
else:
def if_has_module2(f):
return f
@if_has_module2
class TestModule2Bindings(....
The proto-PEP also gives an example of using
with suppress_warnings():
import module3
where some global configuration changes only during import.
In general, "import this" and "import antigravity" - anything with import side-effects - would stop working.
Oh, and as the proto-PEP points out, changes to sys.path and others can cause problems because of the delay between time of lazy import and time of resolution.
Then there's code where you do a long computation then make use of a package which might not be present.
You have to opt in with the lazy import keyword no matter what. This pep also prevents lazy import in try catch. I think your concern matters if ‘module’ itself has a lazy import that you want to check exists. With this you now need to be more rigorous in check those sub dependencies.
This can already happen with non top level imports so it is not a necessarily a new issue, but could become more prevalent if there is an overall uptake in this feature for optional dependencies.
>>> import nonexistent_module
Traceback (most recent call last):
File "<python-input-2>", line 1, in <module>
import nonexistent_module
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1322, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1262, in _find_spec
File "<python-input-0>", line 8, in find_spec
base.loader = LazyLoader(base.loader)
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'loader'
The implementation should probably convert that exception back to ImportError for you, but the point is that the absence of an implementation can still be detected eagerly while the actual loading occurs lazily.
Ahh, so you do the find first, and keep that around before loading.
I have bad memories of using a network filesystem where my Python app's startup time was 5 or more seconds because of all the small file lookups for the import were really slow.
I fixed it by importing modules in functions, only when needed, so the time went down to less than a second. (It was even better using a zipimport, but for other reasons we didn't use that option.)
If I understand things correctly, your code would have the same several-second delay as it tries to resolve everything?
Yes, if checking for a file is slow, then checking for a file is slow. If you need to know up front whether the module exists, then you can't get around using the "figure out whether the module exists" machinery up front. And if the definition of "a module exists" includes cases where the module is represented by a file whose existence you have to check for, then there's no getting around that, either.
(Trying to do "fallback" logic with lazily-loaded modules is also susceptible to race conditions, of course. What if someone defines the module before you try to use it?)
Recursively importing other modules is itself a side effect.
In fact, all the code you see in the module is "side effects", in a sense. A `class` body, for example, has to actually run at import time, creating the class object and attaching it as an attribute of the module object. Similarly for functions. Even a simple assignment of a constant actually has to run at module import. And all of these things add up.
Further, if there isn't already cached bytecode available for the module, by default it will be written to disk as part of the import process. That's inarguably a side effect.
Side-effect means you're changing state outside the scope the code is running.
Sure thing you can declare globals variable and run anything on a module file global scope (outside funcs and class body), but even that 'global' scope is just an illusion, and everything declared there, as yourself said, is scoped to the module's namespace
(and you can't leak the 'globals' when importing the module unless you explicity do so 'from foo import *'. Think of python's import as eval but safer because it doesn't leaks the results from the module execution)
So for a module to have side-effect (for me) it would either:
- Change/Create attributes from other modules
- Call some other function that does side-effect (reflection builtins? IO stuff)
Kinda related, I wish there was an easy way to exclude dependencies at pip-install time and mock them at runtime so an import doesn't cause an exception. Basically a way for me to approximate "extras" when the author isn't motivated to do it for me, even though it'd be super brittle.
This sounds doable, actually. You'd want to pre-install (say, from a local wheel) a matching dummy dependency where the metadata claims that it's the right version of whatever package (so the installer will just see that the dependency is "already satisfied" and skip it), but the actual implementation code just exposes a hook to your mocking system.
Doesn't work if version resolution decides to upgrade or downgrade your installed package, so you need to make sure the declared version is satisfactory, too.
We need some kind of unexec revival in Linux so we don't have to resort to crap like this. Maybe CRIU based. Worst case, some Python specific hack. But Python's import system is wacky enough without lazy loading. This sounds like combining the worst of multiple worlds.
> A world in which Python only supported imports behaving in a lazy manner would likely be great...we do not envision the Python langauge transitioning to a world where lazy imports are the default...this concept would add complexity to our ecosystem.
Why can't lazy be the default and instead propose an `eager` syntax? The only argument I can imagine is that there's some API that runs a side effect based on importing, but perhaps making it eager for modules with side effects would be a sufficient temporary fix?
Really excited about this - we've recently been struggling with making imports lazy without completely messing up the code in DeepInverse https://deepinv.github.io/deepinv/
Very unfortunate how from json lazy import dumps would result in backward compatibility issues. It reads much better and makes it easier to search for lazy imports, especially if in the future something else becomes optionally lazy as well.
> The dominant convention in Python code is to place all imports at the beginning of the file. This avoids repetition, makes import dependencies clear and minimizes runtime overhead.
> A somewhat common way to delay imports is to move the imports into functions, but this practice requires more work [and] obfuscates the full set of dependencies for a module.
The first part is just saying the traditions exist because the traditions have always existed. Traditions are allowed to change!
The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Again, traditions are allowed to change!
The point about the import graph being obfuscated would ring more true if Python didn’t already provide lightning fast static analysis tools like ast. If you care about import graphs at the module level then you’re probably already automating everything with ast anyway, at which point you just walk the whole tree looking for imports rather than the top level.
So, really, the whole argument for a new lazy keyword (instead of inlining giant imports where they are needed) is because people like to see import pytorch at the top
of the file, and baulk at seeing it — and will refuse to even look
for it! — anywhere else? Hmmm.
What does seem like a pain in the ass is having to do this kind of repetitive crap (which they mention in their intro):
>The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Again, traditions are allowed to change!
No, they are saying the tradition is there for a reason. Imports at the beginning of the file makes reasoning about the dependencies of a module much easier and faster. I've had to deal with both and I sure as hell know which I'd prefer. Lazy imports by functions is a sometimes necessary evil and it would be very nice if it became unnecessary.
>The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions.
I don't quite understand why they forbid this from being used in functions.
I mean I get why that makes the most sense in most scenarios, but it is not as if the problem of having to choose between declaring dependencies up front or deferring expensive imports until needed does not happen in functions.
Take for instance a function that quickly fails because the arguments are incorrect, it might do a whole bunch of imports that only make sense for that function but which are made immediately obsolete.
It feels like it is forbidden just because someone thought it wasn't a good coding style but to me there is no obvious reason it couldn't work.
Looks good to me. I use a tab-completion trick where the tab-completer tool calls the script I'm about to invoke with special arguments, and the script reflects on itself and responds with possible completions. But because of slow imports, it often takes a while for the completion to respond.
I could, and sometimes do, go through all the imports to figure out which ones are taking a long time to load, but it's a chore.
This is what I thought of too. I really only know python, do other languages not have that issue? In python it does not seem like a "problem" to me - whenever I have seen circular import issues it is because the code is organized poorly. I worry that this feature will lead to devs "fixing" circular import issues by using lazy imports.
Sometimes it's hard to avoid cyclic imports, without blaming the design. Like if a Parent has a Child, and the Child needs to know of the parent. Only way to solve that in python is to put everything in the same file, which also feels like bad deisgn.
I would say in that case, the Parent and Child shouldn't need to know about each other - some kind of handler in a different file should.
Although I guess that doesn't work in all cases, like defining foreign key relationships when using an orm (like sqlalchemy) for example. But in the orm case, the way to get around that is... lazy resolution :^)
Interesting. I find that I always have this problem in any non-trivial python project, and don't find this to be due to poorly organized code. I have only seen this requirement in Python.
IME circular import errors aren't due to poor organization; they're due to an arbitrary restriction Python has.
Even with eager importing there is only a "circular import problem" if you try to import names `from` the modules — as I pointed out a few days ago (https://news.ycombinator.com/item?id=45400783).
I think a feature like this sees best use in short lived programs (where startup time is a disproportionate percentage of total run time) and programs where really fast startup is essential. There are plenty of places where I could imagine taking advantage of this in my code at work immediately, but I share your concern about unpredictability when libraries we use are also making use of it. It wouldn't be fun to have to dive into dependencies to see what needs to be touched to trigger lazy imports at the most convenient time. Unless I am misunderstanding and a normal import of a module means that all of its lazy imports also become non-lazy?
That's a hack that forces you to duplicate and hide imports. The tradition is to specify imports at the top because it's that much better readability wise.
I know/heard there are "some" (which I haven't seen by the way) libraries that depend on import side effects, but the advantage is much bigger.
First of all, the circular import problem will go away, especially on type hints. Although there was a PEP or recent addition to make the annotation not not cause such issue.
Second and most important of all, is the launch time of Python applications. A CLI that uses many different parts of the app has to wait for all the imports to be done.
The second point becomes a lot painful when you have a large application, like a Django project where the auto reload becomes several seconds. Not only auto reload crawls, the testing cycle is slow as well. Every time you want to run test command, it has to wait several seconds. Painful.
So far the solution has been to do the lazy import by importing inside the methods where it's required. That is something, I never got to like to be honest.
Maybe it will be fixed in Python 4, where the JIT uses the type hints as well /s
Top-level code should not be able to fail except in incredibly deterministic ways that are tested during development. Failing fast is not as good as not failing at all. Lazy imports mean the power to avoid importing things that don't need to be imported at all on this run. Good design also cares about performance to some extent. On my machine, asking pip to do literally nothing takes several times as long as creating a new virtual environment --without-pip .
Python really seems like a bad fit for that. So your imports succeed, what now? Do they have all the functions or fields your program needs? Those are still resolved at the last possible moment. If you want to be sure your program actually runs you will have to write and run tests with and without lazy imports.
You… can? I mean in the strictest sense you're technically not importing by file path but if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree. I think as of Python 3.3 the init file is optional so it will do it by default but I can't remember if there are still some cases where it's required. The only thing you can't do is go "up" to a higher directory than the root module.
Also if that doesn't strike your fancy all of the importlib machinery is at your disposal and it's really not very much work to write an import_path() function. It's one of the patterns plug-in systems use and so is stable and expected to be used by end users. No arcane magic required.
> if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree.
`__init__.py` has nothing to do with making this work. It is neither necessary (as of 3.3, yes, you got it right: see https://peps.python.org/pep-0420/) nor sufficient (careless use of sys.path and absolute imports could make it so that the current folder hasn't been imported yet, so you can't even go "up into" it). The folder will already be represented by a module object.
What `__init__.py` does is:
1. Prevents relative imports from also potentially checking in other paths.
2. Provides a space for code that runs before sub-modules, for example to set useful package attributes such as `__all__` (which controls star-imports).
Relative imports have been supported for approximately forever (https://stackoverflow.com/questions/72852). If you mean "by explicitly specifying a path string" (as opposed to a symbolic name), that has also been supported for approximately forever (https://stackoverflow.com/questions/67631). Today, the `importlib` standard library exposes all the steps of the import process — including figuring out where the source code is, checking the `sys.modules` cache etc. — and lets you hook into everything (in particular, you can bypass the entire "finder" process and turn a file path into a dummy "spec" which is fed to a "loader").
The flexibility of this system also entails that you can in effect define a completely new programming language, describe the process of creating Python bytecode from your custom source, and have clients transparently `import` source in the other language as a result. Or you can define an import process that grabs code from the Internet (not that it would be a good idea...).
If you mean "by explicitly specifying a relative path, and having it be interpreted according to the path of the current module's source code", well first you have to consider that the current module isn't required to have source code. But if it does, then generally it will have been loaded with the default mechanism, which means the module object will have a `__file__` attribute with an absolute path, and you just set your path relative to that.
> Relative imports work by relative package path, which is not at all the same.
Exactly. This gives you the flexibility to distribute a complex package across multiple locations.
> Often when you run Python you don't even have a package path.
Any time you successfully import foo.bar, you necessarily have imported foo (because bar is an attribute of that object!), and therefore bar can `from . import` its siblings.
> Using `importlib` is a horrible hack that breaks basically all tooling. You very very obviously are not supposed to do that.
It is exactly as obvious (and true) that you are not "supposed to", in the exact same sense, directly specify where on disk the source code file you want to import is. After all, this constrains the import process to use a source code file. (Similarly if you specify a .pyc directly.) Your relative path doesn't necessarily make any sense after you have packaged and distributed your code and someone else has installed it. It definitely doesn't make any sense if you pack all your modules into a zipapp.
Well not if you want high quality Python code. Pylint and Pyright won't understand it, and those are absolutely critical to writing Python code that works reliably.
Love this. My https://llm.datasette.io/ CLI tool supports plugins, and people were complaining about really slow start times even for commands like "llm --help" - it turned out there were popular plugins that did things like import pytorch at the base level, so the entire startup was blocked on heavy imports.
I ended up adding a note to the plugin author docs suggesting lazy loading inside of functions - https://llm.datasette.io/en/stable/plugins/advanced-model-pl... - but having a core Python language feature for this would be really nice.
You can implement this from your tool today: https://news.ycombinator.com/item?id=45467489
Note that this is global to the entire process, so for example if you make an import of Numpy lazy this way, then so are the imports of all the sub-modules. Meaning that large parts of Numpy might not be imported at all if they aren't needed, but pauses for importing individual modules might be distributed unpredictably across the runtime.
Edit: from further experimentation, it appears that if the source does something like `import foo.bar.baz` then `foo` and `foo.bar` will still be eagerly loaded, and only `foo.bar.baz` itself is deferred. This might be part of what the PEP meant by "mostly". But it might also be possible to improve my implementation to fix that.
Note the PEP does have a FAQ entry that mentions reasons they believe this proposed solution might be preferable to LazyLoader
https://pep-previews--4622.org.readthedocs.build/pep-0810/#f...
Q: Why not use importlib.util.LazyLoader instead?
A: LazyLoader has significant limitations:
Requires verbose setup code for each lazy import.
Has ongoing performance overhead on every attribute access.
Doesn’t work well with from ... import statements.
Less clear and standard than dedicated syntax.
Thanks for pointing it out!
> Has ongoing performance overhead on every attribute access.
I would have expected so, but in my testing it seems like the lazy load does some kind of magic to replace the proxy with the real thing. I haven't properly dug into it, though. It appears this point is removed in the live version (https://peps.python.org/pep-0810).
> Doesn’t work well with from ... import statements.
Hmm. The PEP doesn't seem to explain how reification works in this case. Per the above it's a solved problem for modules; I guess for the from-imports it could be made to work essentially the same way. Presumably this involves the proxy holding a reference to the namespace where the import occurred. That probably has a lot to do with restricting the syntax to top level. (Which is the opposite of how we've seen soft keywords used before!)
> Requires verbose setup code for each lazy import.
> Less clear and standard than dedicated syntax.
If you want to use it in a fine-grained way, then sure.
Is it another potential solution (until PEP 810 is accepted) to override the NameError exception, decide if it was triggered by an unloaded package from a list, and then running again that line of code? I understand the inefficiency of this solution (e.g. the same line could trigger NameError several times and you need to run it again until all modules are loaded) but this is a good brainstorming thread.
That sounds very unpleasant. However nicely you wrapped it up, you'd still be referring to the code for that process everywhere that the NameError could occur.
It's not only in the case of plugins.
If a tool has different capabilities that use different imports, why load all of them if only a subset is required?
As a simple example, a tool that can generate output in various formats (e.g., json, csv, xml, ...) should only import the appropriate modules to handle the output format after having determined which ones will be used in this invocations.
I think really the problem is that packages like pytorch take so long to import. In my work I've tried a few packages (not AI stuff) that do a lot of work on import. It's actually quite detrimental because I have to setup environment variables to pass things that should be arguments of a setup function in. All things considered a python module shouldn't take any noticeable time to import
Parse the command line and do things like "--help" without doing the imports.
Only do imports when you know you need them -- or as an easy approximation, only if the easy command line options have been handled and there's still something to do.
In the llm project, plugins can modify the command line arguments, so it's not that simple.
Yea, that's the core problem here: plugins can add new CLI subcommands, which means they all need to be loaded on startup.
https://llm.datasette.io/en/stable/plugins/plugin-hooks.html...
Just FWIW, a trick that I'm planning to use for PAPER: first I make separate actual commands — `paper-foo`, `paper-bar` etc. that are each implemented as separate top-level scripts that can import only what they need. Later, the implementation of `paper foo` has the main `paper` script dynamically look up `paper-foo`. (Even a `subprocess.call` would work there but I'd like to avoid that overhead)
Could you cache the help doc after first full loaded run and only regenerate when new plugins are added / updated?
It's not just help - the plugins need to be imported so the root level CLI tool knows what to do if you type "llm subcommand ..." where that subcommand is defined by a plugin.
In that case the CLI only needs to import the plugin that defined that sub command, not all plugins?
It doesn't know which plugin defines a subcommand until it imports the plugin's module.
I'm happy with the solution I have now, which is to encourage plugin authors not to import PyTorch or other heavy dependencies at the root level of their plugin code.
> It doesn't know which plugin defines a subcommand until it imports the plugin's module.
That might be considered a design mistake -- one that should be easy to migrate away from.
You won't need to do anything, of course, if the lazy import becomes available on common Python installs some day in the future. That might take years, though.
Or require plugins to be competently written.
Bad performing third party plugins are user error.
Well yes, or you can just use the `lazy` keyword, when it makes it into core.
You can “just” use a feature which does not exist yet? How is that something you “just” do?
I think Guido's time machine will need some serious overclocking to handle that one!
[flagged]
Lazy imports have been proposed before, and were rejected most recently back in 2022: https://discuss.python.org/t/pep-690-lazy-imports-again/1966.... If I recall correctly, lazy imports are a feature supported in Cinder, Meta's version of CPython, and the PEP was driven by folks that worked on Cinder. Last time, a lot of the discussion centered around questions like: Should this be opt-in or opt-out? At what level? Should it be a build-flag for CPython itself? Etc. The linked post suggests that the Steering Council ultimately rejected it because of the complexity it would introduce to have two divergent "modes" of importing.
I hope this proposal succeeds. I would love to use this feature.
Especially since it is opt in, with various level of granularity, and a global off switch. Very well constructed spec given the constraints.
I also hope this proposal succeeds, but I'm not optimistic. This will break tons of code and introduce a slew of footguns. Import statements fundamentally have side effects, and when and how these side effects are applied will cause mysterious breakages that will keep people up for many nights.
This is not fearmongering. There is a reason why the only flavor of Python with lazy imports comes from Meta, which is one of the most well-resourced companies in the world.
Too many people in this thread hold the view of "importing {pandas, numpy, my weird module that is more tangled than an eight-player game of Twister} takes too long and I will gladly support anything that makes them faster". I would be willing to bet a large sum of money that most people who hold this opinion are unable to describe how Python's import system works, let alone describe how to implement lazy imports.
PEP 690 describes a number of drawbacks. For example, lazy imports break code that uses decorators to add functions to a central registry. This behavior is crucial for Dash, a popular library for building frontends that has been around for more than a decade. At import-time, Dash uses decorators to bind a JavaScript-based interface to callbacks written in Python. If these imports were made lazy, Dash would break. Frontends used by thousands, if not millions of people, would immediately become unresponsive.
You may cry, "But lazy imports are opt-in! Developers can choose to opt-out of lazy imports if it doesn't work for them." What if these imports were transitive? What if our frontend needed to be completely initialized before starting a critical process, else it would cause a production outage? What if you were a maintainer of a library that was used by millions of people? How could you be sure that adding lazy imports wouldn't break any code downstream? Many people made this argument for type hints, which is sensible because type hints have no effect on runtime behavior*. This is not true for lazy imports; import statements exist in essentially every nontrivial Python program, and changing them to be lazy will fundamentally alter runtime behavior.
This is before we even get to the rest of the issues the PEP describes, which are even weirder and crazier than this. This is a far more difficult undertaking than many people realize.
---
* You can make a program change its behavior based on type annotations, but you'd need to explicitly call into typing APIs to do this. Discussion about this is beyond the scope of this post.
Product manager for Dash here At Plotly we're actually pretty excited about the potential for lazy loaded imports, as it could help out a lot with the import performance of Plotly.py.
As this comment mentions Dash apps would not support lazy loaded imports until the underlying Dash library changes how it loads in callbacks and component libraries (the two features which would be most impacted here), but that doesn't mean there's no path to success. We've been discussing some ways we could resolve this internally and if this PEP is accepted we'd certainly go further to see if we can fully support lazy loaded imports (of both the Dash library itself/Dash component libraries and for relative imports in Dash apps).
They are not entitled to hold the opinion that their imports takes too long, if they dont know the inner workings of pythons import system? Do you listen to yourself?
That’s not what I said. Nobody is “entitled” or “not entitled” to hold certain opinions. Please reread my original comment carefully.
Nothing wrong with that statement.
Right now in python, you can move import statement inside a function. Lazy imports at top level are not needed. All lazy imports do is make you think less about what you are writing. If you like that, then just vibe code all of your stuff, and leave the language spec alone.
This is a new syntax, so it is opt-in. The new syntax can be conceived as syntax sugar that lets you rewrite
as Ie, with lazy, the import happens at the site of usage. Since clearly this is code that could already be written, it only breaks things in the sense that someone could already write broken code. Since it is opt in, if using it breaks some code, then people will notice that and choose not to rewrite that code using it.This is already addressed in my comment above.
> This will break tons of code
I don't see how. It adds a new, entirely optional syntax using a soft keyword. The semantics of existing code do not change. Yes, yes, you anticipated the objection:
> What if these imports were transitive? ... How could you be sure that adding lazy imports wouldn't break any code downstream?
I would need to see concrete examples of how this would be a realistic risk in principle. (My gut reaction is that top-level code in libraries shouldn't be doing the kinds of things that would be problematic here, in the first place. In my experience, the main thing they do at top level is just eagerly importing everything else for convenience, or to establish compatibility aliases.)
But if it were, clearly that's a breaking change, and the library bumps the major version and clients do their usual dependency version management. As you note, type hints work similarly. And "explicitly calling into typing APIs" is more common than you might think; https://pypistats.org/packages/pydantic exists pretty much to do exactly this. It didn't cause major problems.
> Import statements fundamentally have side effects, and when and how these side effects are applied will cause mysterious breakages that will keep people up for many nights.
They do have side effects that can be arbitrarily complex. But someone who opts in to changing import timing and encounters a difficult bug can just roll back the changes. It shouldn't cause extended debugging sessions unless someone really needs the benefits of the deferral. And people in that situation will have been hand-rolling their own workarounds anyway.
> Too many people in this thread hold the view of "importing {pandas, numpy, my weird module that is more tangled than an eight-player game of Twister} takes too long and I will gladly support anything that makes them faster".
I don't think they're under the impression that this necessarily makes things faster. Maybe I haven't seen the same comments you have.
Deferring imports absolutely would allow, for example, pip to do trivial tasks faster — because it could avoid importing unnecessary things at all. As things currently stand, a huge fraction of the vendored codebase will get imported pretty much no matter what. It's analogous to tree shaking, but implicitly, at runtime and without actually removing code.
Yes, this could be deferred to explicitly chosen times to get more or less the same benefit. It would also be more work.
I would suggest reading this blog post from Meta, which should resolve your concerns: https://engineering.fb.com/2024/01/18/developer-tools/lazy-i...
Regarding risks in practice:
> Libraries such as PyTorch, Numba, NumPy, and SciPy, among others, did not seamlessly align with the deferred module loading approach. These libraries often rely on import side effects and other patterns that do not play well with Lazy Imports. The order in which Python imports could change or be postponed, often led to side effects failing to register classes, functions, and operations correctly. This required painstaking troubleshooting to identify and address import cycles and discrepancies.
This isn't precisely the scenario I described above, but it is a concrete example of how deferred imports can cause issues that are difficult to debug.
Regarding performance benefits:
> At Meta, the quest for faster model training has yielded an exciting milestone: the adoption of Lazy Imports and the Python Cinder runtime. ... we’ve been able to significantly improve our model training times, as well as our overall developer experience (DevX) by adopting Lazy Imports and the Python Cinder runtime.
Some of these worries make sense, but wouldn’t it be relatively trivial to pass a flag to the interpreter or something similar in order to force all imports to evaluate, as in the current behavior? But to be a bit cheeky if some of these issues cause serious production outages for you it might be time to consider moving on from a scripting language altogether.
The issue is that some imports can be made lazy and some cannot. A binaristic all-or-nothing approach does not address the issue. (I also think that there is zero basis to claim that adding such a flag is trivial, since there’s no reference implementation of this flavor of lazy imports.)
What if we have a program where one feature works only when lazy imports are enabled and one feature only when lazy imports are disabled?
This is not a contrived concern. Let’s say I’m a maintainer of an open-source library and I choose to use lazy imports in my library. Because I’m volunteering my time, I don’t test whether my code works with eager imports.
Now, let’s say someone comes and builds an application on top of this library. It doesn’t work with lazy imports for some unknown reason. If they reach for a “force all imports” flag, their application might break in another mysterious way because the code they depend on is not built to work with eager imports. And even if my dependency doesn’t break, what about all the other packages the application may depend on?
The only solution here would be for the maintainer to ensure that their code works with both lazy and eager imports. However, this imposes a high maintenance cost and is part of the reason why PEP 690 was rejected. (And if your proposed solution was “don’t use libraries made by random strangers on the Internet”, boy do I have news for you...)
My point is that many things _will_ break if migrated to lazy imports. Whether they should have been written in Python in the first place is a separate question that isn’t relevant to this discussion.
Maybe the package that requires lazy can somehow declare that requirement, so another package that tries to force not lazy will fail early and realize it needs to replace this dependency with something compatible or change its ways. It definitely adds complexity, though.
Or check at runtime if it's running with the lazy import feature active. Then instead of breaking in mysterious ways in production it would crash on startup, during development.
Theoretically the implementation may use the approach "as lazy as possible". Traverse lazy imports until you encounter a regular one. I doubt it will make much difference, but at least it gives an option.
Just curious. What changed?
From merely browsing through a few comments, people have mostly positive opinions regarding this proposal. Then why did it fail many times, but not this time? What drives the success behind this PEP?
Oof. I wish they could support version imports
and support having multiple simultaneous versions of any Python library installed. End this conda/virtualenv/docker/bazel/[pick your poison] messIt's been explained many times before why this is not possible: the library doesn't actually have a version number. The distribution of source code on PyPI has a version number, but the name of this is not connected to the name of any module or package you import in the source code. The distribution can validly define zero or more modules (packages are a subset of modules, represented using the same type in the Python type system).
You got three other responses before me all pointing at uv. They are all wrong, because uv did not introduce this functionality to the Python ecosystem. It is a standard defined by https://peps.python.org/pep-0723/, implemented by multiple other tools, notably pipx.
> It's been explained many times before why this is not possible: the library doesn't actually have a version number. The distribution of source code on PyPI has a version number, but the name of this is not connected to the name of any module or package you import in the source code.
You're making the common mistake of conflating how things currently work with how things could work if the responsible group agrees to change how things work. Something being the way it is right now is not the same as something else being "not possible".
No, changing this breaks the world. A huge fraction of PyPI becomes completely invalid overnight, and the rest fails the expected version checks. Not to mention that the language is fundamentally designed around the expectation that modules are singleton. I've written about this at length before but I can't easily find it right now (I have way too many bookmarks and not nearly enough idea how to organize them).
Yes, you absolutely can create a language that has syntax otherwise identical to Python (or at least damn close) which implements a feature like this. No, you cannot just replace Python with it. If the Python ecosystem just accepted that clearly better things were clearly better, and started using them promptly, we wouldn't have https://pypi.org/project/six/ making it onto https://pypistats.org/top (see also https://sethmlarson.dev/winning-a-bet-about-six-the-python-2...).
The hard part is making the change. Adding an escape hatch so older code still works is easy in comparison.
Nobody is claiming this is a trivial problem to solve but its also not an impossible problem. Other languages have managed to figure out how to achieve this and still maintain backwards compatibility.
You're welcome to bring a concrete proposal to, e.g., https://discuss.python.org/c/ideas/6 , or ask around the core devs to find a PEP sponsor.
Note that you will be expected to have familiarized yourself generally with previous failed proposals of this sort, and proactively considered all the reasonably obvious corner cases.
Tbh you’re just reinforcing mine and others point there that the issue isn’t a technical one.
"Talk is cheap. Show me the code."
I’m not going to spend an entire weekend drafting a proposal instead of spending time with my kids, just to win “internet points”.
If you want examples then just look at one of the other languages that have implemented compiler / runtime dependency version checks.
Even Go has better dependency resolution than Python, and Go is often the HN poster child for how not to do things.
The crux of the matter is this is a solvable problem. The real issue isn’t that it’s technically impossible, is that it’s not annoying enough of a day to day problem for people who are in a position to influence this change. I’m not that person and don’t aspire to be that person (I have plenty of other projects on my plate as it is)
In spite of the 'You're welcome to bring' this does not actually sound like an encouragement but more of a veiled statement that some non-technical reason will be found to shoot down the proposal if it were to be made so you might as well not bother.
It's an allusion to the fact that there is a very long history establishing that the problem is not as simple as it sounds, even if you get past the most basic issues, and it's hard to explain it all in a single coherent post.
Nobody suggested it was easy. We were just arguing against your claim that it’s impossible
Saying something is possible isn’t the same as saying something is easy.
If you were to have said “it’s a different problem to solve because of…” then you wouldn’t have had any pushback. But you didn’t. You said it “this is not possible”. And that’s the part that people were disputing.
No, the point is that most people in this thread do not appreciate the complexity of implementing lazy imports. If you disagree, your energy is better spent talking to a CPython core developer about implementation details of making baseless assertions from an ivory tower.
There are many people here who think enabling lazy imports is as simple as flipping a light switch. They have no idea what they're talking about.
You’ve misread the discussion
This thread was a tangent from lazy imports.
And actually people do appreciate the complexities of changes like this. We were responding to a specific comment that that said “it’s impossible”. Saying something is “possible” isn’t the same as saying “it’s easy”.
Reading comprehension failure on your part.
> Not to mention that the language is fundamentally designed around the expectation that modules are singleton.
Modules being singletons is not a problem in itself I think? This could work like having two versions of the same library in two modules named like library_1_23 and library_1_24. In my program I could hypothetically have imports like `import library_1_23 as library` in one file, and `import library_1_24 as library` in another file. Both versions would be singletons. Then writing `import library==1.23` could be working like syntax sugar for `import library_1_23 as library`.
Of course, having two different versions of a library running in the same program could be a nightmare, so all of that may not be a good idea at all, but maybe not because of module singletons.
I know I'm missing something but wouldn't it be possible to just throw an import error when that happens? Would it even break anything? If I try:
import numpy==2.1
And let's say numpy didn't expose a version number in a standard (which could be agreed upon in a PEP) field, then it would just throw an import exception. It wouldn't break any old code. And only packages with that explicit field would support the pinned version import.
And it wouldn't involve trying to extract and parse versions from older packages with some super spotty heuristics.
But it would make new code impossible to use with older versions of python, and older packages, but that's already the case.
Maybe the issue is with module name spacing?
> And let's say numpy didn't expose a version number in a standard (which could be agreed upon in a PEP) field, then it would just throw an import exception. It wouldn't break any old code. And only packages with that explicit field would support the pinned version import.
Yes, this part actually is as simple as you imagine. But that means in practical terms that you can't use the feature until at best the next release of Numpy. If you want to say for example that you need at least version 2 (breaking changes, after all), well, there are already 18 published packages that meet that requirement but are unable to communicate that in the new syntax. This can to my understanding be fixed with post releases, but it's a serious burden for maintainers and most projects are just not going to do that sort of thing (it bloats PyPI, too).
And more importantly, that's only one of many problems that need to be solved. And by far the simplest of them.
You could absolutely have this be part of the language in any regard. The question then becomes how does one implement it in a reasonable way. I think every package should have a __version__ property you should be able to call, then you could have versioned imports.
In fact there's already many packages already defining __version__ at a package level.
https://packaging.python.org/en/latest/discussions/versionin...
Edit: What they are solving with UV is at the moment of standing up an environment, but you're more concerned about code-level protection, where are they're more concerned about environment setup protection for versioning.
> In fact there's already many packages already defining __version__ at a package level.
This only helps for those that do, and it hasn't been any kind of standard the entire time. But more importantly, that helps only the tiniest possible bit with resolving the "import a specific version" syntax. All it solves is letting the file-based import system know whether it found the right folder for the requested (or worse: "a compatible") version of the importable package. It doesn't solve finding the right one if this one is wrong; it doesn't determine how the different versions of the same package are positioned relative to each other in the environment (so that "finding the right one" can work properly); it doesn't solve provisioning the right version. And most importantly, it doesn't solve what happens when there are multiple requests for different versions of the same module at runtime, which incidentally could happen arbitrarily far apart in time, and also the semantics of the code may depend on the same object being used to represent the module in both places.
> It's been explained many times before why this is not possible: the library doesn't actually have a version number.
That sounds like it is absolutely fixable to me, but more of a matter of not having the will to fix it based on some kind of traditionalism. I've used python, a lot. But it is stuff like this that is just maddeningly broken for no good reason at all that has turned me away from it. So as long as I have any alternative I will avoid python because I've seen way too many accidents on account of stuff like this and many lost nights of debugging only to find out that an easily avoidable issue became - once again - the source of much headscratching.
> a matter of not having the will to fix it based on some kind of traditionalism
Do you know what happens when Python does summon the will to fix obviously broken things? The Python 2->3 migration happens. (Perl 6 didn't manage any better, either.) Now "Python 3 is the brand" and the idea of version 4 can only ever be entertained as a joke.
Yes, good point. Compared to how the Erlang community has handled decades of change Python does not exactly deserve the beauty prize. The lack of forethought - not to be confused with a lot of hot air - on some of these decisions is impressive. I think that the ability to track developments in near realtime is in conflict with that though. If you want your language to be everything to everybody then there will be some broken bones along the way.
If vereioned imports were added to the language versioned library support obviously would have to become part of the language as well.
However it isn't trivial. First problem coming to my mind:
module a importing first somelib>=1.2.0 and then b and b then requiring somelib>1.2.1 and both being available, will it be the same or will I have a mess from combining?
> It's been explained many times before why this is not possible: the library doesn't actually have a version number.
Not possible? Come on.
Almost everyone already uses one of a small handfull of conventional ways to specify it, eg `__version__` attribute. It's long overdue that this be standardized so library versions can reliably be introspected at runtime.
Allowing multiple versions to be installed side-by-side and imported explicitly would be a massive improvement.
I believe the charitable interpretation is that it is not possible without breaking an enormous amount of legacy code. Which does feel close enough to “not possible”.
Some situations could be improved by allowing multiple library versions, but this would introduce new headaches elsewhere. I certainly do not want my program to have N copies of numpy, PyTorch, etc because some intermediate library claims to have just-so dependency tree.
What do you do today to resolve a dependency conflict when an intermediate library has a just-so dependency tree?
The charitable interpretation of this proposed feature is that it would handle this case exactly as well as the current situation, if the situation isn't improved by the feature.
This feature says nothing about the automatic installation of libraries.
This feature is absolutely not about supporting multiple simultaneous versions of a library at runtime.
In the situation you describe, there would have to be a dependency resolution, just like there is when installing the deps for a program today. It would be good enough for me if "first import wins".
> What do you do today to resolve a dependency conflict when an intermediate library has a just-so dependency tree?
When an installer resolves dependency conflicts, the project code isn't running. The installer is free to discover new constraints on the fly, and to backtrack. It is in effect all being done "statically", in the sense of being ahead of the time that any other system cares about it being complete and correct.
Python `import` statements on the other hand execute during the program's runtime, at arbitrary separation, with other code intervening.
> This feature says nothing about the automatic installation of libraries.
It doesn't have to. The runtime problems still occur.
I guess I'll have to reproduce the basic problem description from memory again. If you have modules A and B in your project that require conflicting versions of C, you need a way to load both at runtime. But the standard import mechanism already hard-codes the assumptions that i) imports are cached in a key-value store; ii) the values are singleton and client code absolutely may rely on this for correctness; iii) "C" is enough information for lookup. And the ecosystem is further built around the assumption that iv) this system is documented and stable and can be interacted with in many clever ways for metaprogramming. Changing any of this would be incredibly disruptive.
> This feature is absolutely not about supporting multiple simultaneous versions of a library at runtime.
You say that, but you aren't the one who proposed it. And https://news.ycombinator.com/item?id=45467350 says explicitly:
> and support having multiple simultaneous versions of any Python library installed.
Which would really be the only reason for the feature. For the cases where a single version of the third-party code satisfies the entire codebase, the existing packaging mechanisms all work fine. (Plus they properly distinguish between import names and distribution names.)
> and support having multiple simultaneous versions of any Python library installed.
Installed. Not loaded.
The reason is to do away with virtual environments.
I just want to say `import numpy@2.3.x as np` in my code. If 2.3.2 is installed, it gets loaded as the singleton runtime library. If it's not installed, load the closest numpy available and print a warning to stderr. If a transient dependency in the runtime tree wants an incompatible numpy, tough luck, the best you get is a warning message on stderr.
You already have the A, B, C dependency resolution problem you describe today. And if it's not caught at the time of installing your dependencies, you see the failure at runtime.
You'd have to invent a different way, within existing Python syntax, to communicate the version, but you can do this today with sys.path and sys.meta_path hacks.
But virtual environments are quite simply not a big deal. Installed libraries can be hard-linked and maybe even symlinked between environments and this can be set up very quickly. A virtual environment is defined by the pyvenv.cfg marker file; you don't need to use or even have activation scripts, and you especially don't (generally) need a separate copy of pip for each one, even if you do use pip.
On the flip side, allowing multiple versions of a library in a virtual environment has very little effect on package resolution; it just allows success in cases of conflict, but normally there aren't conflicts (because you're typically making a separate environment for a single "root" package, and it's supposed to be possible to use that package in Python as it actually exists, without hacks). The installer still has to scrounge up metadata (and discover it recursively) and check constraints.
You could do that with uv.
How would version imports be handled across the codebase? Also, what do you gain with those over PEP 723 – Inline script metadata? https://packaging.python.org/en/latest/specifications/inline...
Oof. This feature request has nothing to do with lazy imports. It’s also solved far more cleanly with inline script metadata.
NO! I don't want my source code filled with this crap.
I don't want to lose multiple hours debugging why something did go wrong because I am using three versions of numpy and seven of torch at the same time and there was a mixup
Really what is the headache with virtual environments? They’ve been solved. Use UV or python’s built in venv creator and you’re good to go.
uv venv —seed —python=3.12 && source .venv/bin/activate && pip3 install requests && …
That's messy.
I should be able to do "python foo.py" and everything should just work. foo.py should define what it wants and python should fetch it and provide it to foo. I should be able to do "pyc foo.py; ./foo" and everything should just work, dependencies balled up and statically included like Rust or Go. Even NodeJS can turn an entire project into one file to execute. That's what a modern language should look and work like.
The moment I see "--this --that" just to run the default version of something you've lost me. This is 2025.
You mean this?
#!/usr/bin/env -S uv run --script # # /// script # requires-python = ">=3.12" # dependencies = ["httpx"] # ///
import httpx
print(httpx.get("https://example.com"))
https://docs.astral.sh/uv/guides/scripts/#improving-reproduc...
There are also projects py2exe pyinstaller iirc and others that try to get the whole static binary thing going.
You’re trying imo to make Python into Golang and if you’re wanting to do that just use Golang. That seems like a far better use of your time.
The mess has ended thanks to uv.
uv is good.
I don't hate it but I don't love it. It sounds like everyone will start writing `lazy` before essentially every single import, with rare exceptions where eager importing is actually needed. That makes Python code visually noisier. And with no plan to ever change the default, the noise will stay forever.
I would have preferred a system where modules opt in to being lazy-loaded, with no extra syntax on the import side. That would simplify things since only large libraries would have to care about laziness. To be fair, in such a design, the interpreter would have to eagerly look up imports on the filesystem to decide whether they should be lazy-loaded. And there are probably other downsides I'm not thinking of.
If everyone starts favoring lazy imports with not much fuss then it means that lazy should have been the default behavior and eager is the keyword we're missing. This isn't the first time Python revisits this paradigm. Many constructs that used to eagerly produce lists in v2 were turned into generators in v3 with next to no problems.
Yes, there were totally no problems in the transition from 2 to 3 :)
The paradigm I'm alluding to is eager to lazy, not 2 to 3. E.g. dict.items() in v3 works like dict.iteritems() in v2.
We heard that about types, the walrus, asyncio, dataclasses and so much more. But it didn't happen, if people don't need something (and many don't know it exists or what it does), it's unlikely they use it.
In fact, half of the community basically uses only a modernized set of python 2.4 features and that's one of the beauties of the language. You don't need a lot to be productive, and if you want more, you can optionally reach for it.
It has worked very well for the last 2 decades and it will likely work again.
People said the same about Perl and its “there’s more than one way to do things” ethos, which gained much criticism.
Same is true for C++.
In this specific case, I think a lazy load directive isn’t a bad addition. But one does need to be careful about adding new language features just because you have an active community.
Perhaps people won't use it. But I for one want it to be used, and I will certainly be using it in my own code if the PEP is accepted. Startup time is very important to me, more important than the cost of making the code noisier. I just wish I didn't have to make that tradeoff in the first place.
I would gladly take a command line flag that I can pass to python that makes all module loading lazy.
Unless you are writing scripts or very simple stuff running side effects when modules are loaded should be avoided at all cost anyway.
The PEP includes the ability to enable (or disable) lazy imports globally via a command-line flag or environment variable, in addition to the import syntax.
> I would gladly take a command line flag that I can pass to python that makes all module loading lazy.
oh, you want a "break my libraries" flag? :D
seriously, in theory lazy imports may be "transparent" for common use cases, but I've saw too many modules rely on the side effects of the importing, that I understand why they needed to make this a "double opt in" feature
That's already part of the PIP. There is a flag to enable lazy imports for all possible imports.
You can do it today with a few lines of code, although the implementation I show is not particularly robust (since it works by metaprogramming the import system, of course other code could interfere): https://news.ycombinator.com/item?id=45467489
I don't think this makes sense to be on the module side, the caller is the one with the information as to whether the module can or needs to be lazily loaded. There's nothing really for the module being imported to decide, every module can be lazily loaded. Even if it has side effects the caller may want to defer those as well.
I think side-effects are exactly the problem, you can't have the runtime default to lazy-loading all modules without breaking code that e.g. relies on side effects running before thread creation or forking.
> Even if it has side effects the caller may want to defer those as well.
But that's rare, and could be handled with existing workarounds.
Normally, a module needs to be eagerly imported if and only if it has side effects.
Wish pyproject.toml was enhanced to specify lazy loading via regexs.
I don't follow your reasoning. `pyproject.toml` has nothing to do with what happens at runtime. It's about building and packaging the code. It also doesn't say anything related to the modules that will be imported at runtime. It deals in the names of distributions, which are completely independent of `import` statements.
What I mean is that the `lazy` keyword need not be specified in the source Python files. Instead you can mark off a bunch of modules identified by regex as lazy. Perhaps `pyproject.toml` isn't the best place and it should be an independent python file.
The issue is that Python code may validly stand apart from `pyproject.toml` or any other metadata.
If anyone's interested I've implemented a fairly user friendly lazy import mechanism in the form of context managers (auto_proxy_import/init) at https://pypi.org/project/lazyimp/ that I use fairly heavily. Syntactically it's just wrapping otherwise unmodified import statements in a with block, so tools 'just work' and it can be easily disabled or told to import eagerly for debugging. It's powered primarily by swapping out the frame's f_builtins in a cext (as it needs more power than importlib hooks provide), but has a lame attempt at a threadsafe pure python version, and a super dumb global hook version.
I was skeptical and cautious with it at first but I've since moved large chunks of my codebase to it - it's caused surprisingly few problems (honestly none besides forgetting to handle some import-time registration in some modules) and the speed boost is addictive.
looks very interesting! i might use this for some of my projects as well
I love the feature but I really dislike using the word lazy as a new language keyword. It just feels off somehow. I think maybe defer might be a better word. It is at least keeps the grammar right because it would be lazily.
Also the grammar is super weird for from imports.second this.
Just might as well add `defer` keyword like Golang.
One thing the PEP doesn't really talk about, and that I find very annoying is that many python linters will complain if you don't put all of your imports at the top of the file, so you get lint warnings if you do the most obvious way to implement lazy imports.
And that is actually a problem for more than just performance. In some cases, importing at the top might actually just fail. For example if you need a platform specific library, but only if it is running on that platform.
I don't think there is any solution for that but "fix your broken linter".
It isn't just one though. Every linter I've used has warned about that.
Probably because PEP 8 says
> Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants
Ruff doesn't do this, and in fact even lets you specify modules that _must_ not be imported at the top level (banned-module-level-imports = [...])
I banished the worst/heaviest libraries to this list at my workplace and it's been really helpful at keeping startup times from regressing.
It is annoying, but most linters will accept a `#noqa E402` comment to ignore it
>most linters will accept
So, they agreed on a common system of linting error codes? Is that documented somewhere?
> The standard library provides the LazyLoader class to solve some of these inefficiency problems. It permits imports at the module level to work mostly like inline imports do.
The use of these sorts of Python import internals is highly non-obvious. The Stack Overflow Q&A I found about it (https://stackoverflow.com/questions/42703908/) doesn't result in an especially nice-looking UX.
So here's a proof of concept in existing Python for getting all imports to be lazy automatically, with no special syntax for the caller:
We've replaced the "meta path finder" (which implements the logic "when the module isn't in sys.modules, look on sys.path for source code and/or bytecode, including bytecode in __pycache__ subfolders, and create a 'spec' for it") with our own wrapper. The "loader" attached to the resulting spec is replaced with an importlib.util.LazyLoader instance, which wraps the base PathFinder's provided loader. When an import statement actually imports the module, the name will actually get bound to a <class 'importlib.util._LazyModule'> instance, rather than an ordinary module. Attempting to access any attribute of this instance will trigger the normal module loading procedure — which even replaces the global name.Now we can do:
That said, I don't know what the PEP means by "mostly" here.Can you explain why does "threading" needs to be loaded? The rest seems decently straightforward, but what initialization from threading is required and why only in 3.13+?
I didn't investigate deeply (I just happened to stumble on the issue while figuring out the implementation), but it might be needed in older versions too. The loading process instantiates a `threading.RLock`, but from a source code file that doesn't import that (because it's in a file that has to bootstrap the import process). I'm not entirely sure how `threading` is supposed to get imported normally. The other complicating factor is that I was testing this at the REPL, and 3.13 has a new REPL implementation which might change how those initial libraries are brought in.
This is the wrong syntax, comparable to how "u" strings were the wrong syntax and "b" strings are the right syntax.
They make that, what should be the default, a special case. Soon, every new code will use "lazy". The long term effect of such changes is a verbose language syntax.
They should have had a period where one, if they want lazy imports, has to do "from __future__ import lazy_import". After that period, lazy imports become the default. For old-style immediate imports, introduce a syntax: "import now foo" and "from foo import now bar".
All which authors of old code would have to do is run a provided fix script in the root directory of their code.
I have posted this Idea on Python Discussions, and Thomnas Wouters replied to it and to others:
https://discuss.python.org/t/pep-810-explicit-lazy-imports/1...
The specific part of the PEP:
https://github.com/python/peps/pull/4628/files#diff-ca011267...
I think your assessment of what's "the right/wrong" syntax is fair. But the transition you describe takes a long time, even now that the community has figured out a "deprecation cycle" process that seems satisfactory (i.e. won't lead to another Python 3.0 situation).
> All which authors of old code would have to do is run a provided fix script in the root directory of their code.
As I recall, `lib2to3` didn't do a lot to ease tensions. And `six` is still absurdly popular, mainly thanks to `python-dateutil` still attempting to support 2.7.
Honestly, I don't care for 2 to 3 or six (which, btw, has just 1k stars on GitHub, compare to e.g. 90k stars for FastAPI). Someone who bases their code on 2.7 in 2025 must expect to run into trouble.
If people do not run such upgrade scripts, it must be documented better.
Python is free. In order to also stay elegant, they should say to their users: "We expect from you to run an upgrade script on your code once per Python upgrade"
In short, grandiose change, wrong syntax.
Feels like a good feature, with a simple explanation, real world use cases, and a scoped solution (global only, pretty simple keyword). I like it!
Agree, they really did their homework, listed edge cases, made practical compromises, chose not to overdo it, reworked it again and again quite a bit and compared it to real life experience.
It's really beautiful work, especially since touching the back bone (the import system) of a language as popular as Python with such a diverse community is super dangerous surgery.
I'm impressed.
Yeah, I think this is one of the cleanest PEPs to come around in quite a while, at least from the userspace perspective. Interested to see what happens after the traditional syntax bikeshedding ritual has been completed.
Hopefully they learned lessons from why PEP-690 was rejected. I've spent quite a while trying to build this stuff for our codebase and it's never worked well enough to use.
690 is mentioned a few times, including a FAQ note on the differences
https://pep-previews--4622.org.readthedocs.build/pep-0810/#f...
Q: How does this differ from the rejected PEP 690?
A: PEP 810 takes an explicit, opt-in approach instead of PEP 690’s implicit global approach. The key differences are:
Explicit syntax: lazy import foo clearly marks which imports are lazy.
Local scope: Laziness only affects the specific import statement, not cascading to dependencies.
Simpler implementation: Uses proxy objects instead of modifying core dictionary behavior.
I think they're understating the thread safety risks here. The import is going to wind up happening at a random nondeterministic time, in who knows what thread holding who knows what locks (aside from the importer lock).
Previously, if you had some thread hazardous code at module import time, it was highly likely to only run during the single threaded process startup phase, so it was likely harmless. Lazy loading is going to unearth these errors in the most inconvenient way (as Heisenbugs)
(Function level import can trigger this as well, but the top of a function is at least a slightly more deterministic place for imports to happen, and an explicit line of syntax triggering the import/bug)
We tend to prefer explicit top-level imports specifically because they reveal dependency problems as soon as the program starts, rather than potentially hours or days later when a specific code path is executed.
As a counterpoint, having all the imports automatically deferred would instantly dramatically speed up pip for short tasks.
Almost all of this time is spent importing (and later unloading) ultimately useless vendored code. From my testing (hacking the wrapper script to output some diagnostics), literally about 500 modules get imported in total (on top of the baseline for a default Python process), including almost a hundred modules related to Requests and its dependencies, even though no web request was necessary for this command.I think this makes a ton of sense in the very specific narrow use case of python CLI tools. For a web app or other long-lived process, startup time is typically not of extreme concern, and having more simplicity and legibility to the import process seems better.
That's not to say this PEP should not be accepted. One could always apply a no-lazy-imports style rule or disable it via global lazy import control.
https://peps.python.org/pep-0810/#global-lazy-imports-contro...
It saddens me to think that the use case of "python CLI tools" is thought of as anything like "very specific" or "narrow".
There are tons of great CLI tools written in Python, but I would wager that the vast majority of Python code, no matter how you count it, isn't in a CLI tool. Certainly it's a worthy goal to speed them up especially given how frequently things like `pip` are run both by humans who have better things to do than wait on computers, and by automation which does things at giant scale. But most Python project and programmers who are not making CLI tools or other startup-latency sensitive stuff may be better served by the old default import style.
> But most Python project and programmers who are not making CLI tools or other startup-latency sensitive stuff may be better served by the old default import style.
That may well be the case, and Python's development processes would definitely tend to assume the same. That has a lot to do with why PEP 690 was rejected, and why this proposal is opt-in even though many people are concerned that a lot of projects will "opt in everywhere" and create a lot of noise.
Believe it or not, the process is actually very conservative. People complaining about the "churn" caused by deprecations and removals seem not to have any concept of how few suggestions actually get implemented, and how many are rejected (including ones that perennially occur to many new users). A browse through "ideas" forum where new pre-PEP ideas are commonly pitched (https://discuss.python.org/c/ideas/6) gives one the impression of a leisurely stroll through a graveyard.
And many people (including myself) can tell you that you'll often be put through a run-around: if your idea is good and can be implemented, then surely it falls on you to make and publicize (!) a third-party package, to prove the demand and the community support for your specific implementation; but if you somehow get there, now it's trivial for people to install support, so it doesn't need to be in the standard library (cf. Requests).
These tools already have a million ways to avoid this slowdown. Is it so much to ask their authors to care?
Right, Not sure why but a lot of code which claude generates also comes with local dependencies vs globally declared import statements. Don't promote that pattern because
- It reduces visibility into a module’s dependencies.
- It increases the risk of introducing circular dependencies later on.
I haven't fully digested the PEP but perhaps there would be a command-line flag or external tool for dependency validation, a bit like how there are external tools for type annotations?
Who is "we"?
My current team at my current company (see bio if you're really interested), though I should say I'm not authorized to speak on behalf of my employer, so I should really say something more like "I".
This sounds like something that should be covered by tests?
Lazy imports are a great way to create runtime errors far into the operation of a long lived service. Yes, it gives the superficial benefit of 'fast startup', but that upside is negated by the downside of not being sure that once something runs it will run to completion due to a failed import much further down the line. It also allows for some interesting edge cases with the items that are going to be imported no longer being what is on the tin at the time the program is started.
That's fine, because this is still a genuine problem in need of a solution. It's not just about startup time for the sake of it (not that this is even a superficial concern - python startup time with large dependencies quickly gets awful). Large projects can have hefty dependencies that not every user will use. And bundling it all for everyone can sometimes be intractable. The work arounds people use already have the sort of problems you're talking about, on top of being diabolical and hacky. Not having to duplicate and hide imports in functions alone would be a big improvement. It's not like it isn't being proposed as an optional language feature.
An automated test mitigates the risk you describe, and is well worth the tradeoff for fast startup.
I don't consider startup time "superficial" at all; I work in a Django monolith where this problem resulted in each and every management command, test invokation, and container reload incurring a 10-15sec penalty because of just a handful of heavy-to-import libraries used by certain tasks/views. Deferring these made a massive difference.
I don't like the idea of introducing a new keyword. We need a backward compatible solution. I feel like Python needs some kind of universal annotation syntax such as in go (comments) or in Rust (macros). New keyword means all parsers, lsps, editors should be updated.
I’m pretty sure there will be new keywords in Python in the future that only solve one thing.
Not sure if this can be made backward compatible.
Right now all the imports are getting resolved at runtime example in a code like below
When you write this, the entire file1 module is executed right away, which may trigger side effects.If lazy imports suddenly defer execution, those side effects won’t run until much later (or not at all, if the code path isn’t hit). That shift in timing could easily break existing code that depends on import-time behavior.
To avoid using lazy, this there is also a proposal of adding the modules you want to load lazily to a global `__lazy_modules__` variable.
I was talking about syntax. I'm pretty sure there will be new features that will require a new keyword or syntax given the speed of Python growth. It can be universal, for example same as decorator, but it can be applied anywhere.
Let's say this syntax gets introduced in Python 3.16. The @nogil feature can be introduced in 3.17. If such code is running in Python 3.16, the @nogil marker will be ignored.The problem with new keywords is that you have to stick to the newest Python version every time a new keyword is added. Older Python versions will give a syntax error. It's a big problem for libraries. You need to wait for 3-5 years before adding it to a library. There are a lot of people who still use Python 3.8 from 2019.
Understood make sense
I think they mean backwards-compatible syntax-wise, rather than actually allowing this feature to be used on existing code. If I’m understanding correctly they would prefer for the Python grammar to stay the same (hence the comment about updating parsers and IDEs).
But I don’t think I really agree, the extensible annotation syntaxes they mention always feel clunky and awkward to me. For a first-party language feature (especially used as often as this will be), I think dedicated syntax seems right.
Me neither.
Introducing new keyword has become a recent thing in Python.
Seems Python has a deep scare since Python2 to Python3 time and is scared to do anything that causes such drama again.
For me, the worst of all is "async". If 2to3 didn't cause much division, the async definitely divided Python libraries in 2. Sync and Async.
Maybe if they want backward compatible solution, this can be done by some compile or runtime flag like they did with free threading no-gil.
Async or not in modules is a huge pain in the ass for web-dev, but thankfully a lot of Python still isn't web dev. As a Data Scientist you can live your life peacefully without ever worrying about this. But I can see why web devs like it, even though personally I really don't want to see anymore Javascript sneaking inty my Python. Especially when there is already support for IO bound concurrency elsewhere in the language. If I want to do JS syntax, I'll fucking use JS. And I really don't want to see Python go the C++ route where it just wants to be everything and do everything so that you end up with so many possible approaches to the same problem that two devs can't read each others code anymore.
I'm relatively new to Python - how does one do concurrent IO without async/await?
My main complaint, though, about Python async is - because it is opt-in I never know if I forgot a sync IO call somewhere that will block my worker. JS makes everything async by default and there is effectively no chance of blocking.
The same way you always did it: sync io in threads. Caveats are similar to async/await, but stack traces don’t suck.
take your pick: threads, fork, non-explicit coroutines (gevent), io multiplexing (select/poll/epoll/kqueue), virtual threads/goroutines
Async was a soft keyword for many, many years in order to maintain compat. And before that, asyncio used yield to be even more compatible.
They took a decade to solidify that. At some point, you have to balance evolution and stability. For a language as popular as Python, you can not break the world every week, but you can't stay put for 5 years.
They thought about backward compatibility, and offer an alternative syntax that use no keyword for library that want to activate it yet stay compact with old version. It's already in the spec.
Yeah I like the feature but hate the keyword. Dunder lazy imports sounds good enough imho.
I'm a fan because it's something you can explicitly turn on and off. For my Docker based app, I really want to verify the completeness of imports. Preferably, at build and test time. In fact, most of the time I will likely disable lazy loading outright. But, I would really appreciate a faster loading CLI tool.
However, there is a pattern in python to raise an error if, say, pandas doesn't have an excel library installed, which is fine. In the future, will maintainers opt to include a bunch of unused libraries since they won't negatively impact startup time? (Think pandas including 3-4 excel parsers by default, since it will only be loaded when called). It's a much better UX, but, now if you opt out of lazy loading, your code will take longer to load than without it.
This is needed, but I don't like new keywords. What I would love, for many reasons, is if we could decorate statements. Then things like:
import expensive_module
could be:
@lazy
import expensive_module
or you could do:
@retry(3)
x = failure_prone_call(y)
lazy is needed, but maybe there is a more basic change that could give more power with more organic syntax, and not create a new keyword that is special purpose (and extending an already special purpose keyword)
As a side note, It would be great to have a 'preview' for HN comments. I updated the above because I forgot to add line breaks. Sigh. I bet 'preview' would stop a lot of not well thought out comments too.
you can set “delay” in your profile to delay the live posting of a comment after you click the button.
gives me ten minutes to edit (rewrite if i’m honest) before other people see it.
Would this statement decorator then manipulate the AST of the following statement or how would that work?
For what it's worth, you can already do: x = retry(3)(failure_prone_call)(y)
And, for that matter, expensive_module = lazy(importlib.import_module)('expensive_module') .
good luck with that syntax - it would be as possible as passing an inline-defined def as parameter instead of a lambda
It would be interesting if instead you added a syntax whereby a module could declare that it supported lazy importing. Maybe even after running some code with side effects that couldn't be done lazily. For one thing, this would have a much broader performance impact, since it would benefit all users of the library, not just those who explicitly tagged their imports as lazy. For another, it would minimize breakage, since a module author knows best whether, and which parts of, their module can be lazily loaded.
On the other hand, it would create confusion for users of a library when the performance hit of importing a library was delayed to the site of usage. They might not expect, for example, a lag to occur there. I don't think it would cause outright breakage, but people might not like the way it behaved.
I wonder if this proposal suffers an because of Python's extremely generous support period and perhaps the ship has sailed.
- lazy imports are a hugely impactful feature
- lazy imports are already possible without syntax
This means any libraries that get large benefit from lazy imports already use import statements within functions. They can't really use the new feature since 3.14 EoL is _2030_, forever from now. The __lazy_modules__ syntax preserves compatibility only but being eager, not performance - libraries that need lazy imports can't use it until 2030.
This means that the primary target for a long time is CLI authors, which can have a more strict python target and is mentioned many times in the PEP, and libraries that don't have broad Python version support (meaning not just works but works well), which indicates they are probably not major libraries.
Unless the feature gets backported to 2+ versions, it feels not so compelling. But given how it modifies the interpreter to a reasonable degree, I wonder if even any backport is on the table.
In at least the scientific python environment, there's "SPEC0" in which a lot of the de facto core libraries have basically agreed to support the last three versions of Python, no more.
For other libraries they can of course choose as they want, but generally I don't think it's so common for libraries to be as generous with the support length as cpython.
Ugh...I like the idea, but I wish lazy imports were the default. Python allows side effects in the top level though so that would be a breaking change.
Soooo instead now we're going to be in a situation where you're going to be writing "lazy import ..." 99% of the time: unless you're a barbarian, you basically never have side effects at the top level.
Given all the problems people are mentioning, it seems like this proposal is on the wrong side. There should be an easy way for a module to declare itself to be lazy loaded. The module author, not the user, is the one who knows whether lazy loading will break stuff.
> There should be an easy way for a module to declare itself to be lazy loaded.
It can just implement lazy loading itself today, by using module-level __getattr__ (https://docs.python.org/3/reference/datamodel.html#customizi...) to overwrite itself with a private implementation module at the appropriate time. Something like:
Genericizing this is left as an exercise.It's also simpler. It could even be a package level thing similar to typed.py marker. I don't want to pepper literally all my modules with loads of explicit lazy keywords.
I like the approach of ES6 where you pull in bindings that are generally lazily resolved. That is IMO the approach that should be the general strategy for Python.
The import/require schism was necessary precisely of all the issues with module side effects the python community is rediscovering now. I suppose they won’t get around a similar solution eventually; but judging from years of discussions in the JS world, this is going to be dragging on for a while.
This seems unnecessary if we can run imports from functions, right? This feels like a layer of indirection and a potential source for confusion. Rather than implicitly importing a library when the variable is first used, why don't you just explicitly do it?
Edit
> A somewhat common way to delay imports is to move the imports into functions (inline imports), but this practice requires more work to implement and maintain, and can be subverted by a single inadvertent top-level import. Additionally, it obfuscates the full set of dependencies for a module. Analysis of the Python standard library shows that approximately 17% of all imports outside tests (nearly 3500 total imports across 730 files) are already placed inside functions or methods specifically to defer their execution. This demonstrates that developers are already manually implementing lazy imports in performance-sensitive code, but doing so requires scattering imports throughout the codebase and makes the full dependency graph harder to understand at a glance.
I think this is a really weak foundation for this language feature. We don't need it.
On the contrary, I think "everyone is already doing this in their own way, so we will offer it as a language feature, which brings these extra benefits" is one of the stronger arguments for introducing a language feature.
> Rather than implicitly importing a library when the variable is first used, why don't you just explicitly do it?
I think it's on you to explain why that's a better approach for everyone's use cases instead of this language feature
I don’t want lazy imports. That’s just makes performance shitty later and harder to debug. It’s a hacky workaround.
What I want is for imports to not suck and be slow. I’ve had projects where it was faster to compile and run C++ than launch and start a Python CLI. It’s so bad.
imports run arbitrary Python code, making them faster is the same as making every other language feature faster
Can you show me a profile that demonstrate the slowness is because imports are running large amounts of arbitrary Python code? Maybe they shouldn't do that. Maybe libraries should be pushed to not run expensive arbitrary code execution at import time.
This is a great compromise given how much would break if this was the default. Making this the default would be better in the long-run, require taming the abuse of side-effects in imports too, win-win.
If one could dream, modules should have to explicitly declare whether they have side-effects or not, with a marker at the top of the module. Not declaring this and trying anything except declaring a function or class should lead to type-checker errors. Modules declaring this pure-marker then automatically become lazy. Others should require explicit "import_with_side_effects" keyword.
All of this would be impossible today, given how much the Python relies on metaprogramming. Even standard library exposes functions to create classes on the fly like Enum and dataclasses, that are difficult to assert as either pure or impure. With more and more of the ecosystem embracing typed Python, this metaprogramming is reducing into a less dynamic subset. Type checkers and LSPs must have at least some awareness of these modules, without executing them as plain python-code.This would be a huge deal for Python startup time *if* it was applied to all the standard library packages recursively. Right now importing asyncio brings in half the standard library through transitive imports.
It's bad, but it's not that bad.
For comparison, with the (seemingly optimized) Numpy included with my system:Remember mercurial? Me neither. But what I remeber is this article I've read about all the hacks they had to do to achieve reasonable startup time for CLI in python. And the no #1 cause was loading the whole world you don't ever need. As I recall they somehow monkeypatched the interpreter to ignore imports and just remember their existence until they were actually needed, at which point the import happened. So all the dead paths were just skipped.
I recall chg was a must.
The link target should be https://peps.python.org/pep-0810/ not a preview build for a PR. @dang
I wonder how much things would break if all imports were lazy by default.
All my code which uses import probing would fail, such as fallbacks:
Conditional support testing would also break, like having tests which only run if module2 is available: The proto-PEP also gives an example of using where some global configuration changes only during import.In general, "import this" and "import antigravity" - anything with import side-effects - would stop working.
Oh, and as the proto-PEP points out, changes to sys.path and others can cause problems because of the delay between time of lazy import and time of resolution.
Then there's code where you do a long computation then make use of a package which might not be present.
All of these would be replaced with "import module; module.__name__" or something to force the import, or by an explicit use to __import__.You have to opt in with the lazy import keyword no matter what. This pep also prevents lazy import in try catch. I think your concern matters if ‘module’ itself has a lazy import that you want to check exists. With this you now need to be more rigorous in check those sub dependencies.
This can already happen with non top level imports so it is not a necessarily a new issue, but could become more prevalent if there is an overall uptake in this feature for optional dependencies.
My post was a response to the sedatk conjectural "I wonder how much things would break if all imports were lazy by default."
I have zero concerns about this PEP and look forward to its implementation.
With the LazyLoader technique I described at https://news.ycombinator.com/item?id=45467489 , there is no problem:
The implementation should probably convert that exception back to ImportError for you, but the point is that the absence of an implementation can still be detected eagerly while the actual loading occurs lazily.Ahh, so you do the find first, and keep that around before loading.
I have bad memories of using a network filesystem where my Python app's startup time was 5 or more seconds because of all the small file lookups for the import were really slow.
I fixed it by importing modules in functions, only when needed, so the time went down to less than a second. (It was even better using a zipimport, but for other reasons we didn't use that option.)
If I understand things correctly, your code would have the same several-second delay as it tries to resolve everything?
Yes, if checking for a file is slow, then checking for a file is slow. If you need to know up front whether the module exists, then you can't get around using the "figure out whether the module exists" machinery up front. And if the definition of "a module exists" includes cases where the module is represented by a file whose existence you have to check for, then there's no getting around that, either.
(Trying to do "fallback" logic with lazily-loaded modules is also susceptible to race conditions, of course. What if someone defines the module before you try to use it?)
Maybe nothing would break ?
edit: ok well "xxx in sys.modules" would indeed be a problem
Yeah unfortunately in real world Python code people put side effects in their modules all the time.
In Python? I almost never see that. And I’ve certainly never signed off on a PR that did that.
Recursively importing other modules is itself a side effect.
In fact, all the code you see in the module is "side effects", in a sense. A `class` body, for example, has to actually run at import time, creating the class object and attaching it as an attribute of the module object. Similarly for functions. Even a simple assignment of a constant actually has to run at module import. And all of these things add up.
Further, if there isn't already cached bytecode available for the module, by default it will be written to disk as part of the import process. That's inarguably a side effect.
Side-effect means you're changing state outside the scope the code is running.
Sure thing you can declare globals variable and run anything on a module file global scope (outside funcs and class body), but even that 'global' scope is just an illusion, and everything declared there, as yourself said, is scoped to the module's namespace
(and you can't leak the 'globals' when importing the module unless you explicity do so 'from foo import *'. Think of python's import as eval but safer because it doesn't leaks the results from the module execution)
So for a module to have side-effect (for me) it would either:
- Change/Create attributes from other modules
- Call some other function that does side-effect (reflection builtins? IO stuff)
Often it's things like registering functions in a global registry, or patching things about how Python works (e.g. adding stuff to the module path).
You’re right. “Side effect” here means that your program’s behavior changes simply by importing something, eg by monkey patching something else.
If’s not:
* Importing other modules.
* Taking a long time to import.
* Writing .pyc files.
If any program can “import foo” and still execute exactly the same bytecode afterward as before, you can say that foo doesn’t have side effects.
Maybe your code is awesome, but you're almost certainly using a library which does it.
This. The most egregious example I've ever seen is the inflect library, takes almost 2 seconds(!) to import: https://github.com/jaraco/inflect/issues/212
There will be chaos, all code bases usually have tons of side effects.
Kinda related, I wish there was an easy way to exclude dependencies at pip-install time and mock them at runtime so an import doesn't cause an exception. Basically a way for me to approximate "extras" when the author isn't motivated to do it for me, even though it'd be super brittle.
This sounds doable, actually. You'd want to pre-install (say, from a local wheel) a matching dummy dependency where the metadata claims that it's the right version of whatever package (so the installer will just see that the dependency is "already satisfied" and skip it), but the actual implementation code just exposes a hook to your mocking system.
Doesn't work if version resolution decides to upgrade or downgrade your installed package, so you need to make sure the declared version is satisfactory, too.
We need some kind of unexec revival in Linux so we don't have to resort to crap like this. Maybe CRIU based. Worst case, some Python specific hack. But Python's import system is wacky enough without lazy loading. This sounds like combining the worst of multiple worlds.
That seems really great, .moving imports in-line for CLI tools startup and test discovery has always been a pain.
I was looking at PEP 690 and I saw
> A world in which Python only supported imports behaving in a lazy manner would likely be great...we do not envision the Python langauge transitioning to a world where lazy imports are the default...this concept would add complexity to our ecosystem.
Why can't lazy be the default and instead propose an `eager` syntax? The only argument I can imagine is that there's some API that runs a side effect based on importing, but perhaps making it eager for modules with side effects would be a sufficient temporary fix?
Really excited about this - we've recently been struggling with making imports lazy without completely messing up the code in DeepInverse https://deepinv.github.io/deepinv/
Very unfortunate how from json lazy import dumps would result in backward compatibility issues. It reads much better and makes it easier to search for lazy imports, especially if in the future something else becomes optionally lazy as well.
For posterity, as the submitted link seems temporary: https://github.com/python/peps/pull/4622
The proper permanent link is https://peps.python.org/pep-0810/ .
I don’t really agree with the premise:
> The dominant convention in Python code is to place all imports at the beginning of the file. This avoids repetition, makes import dependencies clear and minimizes runtime overhead.
> A somewhat common way to delay imports is to move the imports into functions, but this practice requires more work [and] obfuscates the full set of dependencies for a module.
The first part is just saying the traditions exist because the traditions have always existed. Traditions are allowed to change!
The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Again, traditions are allowed to change!
The point about the import graph being obfuscated would ring more true if Python didn’t already provide lightning fast static analysis tools like ast. If you care about import graphs at the module level then you’re probably already automating everything with ast anyway, at which point you just walk the whole tree looking for imports rather than the top level.
So, really, the whole argument for a new lazy keyword (instead of inlining giant imports where they are needed) is because people like to see import pytorch at the top of the file, and baulk at seeing it — and will refuse to even look for it! — anywhere else? Hmmm.
What does seem like a pain in the ass is having to do this kind of repetitive crap (which they mention in their intro):
But perhaps the solution is a pattern where you put all your stuff like that in your own module and it’s that module which is lazy loaded instead?>The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Again, traditions are allowed to change!
No, they are saying the tradition is there for a reason. Imports at the beginning of the file makes reasoning about the dependencies of a module much easier and faster. I've had to deal with both and I sure as hell know which I'd prefer. Lazy imports by functions is a sometimes necessary evil and it would be very nice if it became unnecessary.
>The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions.
Think you need to read again.
I don't quite understand why they forbid this from being used in functions.
I mean I get why that makes the most sense in most scenarios, but it is not as if the problem of having to choose between declaring dependencies up front or deferring expensive imports until needed does not happen in functions.
Take for instance a function that quickly fails because the arguments are incorrect, it might do a whole bunch of imports that only make sense for that function but which are made immediately obsolete.
It feels like it is forbidden just because someone thought it wasn't a good coding style but to me there is no obvious reason it couldn't work.
Python imports are already way too complicated. I don’t think even more syntax should be added.
Looks good to me. I use a tab-completion trick where the tab-completer tool calls the script I'm about to invoke with special arguments, and the script reflects on itself and responds with possible completions. But because of slow imports, it often takes a while for the completion to respond.
I could, and sometimes do, go through all the imports to figure out which ones are taking a long time to load, but it's a chore.
Won’t this have a very noticeable performance hit on the first request? Thinking web frameworks like Flask and Django.
Does this fix the circular imports problem that comes up if you don't structural your programs in a hierarchical way?
This is what I thought of too. I really only know python, do other languages not have that issue? In python it does not seem like a "problem" to me - whenever I have seen circular import issues it is because the code is organized poorly. I worry that this feature will lead to devs "fixing" circular import issues by using lazy imports.
Sometimes it's hard to avoid cyclic imports, without blaming the design. Like if a Parent has a Child, and the Child needs to know of the parent. Only way to solve that in python is to put everything in the same file, which also feels like bad deisgn.
I would say in that case, the Parent and Child shouldn't need to know about each other - some kind of handler in a different file should.
Although I guess that doesn't work in all cases, like defining foreign key relationships when using an orm (like sqlalchemy) for example. But in the orm case, the way to get around that is... lazy resolution :^)
Interesting. I find that I always have this problem in any non-trivial python project, and don't find this to be due to poorly organized code. I have only seen this requirement in Python.
IME circular import errors aren't due to poor organization; they're due to an arbitrary restriction Python has.
Even with eager importing there is only a "circular import problem" if you try to import names `from` the modules — as I pointed out a few days ago (https://news.ycombinator.com/item?id=45400783).
It should, when you do lazy imports on your own it fixes the problem.
Crossing my fingers that it goes through this time. Been in the top 3 of my Python wishlist for nearly a decade.
why does this have to be a syntax feature and not a lazy code loader at the intepreter level?
doesn't this make the language a little unpredictable in terms of loading times? requiring to touch all parts to fully load the app?
I think a feature like this sees best use in short lived programs (where startup time is a disproportionate percentage of total run time) and programs where really fast startup is essential. There are plenty of places where I could imagine taking advantage of this in my code at work immediately, but I share your concern about unpredictability when libraries we use are also making use of it. It wouldn't be fun to have to dive into dependencies to see what needs to be touched to trigger lazy imports at the most convenient time. Unless I am misunderstanding and a normal import of a module means that all of its lazy imports also become non-lazy?
Make Python more like Haskell, LFG!
YES!!!
what is the point of this? you can just import inside function definitions:
thus numpy.linalg is only imported the first time you call the antislash function. Much cleaner than a global import.Ignore wrong traditions. Put all imports in the innermost scopes of your code!
That's a hack that forces you to duplicate and hide imports. The tradition is to specify imports at the top because it's that much better readability wise.
I wish all imports were lazy by default.
I know/heard there are "some" (which I haven't seen by the way) libraries that depend on import side effects, but the advantage is much bigger.
First of all, the circular import problem will go away, especially on type hints. Although there was a PEP or recent addition to make the annotation not not cause such issue.
Second and most important of all, is the launch time of Python applications. A CLI that uses many different parts of the app has to wait for all the imports to be done.
The second point becomes a lot painful when you have a large application, like a Django project where the auto reload becomes several seconds. Not only auto reload crawls, the testing cycle is slow as well. Every time you want to run test command, it has to wait several seconds. Painful.
So far the solution has been to do the lazy import by importing inside the methods where it's required. That is something, I never got to like to be honest.
Maybe it will be fixed in Python 4, where the JIT uses the type hints as well /s
Lazy imports mean late errors. Fail fast is a good design principle.
Top-level code should not be able to fail except in incredibly deterministic ways that are tested during development. Failing fast is not as good as not failing at all. Lazy imports mean the power to avoid importing things that don't need to be imported at all on this run. Good design also cares about performance to some extent. On my machine, asking pip to do literally nothing takes several times as long as creating a new virtual environment --without-pip .
Python really seems like a bad fit for that. So your imports succeed, what now? Do they have all the functions or fields your program needs? Those are still resolved at the last possible moment. If you want to be sure your program actually runs you will have to write and run tests with and without lazy imports.
[dead]
Wake me up when we can import a module by relative file path.
You… can? I mean in the strictest sense you're technically not importing by file path but if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree. I think as of Python 3.3 the init file is optional so it will do it by default but I can't remember if there are still some cases where it's required. The only thing you can't do is go "up" to a higher directory than the root module.
Also if that doesn't strike your fancy all of the importlib machinery is at your disposal and it's really not very much work to write an import_path() function. It's one of the patterns plug-in systems use and so is stable and expected to be used by end users. No arcane magic required.
> if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree.
`__init__.py` has nothing to do with making this work. It is neither necessary (as of 3.3, yes, you got it right: see https://peps.python.org/pep-0420/) nor sufficient (careless use of sys.path and absolute imports could make it so that the current folder hasn't been imported yet, so you can't even go "up into" it). The folder will already be represented by a module object.
What `__init__.py` does is:
1. Prevents relative imports from also potentially checking in other paths.
2. Provides a space for code that runs before sub-modules, for example to set useful package attributes such as `__all__` (which controls star-imports).
> if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree
haha no
> all of the importlib machinery is at your disposal
This breaks all tooling. Awful option.
Relative imports have been supported for approximately forever (https://stackoverflow.com/questions/72852). If you mean "by explicitly specifying a path string" (as opposed to a symbolic name), that has also been supported for approximately forever (https://stackoverflow.com/questions/67631). Today, the `importlib` standard library exposes all the steps of the import process — including figuring out where the source code is, checking the `sys.modules` cache etc. — and lets you hook into everything (in particular, you can bypass the entire "finder" process and turn a file path into a dummy "spec" which is fed to a "loader").
The flexibility of this system also entails that you can in effect define a completely new programming language, describe the process of creating Python bytecode from your custom source, and have clients transparently `import` source in the other language as a result. Or you can define an import process that grabs code from the Internet (not that it would be a good idea...).
If you mean "by explicitly specifying a relative path, and having it be interpreted according to the path of the current module's source code", well first you have to consider that the current module isn't required to have source code. But if it does, then generally it will have been loaded with the default mechanism, which means the module object will have a `__file__` attribute with an absolute path, and you just set your path relative to that.
Nope. Relative imports work by relative package path, which is not at all the same. Often when you run Python you don't even have a package path.
Using `importlib` is a horrible hack that breaks basically all tooling. You very very obviously are not supposed to do that.
> Relative imports work by relative package path, which is not at all the same.
Exactly. This gives you the flexibility to distribute a complex package across multiple locations.
> Often when you run Python you don't even have a package path.
Any time you successfully import foo.bar, you necessarily have imported foo (because bar is an attribute of that object!), and therefore bar can `from . import` its siblings.
> Using `importlib` is a horrible hack that breaks basically all tooling. You very very obviously are not supposed to do that.
It is exactly as obvious (and true) that you are not "supposed to", in the exact same sense, directly specify where on disk the source code file you want to import is. After all, this constrains the import process to use a source code file. (Similarly if you specify a .pyc directly.) Your relative path doesn't necessarily make any sense after you have packaged and distributed your code and someone else has installed it. It definitely doesn't make any sense if you pack all your modules into a zipapp.
> Using `importlib` is a horrible hack that breaks basically all tooling. You very very obviously are not supposed to do that.
This is an assertion that has absolutely no reasoning behind it. I'm not saying I disagree; I'm just saying there is a time and a place for importlib.
Well not if you want high quality Python code. Pylint and Pyright won't understand it, and those are absolutely critical to writing Python code that works reliably.
> those are absolutely critical to writing Python code that works reliably.
Curious how much reliable Python code was written before those tools existed.
For that matter, curious how much was written before the `types` and `typing` standard library modules appeared.