Go goes at least part-way there.
https://golangbyexample.com/go-mod-tidy/https://matthewsetter.com/go-mod-tidy-quick-intro/
You write your module source. You then run go mod tidy. This reads your sources for imports and automatically creates the go.mod and go.sum files
What's nice about this is that it ensures reproducible builds, so you should add those files to your revision control repo.
It would be great to eliminate manifest files but Dependency Confusion, supply chain attacks and malicious project takeovers are a huge security challenge right now.
>it often feels to me that the dependency requirements list in pyproject.toml, requirements.json, maven.xml, CMakeLists.txt, contains information that is redundant to the import or include statements at the top of each file.
It doesn't. The name that you use for a third-party library in software generally isn't remotely enough information to obtain it, and it would be bad to have an ecosystem where it were - since you'd be locked in to implementations for everything and couldn't write software that dynamically (even at compile time) chooses a backend. On the other hand, many people need to care about the provenance of a library and e.g. can't rely on a public repository because of the risk of supply-chain attacks. Lockfiles - like the sort described in the draft PEP 751 (https://peps.python.org/pep-0751/), or e.g. Gemfile.lock for Ruby) - include a lot more information than a package name and version number for that reason (in particular, typically they'll have a file size and hash for the archive file representing the package).
>It seems to me that a reasonable design decision, especially for a scripting language like python, would be to allow specification of versions in the import statement (as pipreqs does) and then have a standard installation process download and install requirements based on those versioned import statements.
It's both especially common for naive Python developers to think this makes sense, and especially infeasible for Python to implement it.
First off, Python modules are designed as singleton objects in a process-global cache (`sys.modules`). Code commonly depends on this for correctness - modules will define APIs that mutate global state, and the change has to be seen program-wide.
Even if the `import` syntax, the runtime import system and installers all collaborated to let you have separate versions of a module loaded in `sys.modules` (and an appropriate way to key them), it'd be impractical for different versions of the same module to discover each other and arrange to share that state. Plus, library authors would have to think about whether they should share state between different versions of the library. There are probably cases where it would be required for correctness, and probably cases where it must not happen for correctness. And it's even worse if the library author ever contemplates changing that aspect of the API.
Second, there's an enormous amount of legacy that would have to be cleaned up. Right now, there is no mapping from import names to the name you install - and there cannot be, for many reasons. Most notably, an installable package may legally provide zero or more import names.
Third, Python is just way too dynamic. An `import` statement is a statement - i.e., actual code that runs when the code does, a step at a time, not just some compile-time directive or metadata. It can validly be anywhere in the file (including within a function - which occasionally solves real problems with circular imports); you can validly import modules in other ways (including ones which bypass the global cache by default - there are good system architecture reasons to use this); and the actual semantics can be altered in a variety of ways (the full system is so complex that I can't even refer you to a single overall document with a proper overview).
> For example, you have to figure out what happens if different versions of a requrement are specified in different files of the same package (in a sense, the concept of "package" starts to weaken or break down in a case like that).
As I hope I explained above, it's even harder than you seem to think. But also, this would be the only real benefit. If you want to have multiple files that always use the same version of a library, then it makes no sense to specify that version information repeatedly. (Repeating the import statement itself is valuable for namespacing reasons.)
> But in some cases, e.g. a single-file python script, it seems like it would be great.
Please read up on PEP 723 "Inline script metadata" - https://peps.python.org/pep-0723/. It does exactly what you appear to want for the single-file case - but through comment pragmas rather than actual syntax - and is supported by Pipx and uv (and perhaps others - and is in scope for my own project in this general area, Paper).
> Has anyone hacked or extended python / setuptools to work this way?
Setuptools has nothing to do with this and is not in a position to do anything about it.
Wow, thank you for taking the time to write this comprehensive answer! In the python context, everything you wrote makes sense to me, and I do have more to learn here. I had thought about imports lower down in a file, but I had forgotten about global state mutation.
Fortunately for you: I routinely search HN for posts about Python, and my current projects are a build backend (bbbb), an installer/environment manager (Paper), and writing about Python packaging on my blog.
By the way... as helpful as you've been here, I thought I should let you know you come off as rude and arrogant, in case you aren't aware of it. I'm not personally offended at all, but the "naive" and "fortunately for you" comments are the type of thing that can make people ignore the substance of what you're saying, which I doubt is your intent. I know this sort of interpersonal navigation can be a pain in the butt.
It wouldn't be at all the first time I've heard it. Thanks for the specific hints.
A fair amount of the time, I do notice that my writing could have an unintended, uncharitable reading like that, but I either can't think of a good way to improve it or (more commonly) just don't feel responsible for doing so. Tone of voice is notoriously hard to communicate; even explicit markers (such as smilies) are often more easily interpreted as sarcastic or insincere.
I especially don't like adding a lot of words to try to avoid giving that impression - because when other people do it, I often feel like they're wasting my time with all the extra words I have to read. (But of course, they don't know that I was already willing to read them charitably....)
Go goes at least part-way there. https://golangbyexample.com/go-mod-tidy/ https://matthewsetter.com/go-mod-tidy-quick-intro/ You write your module source. You then run go mod tidy. This reads your sources for imports and automatically creates the go.mod and go.sum files What's nice about this is that it ensures reproducible builds, so you should add those files to your revision control repo.
Thanks! Sounds like it's time for me to try go.
It would be great to eliminate manifest files but Dependency Confusion, supply chain attacks and malicious project takeovers are a huge security challenge right now.
>it often feels to me that the dependency requirements list in pyproject.toml, requirements.json, maven.xml, CMakeLists.txt, contains information that is redundant to the import or include statements at the top of each file.
It doesn't. The name that you use for a third-party library in software generally isn't remotely enough information to obtain it, and it would be bad to have an ecosystem where it were - since you'd be locked in to implementations for everything and couldn't write software that dynamically (even at compile time) chooses a backend. On the other hand, many people need to care about the provenance of a library and e.g. can't rely on a public repository because of the risk of supply-chain attacks. Lockfiles - like the sort described in the draft PEP 751 (https://peps.python.org/pep-0751/), or e.g. Gemfile.lock for Ruby) - include a lot more information than a package name and version number for that reason (in particular, typically they'll have a file size and hash for the archive file representing the package).
>It seems to me that a reasonable design decision, especially for a scripting language like python, would be to allow specification of versions in the import statement (as pipreqs does) and then have a standard installation process download and install requirements based on those versioned import statements.
It's both especially common for naive Python developers to think this makes sense, and especially infeasible for Python to implement it.
First off, Python modules are designed as singleton objects in a process-global cache (`sys.modules`). Code commonly depends on this for correctness - modules will define APIs that mutate global state, and the change has to be seen program-wide.
Even if the `import` syntax, the runtime import system and installers all collaborated to let you have separate versions of a module loaded in `sys.modules` (and an appropriate way to key them), it'd be impractical for different versions of the same module to discover each other and arrange to share that state. Plus, library authors would have to think about whether they should share state between different versions of the library. There are probably cases where it would be required for correctness, and probably cases where it must not happen for correctness. And it's even worse if the library author ever contemplates changing that aspect of the API.
Second, there's an enormous amount of legacy that would have to be cleaned up. Right now, there is no mapping from import names to the name you install - and there cannot be, for many reasons. Most notably, an installable package may legally provide zero or more import names.
I wrote about this recently on my blog: https://zahlman.github.io/posts/2024/12/24/python-packaging-... (see section "Your package name that ain't what you `import`, makes me frustrated").
Third, Python is just way too dynamic. An `import` statement is a statement - i.e., actual code that runs when the code does, a step at a time, not just some compile-time directive or metadata. It can validly be anywhere in the file (including within a function - which occasionally solves real problems with circular imports); you can validly import modules in other ways (including ones which bypass the global cache by default - there are good system architecture reasons to use this); and the actual semantics can be altered in a variety of ways (the full system is so complex that I can't even refer you to a single overall document with a proper overview).
> For example, you have to figure out what happens if different versions of a requrement are specified in different files of the same package (in a sense, the concept of "package" starts to weaken or break down in a case like that).
As I hope I explained above, it's even harder than you seem to think. But also, this would be the only real benefit. If you want to have multiple files that always use the same version of a library, then it makes no sense to specify that version information repeatedly. (Repeating the import statement itself is valuable for namespacing reasons.)
> But in some cases, e.g. a single-file python script, it seems like it would be great.
Please read up on PEP 723 "Inline script metadata" - https://peps.python.org/pep-0723/. It does exactly what you appear to want for the single-file case - but through comment pragmas rather than actual syntax - and is supported by Pipx and uv (and perhaps others - and is in scope for my own project in this general area, Paper).
> Has anyone hacked or extended python / setuptools to work this way?
Setuptools has nothing to do with this and is not in a position to do anything about it.
Wow, thank you for taking the time to write this comprehensive answer! In the python context, everything you wrote makes sense to me, and I do have more to learn here. I had thought about imports lower down in a file, but I had forgotten about global state mutation.
Fortunately for you: I routinely search HN for posts about Python, and my current projects are a build backend (bbbb), an installer/environment manager (Paper), and writing about Python packaging on my blog.
By the way... as helpful as you've been here, I thought I should let you know you come off as rude and arrogant, in case you aren't aware of it. I'm not personally offended at all, but the "naive" and "fortunately for you" comments are the type of thing that can make people ignore the substance of what you're saying, which I doubt is your intent. I know this sort of interpersonal navigation can be a pain in the butt.
It wouldn't be at all the first time I've heard it. Thanks for the specific hints.
A fair amount of the time, I do notice that my writing could have an unintended, uncharitable reading like that, but I either can't think of a good way to improve it or (more commonly) just don't feel responsible for doing so. Tone of voice is notoriously hard to communicate; even explicit markers (such as smilies) are often more easily interpreted as sarcastic or insincere.
I especially don't like adding a lot of words to try to avoid giving that impression - because when other people do it, I often feel like they're wasting my time with all the extra words I have to read. (But of course, they don't know that I was already willing to read them charitably....)
PEP 723 looks great! I'll give Paper a try one of these days.