I have been thinking about this for months now! Very excited to see you've implemented it, and I'm excited to try this out.
Could be fantastic for my use-case. We have a large repo of notebooks that are generally write-once, sometimes-re-run. Having a separate repo/venv/kernel per notebook would be a big overhead, so I currently just try to do a kind of semantic versioning where I make a new kernel on something like a 6-month cadence and try to keep breaking dependency changes to a minimum in that window. I can then keep around the old kernels for running old notebooks that depend on them. This is not at all an ideal state.
PEP 723 dependency management always struck me as a bit non-DRY. Wonder if there is a better way to do it, like annotating imports with versions or something similar.
Unfortunately it won't, at least due to the fact UV is not available in the in-browser/wasm ecosystem. That would be awesome though. Maybe it is possible to make something close in terms of functionality using a custom pyodide kernel + micropip, but I did not look into that.
doesn't seem like I can just point to a pyproject.toml
I can see the point of PEP-723, in the context of jupyter, but another usecase is having your notebook to work on the same environment as some product, instead of just being a standalone thing.
If you have an environment set up with a pyproject.toml, just select the Jupyter kernel you installed in the environment. That feels like the case that is well handled by current tooling.
I believe this is solving the common complaint that you can't just email a jupyter notebook, since it doesn't capture the dependencies.
let's say you have a project with a pyproject.toml and some notebooks. You'll have to 1. Come up with some name for the kernel, 2. Add a script to install the kernel, polluting the collaborator's jupyter installation 3. Add a README referring to 2.
Hi. I appreciate your feedback. Basically, juvio stores all of the project requirements (versions of the packages and of the python interpreter) directly within the notebook itself using the PEP 723 spec. Then, when you open the notebook, a new ephemeral environment is created on the fly with all of the required dependencies. Therefore, you don't have to maintain a separate e.g. requirements.txt/conda.yaml/uv.lock file.
Did you in the past? Normally Jupyter notebooks just include the package installation commands necessary to set up the environment from scratch. I've never seen a requirements.txt/lockfile distributed alongside a notebook.
That's common when they are distributed as single notebooks (i.e. via Google Colab). When distributed inside repos they usually contain a requirements.txt.
I've done research involving sampling from all extant notebooks on GitHub and manually analyzing them, and have been a heavy user of them in non-CS research contexts for a long time (and Mathematica notebooks for years before iPython Notebooks grew in popularity).
Having manually sifted through hundreds of randomly sampled notebooks, I feel I can confidently speak on the distribution of characteristics in them, at least up through a couple years ago.
1. Notebooks on GitHub are not necessarily an entirely representative sample of notebooks at large. If the author is putting it on GitHub, there's already a significant selection bias towards certain topics, despite notebooks being used in practically any discipline you can think of at least to some extent.
2. Notebooks in repositories that contain requirements.txt files are a minority and tend to be intended/cleaned up for sharing (itself not the norm).
3. What's more common is just a handful of !pip install at the top.
4. Even more common is just some details on dependencies in an adjoining README.
A very non-trivial chunk of notebooks on GitHub are just copies of the "Hands On ML 3" textbook/exercise set. If my memory serves there are tens of thousands of copies of that one repository. The fork count shown by GitHub doesn't account for the plethora of copies that weren't made using it.
This is cool and something that I’ve wanted, but I don’t see hot listings requirements inline foregoes the need for a lock file to maintain reproducibility. What about version ranges? Versions of transitive dependencies?
Hi. Thanks for bringing this up. To be honest, I have never tried juv, but judging from the readme the ideas of juv and juvio are slightly different. In juvio the ephemeral environment is created on kernel startup. Hence, one can have multiple notebooks within the same jupyterlab session, each with its own venv. This seems to be different with juv, but please correct me if I am wrong.
I've been using juv on and off for for ~6 months. From what I can tell of juvio, it is a different model for using uv with jupyter notebooks.
I'm not sure which model fits best, I'll have to see how your juvio handles kernels in jupyter. Does the kernel name change, is it all the default kernel, and what changes when an install happens?
I'm not quite sure what you mean by cleaner git diffs, but hopefully that will become clear with experimentation.
For my particular method of working, I've mostly switched to having each small project (roughly a JIRA ticket) be a separate uv-managed project in a git repo, and I create a kernel for each of the uv projects. This allows me to examine multiple different tickets and kernels without having to launch multiple jupyter labs.
The whole kernel<->venv mapping is another layer of massive complexity on top of the current huge amount of complexity in Python packaging. uv makes it fast , but it does not provide the "correct" or even single route to managing venvs.
> In juvio the ephemeral environment is created on kernel startup. Hence, one can have multiple notebooks within the same jupyterlab session, each with its own venv.
I have been thinking about this for months now! Very excited to see you've implemented it, and I'm excited to try this out.
Could be fantastic for my use-case. We have a large repo of notebooks that are generally write-once, sometimes-re-run. Having a separate repo/venv/kernel per notebook would be a big overhead, so I currently just try to do a kind of semantic versioning where I make a new kernel on something like a 6-month cadence and try to keep breaking dependency changes to a minimum in that window. I can then keep around the old kernels for running old notebooks that depend on them. This is not at all an ideal state.
Thanks for sharing!
Hm, I haven't had any luck making this work. Have opened an issue.
there is https://marimo.io/ that does all this and more
I like Jupyter too! It's has been a quick easy solution for me: https://hub.docker.com/r/lukeyoo/jupyter-polyglot
PEP 723 dependency management always struck me as a bit non-DRY. Wonder if there is a better way to do it, like annotating imports with versions or something similar.
The "git friendly format" is nice! How do markdown cells look like? Are they embedded as a python comment?
Has anyone compared juv, juvio and marimo?
Seems awesome ! I'll try it soon.
Thank you! I am looking forward to your feedback.
Would it work on Jupyter lite?
Unfortunately it won't, at least due to the fact UV is not available in the in-browser/wasm ecosystem. That would be awesome though. Maybe it is possible to make something close in terms of functionality using a custom pyodide kernel + micropip, but I did not look into that.
doesn't seem like I can just point to a pyproject.toml
I can see the point of PEP-723, in the context of jupyter, but another usecase is having your notebook to work on the same environment as some product, instead of just being a standalone thing.
If you have an environment set up with a pyproject.toml, just select the Jupyter kernel you installed in the environment. That feels like the case that is well handled by current tooling.
I believe this is solving the common complaint that you can't just email a jupyter notebook, since it doesn't capture the dependencies.
let's say you have a project with a pyproject.toml and some notebooks. You'll have to 1. Come up with some name for the kernel, 2. Add a script to install the kernel, polluting the collaborator's jupyter installation 3. Add a README referring to 2.
> Why Use Juvio?
> No additional lock or requirements files are needed
Additional to what?
> Guaranteed reproducibility
Of what?
I probably need your project, but I don't understand what it is for.
Hi. I appreciate your feedback. Basically, juvio stores all of the project requirements (versions of the packages and of the python interpreter) directly within the notebook itself using the PEP 723 spec. Then, when you open the notebook, a new ephemeral environment is created on the fly with all of the required dependencies. Therefore, you don't have to maintain a separate e.g. requirements.txt/conda.yaml/uv.lock file.
Did you in the past? Normally Jupyter notebooks just include the package installation commands necessary to set up the environment from scratch. I've never seen a requirements.txt/lockfile distributed alongside a notebook.
That's common when they are distributed as single notebooks (i.e. via Google Colab). When distributed inside repos they usually contain a requirements.txt.
I've done research involving sampling from all extant notebooks on GitHub and manually analyzing them, and have been a heavy user of them in non-CS research contexts for a long time (and Mathematica notebooks for years before iPython Notebooks grew in popularity).
Having manually sifted through hundreds of randomly sampled notebooks, I feel I can confidently speak on the distribution of characteristics in them, at least up through a couple years ago.
1. Notebooks on GitHub are not necessarily an entirely representative sample of notebooks at large. If the author is putting it on GitHub, there's already a significant selection bias towards certain topics, despite notebooks being used in practically any discipline you can think of at least to some extent.
2. Notebooks in repositories that contain requirements.txt files are a minority and tend to be intended/cleaned up for sharing (itself not the norm).
3. What's more common is just a handful of !pip install at the top.
4. Even more common is just some details on dependencies in an adjoining README.
A very non-trivial chunk of notebooks on GitHub are just copies of the "Hands On ML 3" textbook/exercise set. If my memory serves there are tens of thousands of copies of that one repository. The fork count shown by GitHub doesn't account for the plethora of copies that weren't made using it.
Specifically "Hands On ML 3", or just generally a copy of some exercise set from a class?
This is cool and something that I’ve wanted, but I don’t see hot listings requirements inline foregoes the need for a lock file to maintain reproducibility. What about version ranges? Versions of transitive dependencies?
Likely uses https://docs.astral.sh/uv/guides/scripts/#improving-reproduc...
See also: juv
https://github.com/manzt/juv
Hi. Thanks for bringing this up. To be honest, I have never tried juv, but judging from the readme the ideas of juv and juvio are slightly different. In juvio the ephemeral environment is created on kernel startup. Hence, one can have multiple notebooks within the same jupyterlab session, each with its own venv. This seems to be different with juv, but please correct me if I am wrong.
I've been using juv on and off for for ~6 months. From what I can tell of juvio, it is a different model for using uv with jupyter notebooks.
I'm not sure which model fits best, I'll have to see how your juvio handles kernels in jupyter. Does the kernel name change, is it all the default kernel, and what changes when an install happens?
I'm not quite sure what you mean by cleaner git diffs, but hopefully that will become clear with experimentation.
For my particular method of working, I've mostly switched to having each small project (roughly a JIRA ticket) be a separate uv-managed project in a git repo, and I create a kernel for each of the uv projects. This allows me to examine multiple different tickets and kernels without having to launch multiple jupyter labs.
The whole kernel<->venv mapping is another layer of massive complexity on top of the current huge amount of complexity in Python packaging. uv makes it fast , but it does not provide the "correct" or even single route to managing venvs.
> In juvio the ephemeral environment is created on kernel startup. Hence, one can have multiple notebooks within the same jupyterlab session, each with its own venv.
This should be your primary selling point!