I haven't looked at the storage format here, but for anyone looking to make a Jupyter-allike, please make the storage format friendly for code unmodified code review systems. Jupyter stores python inside JSON, and you can read the diffs, but it's a bit painful.
Jupyter has an API for writing your own storage back-end, and I whipped up a prototype to store as regular python files, except adding a space after the first character in any line matching /#[ ]*#/ and putting all of the Jupyter metadata on lines beginning ##. Stripping out the metadata and getting back the original formatting means just removing all lines beginning ##, and removing the second character in every line matching /#[ ]+#/. Unfortunately, under my setup, I had lots of trouble with Jupyter timing out its calls to my storage back-end, likely due to running on a heavily loaded virtualized host.
Thanks again for the pointer. Pluto really seems to nail the major pain points from notebooks: for a quick summary of Julia's Pluto from https://lwn.net/Articles/835930/ :
1. Pluto's on-disk format is genuine executable Julia, with metadata in comments. Code review happens on the actually executed code, so highly regulated/otherwise conservative environments don't have to worry about the single source of truth deviating from what code reviewers see.
2. Pluto disallows multiple declarations of global variables. This prevents problems with getting different results, depending in which snippets within the notebook get executed in which order.
3. Pluto makes sure the notebook's displayed output is always in-sync with its source code; it eliminates the problem of someone getting some result, changing some code, and sharing a notebook with results that don't match the code. It does this by performing dependency analysis across the code fragments and topologically sorting them. It then re-evaluates all dependent code snippets when a given snippet is modified.
I'm fine with Julia's use of unicode for math symbols in general.
The strange ascii characters I'm speaking of here are used to annotate block boundaries in pluto's generated code, they are literally in # comments so they could have used anything.
I think they are the old school ascii box drawing characters. They might live in a separate unicode era these days.
Jupytext makes this all a lot less painful by keeping a Jupyter notebook up to date with a markdown counterpart (without output).
It works a lot better to only keep the markdown in source control. The downside is that anyone who starts from a clean repository scratch will need instructions on how to generate the .ipynb files.
As far as storing a human readable version for code reviews and diffs, we use nbautoexport for this - https://nbautoexport.drivendata.org/ - which automatically exports plain text source files.
(Disclaimer: this tool was written and maintained by colleagues of mine.)
This tool looks cool. Unfortunately, it looks like this tool in export-only. Ideally, one wants a single source of truth, and if your reviewed artifacts can't be imported for execution, then your code review can't be reviewing your source of truth. Don't get me wrong: it's much better than reviewing JSON diffs, but in highly regulated/conservative industries, we'd prefer to be reviewing the actual code that gets compiled/interpreted.
Can you customize the diff command used by your code review software? There is a diff tool for Jupyter called nbdime you can install with pip.
Of course it doesn't know that you only care about the code, but the diffs are a lot cleaner to read. There's also a version of the diff that will basically create a nicely formatted notbook for viewing differnces in a browser.
In practical terms, it means its easy to hack together your own internal custom notebooks for whatever purposes you'd like (if you know Elixir & Phoenix).
How do you manage the server side programming language of choice execution of the code of the notebook? Do you use something existing like IPython kernels somehow, or do you keep an instance of whatever programming language alive yourself?
I would love to see this with kleppmann's automerge, for a fully crdt collaborative notebook experience. I imagine it would be relatively easy with elixir... If I had one fewer project on my plate I would contrib.
Additionally to what has been said in the comments about the format that should be well diff-able, if you get into it, please also do a complete rewrite of a tool like JupyterHub. Anyone who ever had to modify that thing will thank you.
Exciting project! Last time I've used IElixir (Elixir kernel for Jupyter) to give a presentation (using the Rise extension) it was actually very easy to setup but it wasn't able to "stream" cell output to the UI, apparently it was "waiting" to have gathered all outpout of a cell before displaying it. Maybe Niex will be better at that?
A notebook server being a heavily asynchronous application, it makes a lot of sense to build one using Elixir. It would be a dream come true if Niex could be compatible with the Jupyter kernel protocol to integrate with kernels that already exist for other languages and also compatible with Jupyter frontend extensions. Basically it could replace the Tornado app while leveraging the rest of the existing Jupyter ecosystem. Easier said than done obviously ;-)
21 comments
[ 3.5 ms ] story [ 65.1 ms ] threadJupyter has an API for writing your own storage back-end, and I whipped up a prototype to store as regular python files, except adding a space after the first character in any line matching /#[ ]*#/ and putting all of the Jupyter metadata on lines beginning ##. Stripping out the metadata and getting back the original formatting means just removing all lines beginning ##, and removing the second character in every line matching /#[ ]+#/. Unfortunately, under my setup, I had lots of trouble with Jupyter timing out its calls to my storage back-end, likely due to running on a heavily loaded virtualized host.
Working directly with the source files is just one of the niceties it has.
1. Pluto's on-disk format is genuine executable Julia, with metadata in comments. Code review happens on the actually executed code, so highly regulated/otherwise conservative environments don't have to worry about the single source of truth deviating from what code reviewers see.
2. Pluto disallows multiple declarations of global variables. This prevents problems with getting different results, depending in which snippets within the notebook get executed in which order.
3. Pluto makes sure the notebook's displayed output is always in-sync with its source code; it eliminates the problem of someone getting some result, changing some code, and sharing a notebook with results that don't match the code. It does this by performing dependency analysis across the code fragments and topologically sorting them. It then re-evaluates all dependent code snippets when a given snippet is modified.
Wow. This is the way.
The strange ascii characters I'm speaking of here are used to annotate block boundaries in pluto's generated code, they are literally in # comments so they could have used anything.
I think they are the old school ascii box drawing characters. They might live in a separate unicode era these days.
It works a lot better to only keep the markdown in source control. The downside is that anyone who starts from a clean repository scratch will need instructions on how to generate the .ipynb files.
(Disclaimer: this tool was written and maintained by colleagues of mine.)
Of course it doesn't know that you only care about the code, but the diffs are a lot cleaner to read. There's also a version of the diff that will basically create a nicely formatted notbook for viewing differnces in a browser.
In practical terms, it means its easy to hack together your own internal custom notebooks for whatever purposes you'd like (if you know Elixir & Phoenix).
- Collaborative notebooks.
- Long-running notebook scheduling:no more oops, I closed the browser tab.
- Automatic experiment tracking: detect and save metrics, params, and model so you don't have to remember to do it or pollute your code.
- Deploy a model in one click to a REST endpoint, and monitor your model's performance. Or package it into a Docker image and push it to a registry.
Here's an invite link: https://iko.ai/invite/lEMzE_hKwJ2SUbfLdnK7SbZb1c3zUCOAQexakL...
A notebook server being a heavily asynchronous application, it makes a lot of sense to build one using Elixir. It would be a dream come true if Niex could be compatible with the Jupyter kernel protocol to integrate with kernels that already exist for other languages and also compatible with Jupyter frontend extensions. Basically it could replace the Tornado app while leveraging the rest of the existing Jupyter ecosystem. Easier said than done obviously ;-)