No, jupytext converts the binary .ipynb format into a text format that can be diffed and reasoned about like any other code in a git repo. It hooks into jupyter lab such that it automatically keeps both the .ipynb and text source version of a document in sync. How you review the text version is up to whatever source control system and other tools you use.
I think a notebook is VERY different than an ephemeral REPL. I think that is worth considering when thinking about generating docs, tests and source code from a single source of truth (as opposed to separate files).
I agree with your comment, but I do in fact use org-mode as a single source of truth document that can export documentation, tests, source code, and all that jazz. From what I can tell nbdev and org-mode are about the same in terms of features. It doesn't need any special post-processing to make git-diff useful, because org-mode's file format is plain text.
No snark interpreted at all. Yes I am well aware of org mode and love it! I think it is fantastic. There are some key differences that I think make notebooks a bit more interesting - however it sounds like you've made a system you are happy with? I would love to see what you have! Mind sharing it?
Cool, I also wanted to say that I think nbdev looks great, and while I think it and org-mode get similar results, the user experience is surely quite different compared to emacs.
My org-mode workflows start with doom-emacs[1] because otherwise Emacs customization is too much of a time sink. I've written a couple peer-reviewed papers in org-mode (exporting to pdf), my blog is written org-mode (exported to html), and I've done the "literate" thing witih code+docs+tests in one file.
For me, org-mode is better as a file format because it's plain text. I don't mind making quick edits in GitHub or vim if I'm away from Emacs for some reason. But it's definitely best to work with org-mode from within Emacs.
Notebooks are more enjoyable to look at while you're editing them, whereas org-mode generally needs a separate "export" step to be pretty.
> nbdev is an innovative software development framework for Python that embraces literate and exploratory programming. I worked on nbdev from 2020-2023 with Jeremy Howard and, later, Wasim Lorgat. I had the privilege and excitement of exploring the boundaries of developer tools and exploratory programming while working with very talented software engineers. In addition to creating a tool many people enjoyed, I enjoyed using nbdev for personal and professional projects.
So I still wasn't sure and navigated to nbdev.fast.ai:
> Create delightful software with Jupyter Notebooks
Ah "nb" is notebook and we're talking about jupyter notebooks here!
Good question! Yes we did talk to Anaconda (we talked to Peter Wang), and they seemed interested in related but slightly different approaches. Some of their forays into exploratory programming are pyscript https://pyscript.com/, the also acquired python anywhere https://www.pythonanywhere.com/, which is sort of like Wordpress for Python developers!
We ended up partnering with Posit (formerly RStudio) a bit more on this at this was a bit more natural in terms of alignment of product and vision. You should definitely keep an eye on Posit (https://posit.co/) to see what they come up with in this space.
I don't think there was an opportunity to "be acquired" (not sure if that was what you were asking) because we didn't get far enough down the commercialization path to justify that.
Jupytext provides a system for version controlling notebooks, code reviewing them, and using them in the context of a larger Python project, without imposing any constraints on the larger project, because Jupytext saves notebooks as plain .py files.
I ended up choosing Jupytext for a personal project because it seemed less complex than nbdev. (I never made it through the 90 minute nbdev introductory video: https://www.youtube.com/watch?v=l7zS8Ld4_iA).
Are there advantages of nbdev over Jupytext that I'm missing out on?
Yes, nbdev is a system that "compiles" notebooks to python scripts, tests and documentation all from one source of truth. Jupytext cannot do this (nor was it designed to)
I think you’re right that commercialization today doesn’t make sense, but I feel confident that future tooling/products will draw on aspects of nbdev and will owe you and others some gratitude.
To that end, have you happened to notice some of nbdev’s design choices bleed into adjacent tooling? Which pieces do you hope to endure and take new life in future projects?
What’s the biggest barrier keeping larger audiences from developing in this “literate” manner?
I’ve been a follower of the design philosophy of nbdev for a long time, though admittedly never got to use it. Perhaps now is the time! Cheers
Excellent comment and question. Re: commercializing nbdev, this path didn't seem to make sense to me personally (for reasons I outlined), but they very well could make sense for others.
But I'm with you, I really like many of the ideas presented in nbdev and feel like they have legs. I don't think those ideas should be abandoned or thrown away. Posit (https://posit.co) is developing products that draw from many of the ideas in nbdev - infact we are working with them - and I think the directions they are pursuing are very promising.
However, if you have ideas on how to make tools in the space that draw on these ideas, I encourage you to do so. I will even support you to the best of my ability if you decide to.
I really like nbdev, and the entire paradigm it encourages. I wish the onboarding was slightly smoother (a slick interactive tutorial type thing in vscode would do it a world of good) but it's now my default project base when I'm working on personal projects.
> it was (and still is) a pain to version control notebooks in a way that’s conducive to collaboration. For practical purposes, you cannot perform code reviews of notebooks on GitHub without purchasing a tool called ReviewNB.
I have always been really surprised that the IPython developers made the choice to use JSON despite it being unusable with git (and despite them being excellent software engineers). This is one reason I never use notebooks (though not as big as the difficulty of developing with all that long-lived mutable state). Have they ever commented on this?
28 comments
[ 3.1 ms ] story [ 43.9 ms ] threadThis is false. If you save your notebooks with jupytext, they are just plain python files that can be reviewed normally.
This isn't the same at all.
Often in a well done notebook you'll have data display etc which doesn't translate well to a plain python file.
[editing to be less snarky and more helpful]
I agree with your comment, but I do in fact use org-mode as a single source of truth document that can export documentation, tests, source code, and all that jazz. From what I can tell nbdev and org-mode are about the same in terms of features. It doesn't need any special post-processing to make git-diff useful, because org-mode's file format is plain text.
My org-mode workflows start with doom-emacs[1] because otherwise Emacs customization is too much of a time sink. I've written a couple peer-reviewed papers in org-mode (exporting to pdf), my blog is written org-mode (exported to html), and I've done the "literate" thing witih code+docs+tests in one file.
For me, org-mode is better as a file format because it's plain text. I don't mind making quick edits in GitHub or vim if I'm away from Emacs for some reason. But it's definitely best to work with org-mode from within Emacs.
Notebooks are more enjoyable to look at while you're editing them, whereas org-mode generally needs a separate "export" step to be pretty.
Here's a representative example from a blog post: https://gitlab.com/hoosieree/hoosieree.gitlab.io/-/blob/mast... (result is here: https://alexshroyer.com/posts/2023-05-21-Stack-VM-Language.h...)
The svg images (which gitlab renders) are actually generated by a code block in the org-mode file:
I put my cursor in there and hit Enter, and it generates the svg.> nbdev is an innovative software development framework for Python that embraces literate and exploratory programming. I worked on nbdev from 2020-2023 with Jeremy Howard and, later, Wasim Lorgat. I had the privilege and excitement of exploring the boundaries of developer tools and exploratory programming while working with very talented software engineers. In addition to creating a tool many people enjoyed, I enjoyed using nbdev for personal and professional projects.
So I still wasn't sure and navigated to nbdev.fast.ai:
> Create delightful software with Jupyter Notebooks
Ah "nb" is notebook and we're talking about jupyter notebooks here!
- fastcore: https://github.com/fastai/fastcore - the nbs/ folder contains the source files that generate source code, docs and tests. - ghapi: https://github.com/fastai/ghapi - the notebooks that create source code, docs and tests are located in the root.
We ended up partnering with Posit (formerly RStudio) a bit more on this at this was a bit more natural in terms of alignment of product and vision. You should definitely keep an eye on Posit (https://posit.co/) to see what they come up with in this space.
I don't think there was an opportunity to "be acquired" (not sure if that was what you were asking) because we didn't get far enough down the commercialization path to justify that.
I ended up choosing Jupytext for a personal project because it seemed less complex than nbdev. (I never made it through the 90 minute nbdev introductory video: https://www.youtube.com/watch?v=l7zS8Ld4_iA).
Are there advantages of nbdev over Jupytext that I'm missing out on?
To that end, have you happened to notice some of nbdev’s design choices bleed into adjacent tooling? Which pieces do you hope to endure and take new life in future projects?
What’s the biggest barrier keeping larger audiences from developing in this “literate” manner?
I’ve been a follower of the design philosophy of nbdev for a long time, though admittedly never got to use it. Perhaps now is the time! Cheers
But I'm with you, I really like many of the ideas presented in nbdev and feel like they have legs. I don't think those ideas should be abandoned or thrown away. Posit (https://posit.co) is developing products that draw from many of the ideas in nbdev - infact we are working with them - and I think the directions they are pursuing are very promising.
However, if you have ideas on how to make tools in the space that draw on these ideas, I encourage you to do so. I will even support you to the best of my ability if you decide to.
I have always been really surprised that the IPython developers made the choice to use JSON despite it being unusable with git (and despite them being excellent software engineers). This is one reason I never use notebooks (though not as big as the difficulty of developing with all that long-lived mutable state). Have they ever commented on this?