I’ve invested a lot of time into the IPython & Jupyter tools. In one job, I even wrote a 0MQ kernel to leverage ipython as a repl for a programming language created inside my company (which had horrible developer tooling).
After working on several large machine learning and data science projects over the years, I’ve sadly come to the conclusion that the notebook environment is a very wrong approach to take for shareable, extensible or reproduceable systems. On my team now, aside from the tiniest, most ephemeral prototypes that can be thrown away entirely, nobody ever uses Jupyter notebooks for anything.
The projects I’m referring to are very similar to those of this post too — an in-house image annotation tool we wrote with PyQT instead of notebook widgets, lots of interactive Flask web apps with simple React pages to demo and explore face detection results, diagnostic graphs, search engine output, and prototyping systems for layouts of image search results pages under different ML based ranking algorithms.
The main reason the notebook caused problems is that it leads to poor software craftsmanship, which should be a consideration even from the earliest stages of models and prototypes (because it will make you produce higher quality output faster, not for any spiritual commitment to coding standards).
When notebooks contain code you wouldn’t want merged into a given library or project’s main branch, it needs code review. And notebooks are awful units of work to review because they break models of automated testing, lack the same context for enforced style guidelines, and are generally written with a mix of priorities usually focused on the way the code author wants to think about the prototype, instead of separating these concerns into decoupled units.
Through review, you quickly realize that anything that needs code reuse has to be using good design principles, separating concerns, organizing things into decoupled functions or possibly classes.
When you factor these all out of the notebook and into a helper module (and write tests), then you see a bunch of the magic constants or initialized parameters that need to be factored out into a parameter file so that you can easily vary parameters, have it version controlled, and have a way to connect results artifacts (usually embedded plots or tables and saved data files) with parameter settings used to generate them.
Then you realize you need to make the overall software environment also reproducible, because a notebook itself is never “reproduceable” apart from the overall software environment it was run in, and other people picking up your notebook would need the same data and possibly a Docker container or other container or VM to match the same software, libraries, network settings, port number, etc.
And this process goes on until you realize you had to factor things into a tested module, automate the creation of the surrounding Docker container or other environment, extract out & document all your parameters, and so on. Until finally there is nothing left in the notebook but custom plotting code or data displays.
And the display units could be e.g. a version controlled latex file that picks up image files or renders table templates with data, or a separate web app that uses good design to implement reusable video display or something, all called from a boring non-notebook launch script, all of which could be driven by a simple custom Makefile — making the whole thing just as interactive, generally more so, as the notebook which suffered all the problems.
I fully agree this requires a team with good software craftsmanship skills. But anybody competent enough to write the original notebook can also learn these skills, and reorganize the work with just a few craftsmanship principles that almost immediately reveal the notebook as something that just gets in the way and slows you down.
And for teams needing to make reproducible data studies or reproducible model training environments for real business situations, this effect...
Thanks for the time to write up this thoughtful reply.
I agree with all your major points about maintaining/sharing notebooks. I don't work in the same collaborative setting (only 1 or 2 other people on the project), so I don't encounter the same code review issues, but I can certainly imagine.
However, I do wonder--do you think notebooks are still valuable intermediate steps towards developing the code you eventually commit, even if the notebooks themselves never see the light of day, or do you think using them at all just leads towards lower-quality code and isn't worth the productivity (in practice)?
Trying to use dispassionate code review is a good practice even when it’s a 1-person project. Using e.g. branches in GitHub as logical units of implementation is very important, since it allows you to test incremental changes, group changes into small, conceptually-related batches, and perform code review by inspecting the diff between the branch’s changes and the existing main implementation.
The benefits of this are increased in multi-person collaborations, but it’s an important practice all the time, even when working alone on a small project.
Notebooks, especially if you have to worry about artificial aspects of the diff (like whether you suppressed cell output) are a pain to work with in this type of code review workflow, even for just one person.
The same is true for using automated testing. Many people don’t develop tests for code in a notebook because all along the process they imagine it’s all ad hoc, single-use code, and they arrive at components they wish to reuse in a fragmented and unplanned way. And since it can be hard to create unit tests for functions or cells from a notebook (which is not a module, the language’s standard for how to share an implementation), this means unless you manually convert to a regular source file, then testing is either very shallow (e.g. does some gross-level output of the notebook vaguely appear correct & bug-free?) or else very manually intensive (like manually executing a bunch of function calls or separate cells with different inputs to verify it doesn’t have a bug yourself).
In terms of using a notebook as an integrated development environment, no, I don’t find that it’s useful for that and it also does not seem like one of the notebook’s goals (at least it’s design is antithetical to development environments).
If I need to rapidly see interactive output, then the Jupyter interactive shell program (among many other interactive interpreters) would be fine, but the notebook (which forces cell-based interaction in a separate browser-driven environment) would not be.
Just imagine comparing the developmemt power of using a non-windowed emacs or vim session to write code, then alt-tab to a shell with an interactive prompt to run that code. The productivity loop is much tighter and has less visual clutter than a browser notebook, while letting you simply do plain, boring text editing when writing the code (no cells, no embedded outputs, no need to remember what variable initialization or package imports have to be re-evaluated after you made a code change). And even better, testing can be automated, so you can just invoke a test command after some code changes to get feedback on whether it unexpectedly broke anything— letting you separate your design choices about comprehensiveness of testing from design choices of the code itself.
For me, the notebook is like trying to use MS Word after you’ve already gotten past the learning curve of latex. Word feels restricting, visually cluttered, nasty to do proper version control or peer review, by comparison to even intermediate knowledge of latex. The same comparison is analogously useful for the notebook compared to using proper software approaches and shell-centric tools.
Of course, Word has its use cases, especially for people who only occasionally write documents, or who write documents that rarely need collaborative edits or historical versioned rollbacks. But for people who are professionally writing documents, there are many much better tools. The same for the notebook.
I feel like most of the problems you list come from trying to version control notebooks, which may just be a futile endeavor.
I primarily use notebooks as a worklog for explorative analysis. I can use them to track down the history of experiments and thought processes that led me to those experiments, to help me avoid treading over similar waters and to keep a single cognitive thread throughout the whole process. And if any idea is worth building actual software for, it then becomes a good reference for those projects.
The crucial idea here is to NOT change the notebook over time, just add to it. Then there are no versioning issues. And when it gets too large to navigate efficiently, just break it apart and continue the thread in the next notebook.
Of course in a collaborative environment this kind of thing is less useful, but for keeping track of individual work it can still be effective.
I am sympathetic to your perspective, especially because it's basically what I thought when I was in grad school too, and I totally know where you are coming from. But I wanted to highlight what I think are reasonable counterpoints to some of what you wrote:
> "I can use them to track down the history of experiments and thought processes that led me to those experiments"
But this is exactly what you can't do with notebooks, mainly because they are not well-suited for review. This is true even when working alone, but is amplified when working in a team because even experimental choices require code review.
For example, I worked on a problem once where we needed to train a neural network for age prediction. Someone on the team very familiar with how to write this using Keras began writing up an ad hoc training script, meant to be a "playground" for changing parameters, re-running analysis, etc.
But through code review, even before any experiment was run, we detected bugs in some of the code, mistakes in how the data was being loaded and data cleaning steps, and assumptions that had been made about parameters for less frequently used activation function parameters. By having extra reviewers look at it and review the intended experiment itself, as well as the code to implement it we saved a lot of time, gained from diversity of opinions about how to handle it, avoided needing to debug incorrect diagnostic or accuracy charts to backtrack to these bugs later.
Code review doesn't always solve all of these problems, much the same way that it doesn't solve all the problems of straight software engineering. But it is clearly such a valuable tool that it almost always should be a baked-in, mandatory part of the process, even for greenfield experimental code. And the notebook makes this so hard that people don't do it, and when they do it, they don't benefit much from it.
> "The crucial idea here is to NOT change the notebook over time, just add to it. Then there are no versioning issues."
I wish it were that easy, and that effectively "undo" resolved all versioning problems, but that is extremely rare. In many cases, you need to run the same analysis notebook with a variety of different parameters, and each one corresponds to a separate experiment. Doing this by manually tweaking parameters, hoping you didn't make a typo, and then storing the output artifacts in a file called "run7_alpha3.0_learningrate_0.001_04242018.tar.gz" leads to awful problems of reproducibility.
The other thing is that even if you only add to a notebook, you might be adding the use of new third-party libraries, or adding code that assumes some version of something has been implicitly upgraded in the background. So you'd still need a ground truth source of all your dependencies, like requirements.txt for Python, with possibly more information about the Python version itself (does your notebook code rely on stably ordered dictionaries starting in Python 3.6? How would someone coming to the notebook know that if they are using 3.5 and the results seem funny for no discernible reason?) -- and in general you'll need some overall environment management setup to ensure, for reproducibility, that other people can download the same data, setup the same language environment, get the right versions of the required libraries, and have other system dependencies if there are any (maybe they need a certain version of CUDNN installed? Or maybe you used TensorFlow built from source with extra optimization flags enabled?)
The idea is to take a good "project hygiene" and software craftsmanship approach out of the box, to avoid situations when someone says, "why is this notebook broken on Bob's laptop? It works on Alice's Ubuntu server...". And to always design for code review and testability from the start, because they are huge productivity enhancers and the benefits compound over time. Even for research-o...
In this case - how do you share results between team members?
You can generate files with code and results (numbers, plots) - but for that the easiest step is to use RMarkdown/knitr - which is "close enough" to Jupyter Notebook(plus, the later is visible on GitHub). And the standard solution of sending 3248.png which was generated by some unknown code with unknown parameters isn't good.
While I agree that Jupyter Notebooks are NOT for working code (just for exploration, development and interaction with it).
development -> when working with data virtually all first approaches will go to trash anyway (as they depend on data, yet unknown to us).
> "In this case - how do you share results between team members?"
It depends on what "results" means. Here are some examples.
- Say the result is a set of model parameters from a trained model, and accompanying diagnostic info, like precision/recall, ROC graph, accuracy, standard errors, and so forth. In that case, we would probably write the training program in such a way as to store that data into a database table accompanied by some type of model ID that can uniquely track back to the particular git commit of the repo when it was at the state of the experiment. Downstream libraries that we write as "production quality" code could be used to automatically read from the results database and produce plots locally or displayed in some web service.
- Say the result is a trained model that fulfills some task, like ranking search results or detecting faces. Then in addition to basic diagnostic metrics like above, we might also provide a web service interface that allows you to run the service on example data for diagnostics. The continuous integration tooling would execute some form of acceptance tests and collate the results so you get trackable feedback about accuracy, and an immutable copy of the model (with an ID that maps back to the git commit) might be uploaded to a remote storage server.
While these items require a greater up-front investment for the first few times you do it, it quickly pays off by providing data provenance over exactly how to reproduce a given result, and it also provides more functional display interfaces, rather than mixing them together like a notebook does.
Actually that is probably the main point: a notebook takes multiple concerns that should be treated separately and tries to intermix them in one interface. It sounds like a great promise, but in reality it's a bad idea almost always.
It's kind of like writing a tic-tac-toe implementation where the source code of the GUI elements directly contains the source code of the game's rule logic or current game state. The sort of mixed-concerns problem that an idea like Model-View-Controller was meant to solve.
Only with notebooks it doesn't just mix units of implementation logic with units of display logic -- it also mixes units of configuration (global variables, file paths, parameter settings), units of the dependencies (implicit reliance on the underlying software setup and correctly versioned dependencies) and units of the system overall (like if you needed TensorFlow built from source with optimization flags, or you modified the code for an upgraded version of some library).
It's better to state a clear goal, like, "The team needs to take the following type of result / summary / display and share it with others." And then apply standard software craftsmanship like you would for any other software problem.
Nobody would say, "well we need to share results with teammates, so we should put embedded plotting functions inside this module that manages the database connection." They're clearly different concerns. Yet both types of code would be put into the same source code if using a notebook, highly coupling things that have no business being coupled that way.
> "development -> when working with data virtually all first approaches will go to trash anyway"
I agree with this, which is advice going all the way back to The Mythical Man-Month. But that doesn't mean you should omit obvious quality practices when designing that first system. The point of the "plan to throw one away" thinking is that you architectural understanding of the problem will evolve as you try to solve it. And in fact, the better code hygiene you use on that first solution, the better you can distinguish between parts of the solution that are architecturally bad versus parts that are actually good ideas but are just being implemented in a messy / careless way.
7 comments
[ 1.4 ms ] story [ 25.9 ms ] threadAfter working on several large machine learning and data science projects over the years, I’ve sadly come to the conclusion that the notebook environment is a very wrong approach to take for shareable, extensible or reproduceable systems. On my team now, aside from the tiniest, most ephemeral prototypes that can be thrown away entirely, nobody ever uses Jupyter notebooks for anything.
The projects I’m referring to are very similar to those of this post too — an in-house image annotation tool we wrote with PyQT instead of notebook widgets, lots of interactive Flask web apps with simple React pages to demo and explore face detection results, diagnostic graphs, search engine output, and prototyping systems for layouts of image search results pages under different ML based ranking algorithms.
The main reason the notebook caused problems is that it leads to poor software craftsmanship, which should be a consideration even from the earliest stages of models and prototypes (because it will make you produce higher quality output faster, not for any spiritual commitment to coding standards).
When notebooks contain code you wouldn’t want merged into a given library or project’s main branch, it needs code review. And notebooks are awful units of work to review because they break models of automated testing, lack the same context for enforced style guidelines, and are generally written with a mix of priorities usually focused on the way the code author wants to think about the prototype, instead of separating these concerns into decoupled units.
Through review, you quickly realize that anything that needs code reuse has to be using good design principles, separating concerns, organizing things into decoupled functions or possibly classes.
When you factor these all out of the notebook and into a helper module (and write tests), then you see a bunch of the magic constants or initialized parameters that need to be factored out into a parameter file so that you can easily vary parameters, have it version controlled, and have a way to connect results artifacts (usually embedded plots or tables and saved data files) with parameter settings used to generate them.
Then you realize you need to make the overall software environment also reproducible, because a notebook itself is never “reproduceable” apart from the overall software environment it was run in, and other people picking up your notebook would need the same data and possibly a Docker container or other container or VM to match the same software, libraries, network settings, port number, etc.
And this process goes on until you realize you had to factor things into a tested module, automate the creation of the surrounding Docker container or other environment, extract out & document all your parameters, and so on. Until finally there is nothing left in the notebook but custom plotting code or data displays.
And the display units could be e.g. a version controlled latex file that picks up image files or renders table templates with data, or a separate web app that uses good design to implement reusable video display or something, all called from a boring non-notebook launch script, all of which could be driven by a simple custom Makefile — making the whole thing just as interactive, generally more so, as the notebook which suffered all the problems.
I fully agree this requires a team with good software craftsmanship skills. But anybody competent enough to write the original notebook can also learn these skills, and reorganize the work with just a few craftsmanship principles that almost immediately reveal the notebook as something that just gets in the way and slows you down.
And for teams needing to make reproducible data studies or reproducible model training environments for real business situations, this effect...
I agree with all your major points about maintaining/sharing notebooks. I don't work in the same collaborative setting (only 1 or 2 other people on the project), so I don't encounter the same code review issues, but I can certainly imagine.
However, I do wonder--do you think notebooks are still valuable intermediate steps towards developing the code you eventually commit, even if the notebooks themselves never see the light of day, or do you think using them at all just leads towards lower-quality code and isn't worth the productivity (in practice)?
The benefits of this are increased in multi-person collaborations, but it’s an important practice all the time, even when working alone on a small project.
Notebooks, especially if you have to worry about artificial aspects of the diff (like whether you suppressed cell output) are a pain to work with in this type of code review workflow, even for just one person.
The same is true for using automated testing. Many people don’t develop tests for code in a notebook because all along the process they imagine it’s all ad hoc, single-use code, and they arrive at components they wish to reuse in a fragmented and unplanned way. And since it can be hard to create unit tests for functions or cells from a notebook (which is not a module, the language’s standard for how to share an implementation), this means unless you manually convert to a regular source file, then testing is either very shallow (e.g. does some gross-level output of the notebook vaguely appear correct & bug-free?) or else very manually intensive (like manually executing a bunch of function calls or separate cells with different inputs to verify it doesn’t have a bug yourself).
In terms of using a notebook as an integrated development environment, no, I don’t find that it’s useful for that and it also does not seem like one of the notebook’s goals (at least it’s design is antithetical to development environments).
If I need to rapidly see interactive output, then the Jupyter interactive shell program (among many other interactive interpreters) would be fine, but the notebook (which forces cell-based interaction in a separate browser-driven environment) would not be.
Just imagine comparing the developmemt power of using a non-windowed emacs or vim session to write code, then alt-tab to a shell with an interactive prompt to run that code. The productivity loop is much tighter and has less visual clutter than a browser notebook, while letting you simply do plain, boring text editing when writing the code (no cells, no embedded outputs, no need to remember what variable initialization or package imports have to be re-evaluated after you made a code change). And even better, testing can be automated, so you can just invoke a test command after some code changes to get feedback on whether it unexpectedly broke anything— letting you separate your design choices about comprehensiveness of testing from design choices of the code itself.
For me, the notebook is like trying to use MS Word after you’ve already gotten past the learning curve of latex. Word feels restricting, visually cluttered, nasty to do proper version control or peer review, by comparison to even intermediate knowledge of latex. The same comparison is analogously useful for the notebook compared to using proper software approaches and shell-centric tools.
Of course, Word has its use cases, especially for people who only occasionally write documents, or who write documents that rarely need collaborative edits or historical versioned rollbacks. But for people who are professionally writing documents, there are many much better tools. The same for the notebook.
I primarily use notebooks as a worklog for explorative analysis. I can use them to track down the history of experiments and thought processes that led me to those experiments, to help me avoid treading over similar waters and to keep a single cognitive thread throughout the whole process. And if any idea is worth building actual software for, it then becomes a good reference for those projects.
The crucial idea here is to NOT change the notebook over time, just add to it. Then there are no versioning issues. And when it gets too large to navigate efficiently, just break it apart and continue the thread in the next notebook.
Of course in a collaborative environment this kind of thing is less useful, but for keeping track of individual work it can still be effective.
> "I can use them to track down the history of experiments and thought processes that led me to those experiments"
But this is exactly what you can't do with notebooks, mainly because they are not well-suited for review. This is true even when working alone, but is amplified when working in a team because even experimental choices require code review.
For example, I worked on a problem once where we needed to train a neural network for age prediction. Someone on the team very familiar with how to write this using Keras began writing up an ad hoc training script, meant to be a "playground" for changing parameters, re-running analysis, etc.
But through code review, even before any experiment was run, we detected bugs in some of the code, mistakes in how the data was being loaded and data cleaning steps, and assumptions that had been made about parameters for less frequently used activation function parameters. By having extra reviewers look at it and review the intended experiment itself, as well as the code to implement it we saved a lot of time, gained from diversity of opinions about how to handle it, avoided needing to debug incorrect diagnostic or accuracy charts to backtrack to these bugs later.
Code review doesn't always solve all of these problems, much the same way that it doesn't solve all the problems of straight software engineering. But it is clearly such a valuable tool that it almost always should be a baked-in, mandatory part of the process, even for greenfield experimental code. And the notebook makes this so hard that people don't do it, and when they do it, they don't benefit much from it.
> "The crucial idea here is to NOT change the notebook over time, just add to it. Then there are no versioning issues."
I wish it were that easy, and that effectively "undo" resolved all versioning problems, but that is extremely rare. In many cases, you need to run the same analysis notebook with a variety of different parameters, and each one corresponds to a separate experiment. Doing this by manually tweaking parameters, hoping you didn't make a typo, and then storing the output artifacts in a file called "run7_alpha3.0_learningrate_0.001_04242018.tar.gz" leads to awful problems of reproducibility.
The other thing is that even if you only add to a notebook, you might be adding the use of new third-party libraries, or adding code that assumes some version of something has been implicitly upgraded in the background. So you'd still need a ground truth source of all your dependencies, like requirements.txt for Python, with possibly more information about the Python version itself (does your notebook code rely on stably ordered dictionaries starting in Python 3.6? How would someone coming to the notebook know that if they are using 3.5 and the results seem funny for no discernible reason?) -- and in general you'll need some overall environment management setup to ensure, for reproducibility, that other people can download the same data, setup the same language environment, get the right versions of the required libraries, and have other system dependencies if there are any (maybe they need a certain version of CUDNN installed? Or maybe you used TensorFlow built from source with extra optimization flags enabled?)
The idea is to take a good "project hygiene" and software craftsmanship approach out of the box, to avoid situations when someone says, "why is this notebook broken on Bob's laptop? It works on Alice's Ubuntu server...". And to always design for code review and testability from the start, because they are huge productivity enhancers and the benefits compound over time. Even for research-o...
You can generate files with code and results (numbers, plots) - but for that the easiest step is to use RMarkdown/knitr - which is "close enough" to Jupyter Notebook(plus, the later is visible on GitHub). And the standard solution of sending 3248.png which was generated by some unknown code with unknown parameters isn't good.
While I agree that Jupyter Notebooks are NOT for working code (just for exploration, development and interaction with it).
development -> when working with data virtually all first approaches will go to trash anyway (as they depend on data, yet unknown to us).
It depends on what "results" means. Here are some examples.
- Say the result is a set of model parameters from a trained model, and accompanying diagnostic info, like precision/recall, ROC graph, accuracy, standard errors, and so forth. In that case, we would probably write the training program in such a way as to store that data into a database table accompanied by some type of model ID that can uniquely track back to the particular git commit of the repo when it was at the state of the experiment. Downstream libraries that we write as "production quality" code could be used to automatically read from the results database and produce plots locally or displayed in some web service.
- Say the result is a trained model that fulfills some task, like ranking search results or detecting faces. Then in addition to basic diagnostic metrics like above, we might also provide a web service interface that allows you to run the service on example data for diagnostics. The continuous integration tooling would execute some form of acceptance tests and collate the results so you get trackable feedback about accuracy, and an immutable copy of the model (with an ID that maps back to the git commit) might be uploaded to a remote storage server.
While these items require a greater up-front investment for the first few times you do it, it quickly pays off by providing data provenance over exactly how to reproduce a given result, and it also provides more functional display interfaces, rather than mixing them together like a notebook does.
Actually that is probably the main point: a notebook takes multiple concerns that should be treated separately and tries to intermix them in one interface. It sounds like a great promise, but in reality it's a bad idea almost always.
It's kind of like writing a tic-tac-toe implementation where the source code of the GUI elements directly contains the source code of the game's rule logic or current game state. The sort of mixed-concerns problem that an idea like Model-View-Controller was meant to solve.
Only with notebooks it doesn't just mix units of implementation logic with units of display logic -- it also mixes units of configuration (global variables, file paths, parameter settings), units of the dependencies (implicit reliance on the underlying software setup and correctly versioned dependencies) and units of the system overall (like if you needed TensorFlow built from source with optimization flags, or you modified the code for an upgraded version of some library).
It's better to state a clear goal, like, "The team needs to take the following type of result / summary / display and share it with others." And then apply standard software craftsmanship like you would for any other software problem.
Nobody would say, "well we need to share results with teammates, so we should put embedded plotting functions inside this module that manages the database connection." They're clearly different concerns. Yet both types of code would be put into the same source code if using a notebook, highly coupling things that have no business being coupled that way.
> "development -> when working with data virtually all first approaches will go to trash anyway"
I agree with this, which is advice going all the way back to The Mythical Man-Month. But that doesn't mean you should omit obvious quality practices when designing that first system. The point of the "plan to throw one away" thinking is that you architectural understanding of the problem will evolve as you try to solve it. And in fact, the better code hygiene you use on that first solution, the better you can distinguish between parts of the solution that are architecturally bad versus parts that are actually good ideas but are just being implemented in a messy / careless way.