Ask HN: How do you organize, maintain and collaborate on Jupyter notebooks?

8 points by nullspace ↗ HN
We are starting to have large and important jupyter notebooks, that multiple people work on. The primary use cases we have are in-depth analytics and building machine learning models.

It typically works fine when you first create and build the notebook for the initial task. But as soon as we iterate on it with other team members and new features or questions to be answered, it leads to messy, non-DRY and hard to review code. In addition, there's always uncertainty as to whether changing a function breaks other parts of the notebook.

This paradigm feels a bit different from the traditional paradigm where we have different files, modules, and various kinds of tests for each of them.

How do you and your team maintain, organize and collaborate on notebooks effectively?

3 comments

[ 4.6 ms ] story [ 15.6 ms ] thread
> hard to review code

I built ReviewNB specifically for Jupyter notebook code reviews [1]

> various kinds of tests for each of them

Checkout treon, open source test framework for Jupyter Notebooks [2]

Some other tips,

- Move code into Python modules whenever possible

- Maintain dockerfile / requirements.txt so your notebooks are easily reproducible

- Setup treon in a CI/CD pipeline

- Setup internal Binder [3] for quickly launching notebooks on cloud machines

[1] https://reviewnb.com

[2] https://github.com/reviewnb/treon

[3] https://gke.mybinder.org/