Launch HN: MutableAI (YC W22) – Automatically clean Jupyter notebooks using AI (mutable.ai)
I'm obsessed with clean code because I've written so much terrible code in the past. I went from being a theoretical physics PhD dropout -> data scientist -> software engineer at Google -> research engineer at DeepMind -> ML engineer at Apple. In that time I've grown to tremendously value code quality. Clean code is not only more maintainable but also more extensible as you can more readily add new features. It even enables you to think thoughts that you may have never considered before.
I want to reduce the cost of clean, production-quality code using AI, and am starting with a niche I'm intimately familiar with (Jupyter), because it's particularly prone to bad code. Jupyter notebooks are beloved by data scientists, but notorious for having spaghetti code that is low on readability, hard to maintain, and hard to move into a production codebase or even share with a colleague. That’s why a Kaggle Grandmaster shocked his audience and recommended that they do not use Jupyter notebooks [1].
MutableAI allows developers to get the best of both worlds: Jupyter’s easy prototyping and visualization, plus greatly improved quality with our AI product. We also offer a full featured AI autocomplete to help prototyping go faster. I think the quadrant of "easy to develop in" and "easy to create high quality code" has been almost empty, and AI can help fill this gap.
Right now there are two ways of manipulating programs: PL techniques for program analysis and transformation, and large scale transformers from OpenAI/DeepMind, which are trained on code treated as text (tokens) and don't look at the tree structure of code (ASTs). MutableAI combines OpenAI Codex / Copilot with traditional PL analysis (variable lifetimes, scopes, etc.) and statistical filters to identify AST transformations that, when successively applied, produce cleaner code.
We use OpenAI's Codex to document and type the code, and for AI autocomplete. We use PL techniques to refactor the code (e.g. extract methods), remove zombie code, and normalize formatting (e.g. remove weird spacing). We use statistical filters to detect opportunities for refactoring, for example when a large grouping of variable lifetimes are suddenly created and destroyed, which can be an opportunity to extract a function.
Some of the PL techniques are similar to traditional refactoring tools, but those tools don’t help you decide when and how to refactor. We use AI and stats to do that, as well as to generate names when the new code needs them.
A tool that reduces the time to productionize code can be compared to having an extra engineer on staff. If you take this seriously, that’s a pretty big market. Stripe Research claims that developer inefficiency is a $300B problem [2]. Just about every tech company would become more efficient through increased velocity, fewer errors, and the ability to tackle more complex problems. It may even become unthinkable to write software without this sort of tool, the same way most people don't write assembly and use a compiler.
You can try the product by visiting our website, https://mutable.ai and creating an account on the setup page https://mutable.ai/setup.html. License keys are on the setup page once you’ve signed up (check your mailbox for an email verification link). I’ve bumped up the budget for free accounts temporarily for the day, I hope you enjoy the product !
In addition to inviting the HN community to try out the product, I’d love it if you would share any tips for reducing code complexity...
50 comments
[ 3.0 ms ] story [ 114 ms ] threadPersonally, I wouldn't pay those prices to hide the data scientist's problems. However, if you developed extensions to prevent and encourage the user to overcome the tool's problems, I would buy that for myself.
Also, you probably don't a demo that would fail to run in the second cell (in both unclean and clean code) .
The extraction is based partly on statistical edge detection. We are working on training transformers on actual diffs on GitHub that would be more natural.
I have data scientists that work for me that have this problem. I'd rather that everyone I hire wrote the kind of code I want out of the box, but for lots of reasons it's not always possible. I have tried enforcing frameworks and ended up wasting more money trying to get people to change their workflow. A developer or data scientist (even an inexperienced one) is expensive enough that $30/mo is nothing if it makes them more productive. I'd guess this is what the company is banking on.
(You'll see I made another comment where i say i don't like their pricing model, this is a matter of principle because I don't believe in just trying to wrap a program in SaaS in order to get more money without offering something beyond "you can run my code if you pay me". But the cost / benefit is still easy to justify)
I think developers in the future will have way more mental free space because of this trend towards accelerating developer tooling with AI.
I can't tell from the website, but what I would absolutely love is if this could turn the notebook into a "train.py" with argparse to parse the arguments and model / program output saved on completion. This is a definite pain point for me, there are things I'd like to test interactively, but I default to trying to get out of a jupyter notebook as quickly as possible because all the code ends up having to be rewritten "properly".
For ML in particular, I bet there are cool integrations you could do with pytorch lightning (and other frameworks) to take me from trying a forward pass in the jupyter notebook to having the datamodel with dataloaders declared and then set up in a main() . There is lots that could be automated.
*(except the SaaS model, I have a really hard time supporting "api call" limited services that seem to just be charging me a subscription fee for the priviledge of running code, but I don't want to get into that.)
(Totally understand you on the SaaS objections -- I don't think you should ever feel like you can't run your code! In the individual plan you should not hit the limit unless many people are using the same key.)
We DO offer an on-prem version of the product that while requiring a decent GPU, does not exactly require people to go out and buy a fleet of TPUs.
I personally think there will be a lot of developments in the efficiency of these models (e.g. DeepMind's RETRO) as well as the usual compute efficiencies always improving.
However, I also believe there will always be an incentive to make the models as big as resources allow, because "more is different" as PW Anderson said in his famous essay and I give kudos to the OpenAI team for pushing the limit on these models.
At MutableAI we're laser focused on actually _transforming_ Jupyter notebooks (beyond formatting). Meaning we will actually remove dead code and in some cases refactor your code for you. We also use large scale transformers / neural networks to document your code, which IIRC prettier does not.
The solution to bad code written by data scientists is not more AI tools that write non-verifiable code. Its ergonomic API frameworks which can relieve the pain-points / bad practices via throughly testable / verifiable / deterministic code.
The site even promises test-case generation in the future. This seems very flaky at best.
There is no replacing the developer (anytime soon) that's why we don't overwrite your code. You still have to review the suggested changes yourself. I personally don't want to spend more time making changes an AI can guess for me to improve my code and even if I have to review every change that still saves a lot of time.
Also I agree, test cases are also hard because while it's easy to generate some tests that verify some trivial behavior, most of the meaningful behavior of programs requires knowing more than just the types of the inputs & outputs. But I believe AI can help with even this problem.
We're working on notebook tooling as well (https://github.com/ploomber/ploomber), but our focus is at the macro level so to speak (how to develop projects that are made up of several notebooks). In recent conversations with data teams, the question "how do you ensure the code quality of each notebook?" has come up a lot, and it's great to see you are tackling that problem. It'll be exciting to see people using both MutableAI and Ploomber! Very cool stuff! I'll give it a try!
It’s only inefficient to move out of notebooks if you’ve written poorly structured, messy code in the first place. We wouldn’t accept that out of other devs, so I’m not sure why data scientists should magically get a pass.
If they can’t do the former, may as well just hire a software developer and a statistician for near-cheaper and have them work together - you save money and get better code.
A few notes:
1. The progress that was made by Omar and the team (from alpha to working version that is useful) is astounding. I am eagerly waiting for more improvements that will make my regular daily workflow even more fun.
2. The tool is useful for both junior and senior developers and data scientists. They get different things from it though: junior developers get cleaner code, fixes for simple mistakes, structure and demonstration of how things “should be done”. Senior developers get to skip mundane tasks, go from prototype to production code quicker and can concentrate on more complex things/details.
3. I especially like that there is an option for “on prem” installation. My current employer is very strict about using “online” tools that can leak our code outside. “cloud only” tool would be a deal breaker for us.
4. Considering the amount of time this tool can potentially safe, it will pay for itself in no time. I am mean really – if it saves me just a few minutes of mindless reformatting/editing/adding comments – it already paid for itself.
“Should we get our data scientists to make the merest amount of effort to stop writing bad code?”
“No, we’ll use an AI to fix their code”
Just what? Notebooks are nice for exploratory code, but that’s it. Got problems with data scientists writing terrible code in notebooks? Make them stop then. Stop accepting PR’s, stop letting them do it.
I’m not saying this as an outsider, I’m a data scientist by trade too, I don’t think we should be enabling bad practices and habits that we wouldn’t tolerate anywhere else in software development.
We should certainly all strive to learn how to write better code and I think regular use of an AI tool like ours _can_ be pedagogical. Many people who did not have the time or opportunity to learn best practices can now join us in building technology because the AI is always there for them and never tires. AI assisted programming will help level out the playing field for talent around the world especially those without access to instructors or good job opportunities.
Will the be used for learning better practices, or will it be used to avoid ever learning them since the AI takes care of it?
But I think people will also learn tons too. At the end of the day, most people learn by example. If you can see what the AI does to your code or suggests as a completion, you will learn best practices faster.
We have customers running batch data pipelines in production that contain notebooks. It’s because of the first class integration of notebooks in the orchestration tool we’ve built (https://github.com/orchest/orchest). It would be great to help our users catch errors before they bite them in prod. Let’s have a chat sometime :-)!
Are there more examples that we can see?
I will definitely post more examples over the coming days. In the meantime, why don't you play with the system yourself and feel free to ping me at omar@mutable.ai with any more feedback.
But the fact you need AI to convert a 'notebook' into 'proper code' shows you just how bad jupyter notebooks are.
(I don't mean 'notebooks' in general. I mean 'jupyter notebooks' specifically.)
It used to be that programmers were taught that "your main function should read like a table of contents, and delegate details appropriately". Now it's "Y U NO FIT EVRYTHIN IN ONE PAEG?"