Hi HN, we're the team behind Heal. We recently wanted to measure the actual limits of static reasoning in LLMs: specifically, what classes of bugs AI code reviewers are fundamentally blind to because they lack a runtime.
To measure this, we ran a benchmark. We took 5 products (3 open-source, 2 private) and replayed 236 real pull requests from their history. We ran three popular AI PR reviewers on default settings, and then ran Heal, our agent that builds an application sandbox (to control the seeds, mock dependencies etc)around the application and drives it end-to-end like a user would.
- A runtime allows LLMs to test and verify assumptions without risking increasing noise level.
- PR bots are still great for style, consistency, and files outside the user path (CI config, docs, etc.)
We stopped after spending a few thousand dollars running this because the signal was strong enough, but we’re hoping to grow the dataset. We also acknowledge that requiring a reproducible test (what Heal does) is a different standard than an LLM just leaving a comment.
We’d love to hear your thoughts on the methodology, or if you’ve run into the limits of static AI reviewers in your own workflows. Happy to answer any questions.
We recently replaced vendor solutions with a home grown one. It's doing so much better and is cheaper. The nicest thing is that we can iterate on our own, customizing it to our developer experience, feedback, and learning. For what you are offering, I imagine we would do the same (home grown) if we ever feel the need.
> We stopped after spending a few thousand dollars
How do you plan to charge for this? What are your thoughts on the SaaSpocolypse being likely to effect all the Ai startups? (ai tooling, not labs)
2 comments
[ 0.16 ms ] story [ 6.5 ms ] threadTo measure this, we ran a benchmark. We took 5 products (3 open-source, 2 private) and replayed 236 real pull requests from their history. We ran three popular AI PR reviewers on default settings, and then ran Heal, our agent that builds an application sandbox (to control the seeds, mock dependencies etc)around the application and drives it end-to-end like a user would.
A few insights:
- Stacking code reviewers doesn’t yield better results.
- A runtime allows LLMs to test and verify assumptions without risking increasing noise level.
- PR bots are still great for style, consistency, and files outside the user path (CI config, docs, etc.)
We stopped after spending a few thousand dollars running this because the signal was strong enough, but we’re hoping to grow the dataset. We also acknowledge that requiring a reproducible test (what Heal does) is a different standard than an LLM just leaving a comment.
We’d love to hear your thoughts on the methodology, or if you’ve run into the limits of static AI reviewers in your own workflows. Happy to answer any questions.
> We stopped after spending a few thousand dollars
How do you plan to charge for this? What are your thoughts on the SaaSpocolypse being likely to effect all the Ai startups? (ai tooling, not labs)