I’m all in for building more intuitive UIs to make the review process less cumbersome. But with the current capabilities of LLMs, under no circumstances should we allow AI to be the final judge of whether something should be merged into trunk.
Code review is the last line of defense we have against our systems being invaded by the massive amount of slop that’s getting generated left and right.
Instead of trying to automate the code review process, maybe we should spend more energy on making the scaffolding around it better: better diff tools, semantically grouped files (as Devin mentioned), and better UI for large diffs (GitHub’s UI is horrible for anything beyond a thousand lines).
I can foresee a future of induced demand, where by making PRs "easier" to review, you will end up with way more PRs to review, leading PR backlogs as backed up with PRs as ever. Except now dev teams will have trust-me-bro LLM reviews convincing them that they don't actually need to do full code reviews on code they're putting into production. What could go wrong?
> Devin Review is free and available for PRs on regular GitHub repositories (not GitHub Enterprise). Public PRs don’t require a Devin account.
I guess the tokens are cheap enough or their pockets are deep enough, but this still seems surprising. I guess they can chalk it up to a marketing cost.
Overall I've been really impressed with Devin. IMO it's the best tool for AI generating features if you know what you're looking for, have patterns to follow, etc. I suspect the context they build about your project helps a ton.
I was literally just working on a system, using Devin to do the review no less, to add a bunch of the rules we have that are outside of linting's capability to tackle the same kind of thing. Tools like Copilot and Qodo have very high noise ratios, but do occasionally catch legit bugs. Devin Review could be a great complement, and hopefully they'll make it so we can add our own rules soon.
Devin? Now that's a name I've not heard in a long time...a lonnng time.
Seriously, in this age of Claude Code and Codex, does anyone use Devin, or even know someone who does? Do they have any users at all?
Ironically, their product has probably got massively better in the last couple of years, because the underlying LLMs got massively better at coding and long-context tasks. But that doth not a successful business model make, and unless you’re Cursor (and even then I’m not so sure) this is a very very hard space to succeed in without owning your own frontier model (i.e being Anthropic, OpenAI, or Google).
> code review—not code generation—is now the bottleneck to shipping great products.
Unsurprising, since a human still needs to understand and verify the code, be that as it's written or as it's reviewed. AI's only managed to move the brainpower required from the fun part to the tedious and boring part.
I wanted to look into their pricing for Devin+ and I have to say, ACU are entirely too opaque/confusing/complicated. The entire description of them is shrouded in mystery. And this part confuses me even more:
> Aside from the few ACUs required to keep the Devin VM running, Devin will not consume ACUs when:
> Waiting for your response
> Waiting for a test suite to run
> Setting up and cloning repositories
Ok, that kind of makes sense, but what does "the few ACUs required to keep the Devin VM running" mean? These cost $2.50/ea so "a few" means $5+ and on what time scale? Daily? Monthly?
The lowest plan comes with $20 ACUs but they don't list anywhere how far that gets you or even rough examples. I guess if you want to kick the tires $20 isn't a crazy amount to test it out yourself and maybe I'm just not the target market (I kind of feel like I am though?) but I wish their pricing made sense.
Have been using Devin Review for a little bit, and I think it's the first of the many "code review" LLM-bots that have come out that doesn't actively feel like "slop". Seems like they must have some integrations with codemaps or deepwiki (the Cognition products I use most often) to power the insights.
My favorite feature has been organizing the files by "logical flow" rather than alphabetically, which feels like such a tiny change but it's such a huge QOL upgrade. A lot of the features seem inspired by Graphite, which is also really enjoyable.
I just tried this on a production PR and I liked it. It found some things that Claude review missed, but missed some that Codex review found.
I'd rank this the best of the three.
Generally I actually like Gemini reviews a lot (I guess code review tasks stop it going off track drunk like Gemini tends to do when coding?) but at the moment for some reason my Gemini auth is broken and I can't work out how to fix it. Yay Google.
15 comments of 32
[ 0.31 ms ] story [ 31.4 ms ] threadso why would they be any better at reviewing?
Code review is the last line of defense we have against our systems being invaded by the massive amount of slop that’s getting generated left and right.
Instead of trying to automate the code review process, maybe we should spend more energy on making the scaffolding around it better: better diff tools, semantically grouped files (as Devin mentioned), and better UI for large diffs (GitHub’s UI is horrible for anything beyond a thousand lines).
I guess the tokens are cheap enough or their pockets are deep enough, but this still seems surprising. I guess they can chalk it up to a marketing cost.
I was literally just working on a system, using Devin to do the review no less, to add a bunch of the rules we have that are outside of linting's capability to tackle the same kind of thing. Tools like Copilot and Qodo have very high noise ratios, but do occasionally catch legit bugs. Devin Review could be a great complement, and hopefully they'll make it so we can add our own rules soon.
Seriously, in this age of Claude Code and Codex, does anyone use Devin, or even know someone who does? Do they have any users at all?
Ironically, their product has probably got massively better in the last couple of years, because the underlying LLMs got massively better at coding and long-context tasks. But that doth not a successful business model make, and unless you’re Cursor (and even then I’m not so sure) this is a very very hard space to succeed in without owning your own frontier model (i.e being Anthropic, OpenAI, or Google).
https://arxiv.org/abs/2510.15061
Unsurprising, since a human still needs to understand and verify the code, be that as it's written or as it's reviewed. AI's only managed to move the brainpower required from the fun part to the tedious and boring part.
> Aside from the few ACUs required to keep the Devin VM running, Devin will not consume ACUs when:
> Waiting for your response
> Waiting for a test suite to run
> Setting up and cloning repositories
Ok, that kind of makes sense, but what does "the few ACUs required to keep the Devin VM running" mean? These cost $2.50/ea so "a few" means $5+ and on what time scale? Daily? Monthly?
The lowest plan comes with $20 ACUs but they don't list anywhere how far that gets you or even rough examples. I guess if you want to kick the tires $20 isn't a crazy amount to test it out yourself and maybe I'm just not the target market (I kind of feel like I am though?) but I wish their pricing made sense.
My favorite feature has been organizing the files by "logical flow" rather than alphabetically, which feels like such a tiny change but it's such a huge QOL upgrade. A lot of the features seem inspired by Graphite, which is also really enjoyable.
I'd rank this the best of the three.
Generally I actually like Gemini reviews a lot (I guess code review tasks stop it going off track drunk like Gemini tends to do when coding?) but at the moment for some reason my Gemini auth is broken and I can't work out how to fix it. Yay Google.