I think approaches like this don't need to run other than locally. Maybe integrated as pre-push hook. The system is nondeterministic, so it's at odds with the purpose of CI.
I built a more naive version for our team using Copilot and GitHub actions and it works quite well (wish I had metrics too). The team loves it.
The ROI here is so high that I don't mind using the strongest model available for the actual code review. I don't trust Sonnet and such. Just let Opus or GPT 5.5 do the whole thing and pay a bit more for less complexity.
> One of the operational headaches we didn’t predict was that large, advanced models like Claude Opus 4.7 or GPT-5.4 can sometimes spend quite a while thinking through a problem, and to our users this can make it look exactly like a hung job.
I had the same problem in my recursive agent harness. It would always come back, but it could sometimes take up to 10 minutes depending. I fixed this by adding a required "purpose" argument to every tool and call/return event. As the recursive evaluation proceeds, every single thing that happens streams incremental purpose text to the user's browser (also using the magic of JSONL for this). The incremental progress events contain the purpose and a detail section (tool arg JSON) that the user can expand/collapse.
> Today, when an engineer at Cloudflare opens a merge request, it gets an initial pass from a coordinated smörgåsbord of AI agents.
I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.
Every iteration something can be found. How many times do you iterate e.g. on performance - use optimized struct, oh, you can change the architecture etc.? At that point one can just have a while loop for the agents to make changes until no comments left.
As a solo dev or rather nowadays more so only a decision maker / agent overseer, I came to enjoy letting my agents develop against a Gerrit repository / workflow. Dev agent pushes a CL, review agent picks it up (not just the diff, but the full repo), runs tests/reviews/review-subagents and concludes by posting a review as well as a vote. This goes back and forth with new patch sets / replies to the threads. Eventually the CL gets a +2 or whatever and I have the final call to manually submit it.
It is way slower compared to just pushing through development with one agent doing everything yolo against a normal repository, but it seems to me that the additional time is well spent (no, I don't have fancy graphs or similar analysis to prove this other than my gut feeling after looking at recent development results).
we’ve been struggling with review throughput. this actually seems worthwhile to build at this point though i remain fairly skeptical of workflows that are agent-only, at a point it seems like the only practical solution.
we are finding lots of value in self review. its the “imagine you are doing a synchronous paired review with someone - anything that is difficult to explain, has a code smell, doesnt fit the architecture of the system around you, write a comment.” then at the end, agents do a good job of looping over PR comments.
the second thing would be a guided, educational code review tool -
there are a few attempts at this, but nothing that has a good enough interface to actually stick. organize hunks by semantic importance, spend some tokens exploring the surrounding systems, showing how new code, public apis and data model flow with the existing design, and allow a human to traverse larger PRs more quickly.
>Code review is a fantastic mechanism for catching bugs and sharing knowledge
"Sharing knowledge" is one of the first phrases in the article, and highlighted as a key benefit of code review. But the loss to human-capital from this process is never examined in the post.
> Trivial reviews (typo fixes, small doc changes) cost 20 cents on average
They did around 25,000 of these runs (about 20% of total). So CF spent $5k in the period making language models run through PRs which were <10 lines long. I get that CF engineers are paid well, but the labour cost of having an intern/entry level engineer spend ~30-60s looking through these is likely close to $0.20, and that engineer builds some human-capital while they're at it.
This blog post is full of small inconsistencies that make it read like a low quality SEO piece.
> We also extract a shared context file (shared-mr-context.txt) from the coordinator's prompt and write it to disk. Sub-reviewers read this file instead of having the full MR context duplicated in each of their prompts. This was a deliberate decision, as duplicating even a moderately-sized MR context across seven concurrent reviewers would multiply our token costs by 7x.
No, it would not, because neither is the prompt of the subagent 100% of its token usage, nor will the "shared-mr-context.txt" which is then being read have a size of zero compared to the creation of this shared context.
> You don't need seven concurrent AI agents burning Opus-tier tokens to review a one-line typo fix in a README.
Yeah, well you wouldn't have anyways. Earlier in the post it says that Opus is "exclusively for the Review Coordinator".
The suits have taken over Cloudflare. All buzzwords are on the bingo card: Using Bun, modeling agent roles after management, graphs, you name it.
They apparently think they need to cash in on AI by serving models and at the same time blocking scrapers. So they need to fuel the hype by pretending to use it.
This shows how the US economy is fundamentally broken: companies that provide a useful service (in theory, if you discount SSL MITM and turnstile gatekeeping) struggle, quasi-religious scams like OpenAI and Anthropic get funded by mentally ill Boomers and Gen-Xers.
> When we first started experimenting with AI code review, we took the path that most other people probably take: we tried out a few different AI code review tools and found that a lot of these tools worked pretty well, and a lot of them even offered a good amount of customisation and configurability! Unfortunately, though, the one recurring theme that kept coming up was that they just didn’t offer enough flexibility and customisation for an organisation the size of Cloudflare.
Most people I know had the experience that signal to noise was way off, regardless of scale. So it was a burden rather than a help. Code review by AI ended up being a skill before creating the PR so the dev owning the PR addressed everything before the team got bogged down with it in review
I’ve built something similar internally, but under the hood it’s mostly codex exec + Git worktrees. The main advantage over diff-only review is that it can walk the entire codebase, trace dependencies, and understand architectural or cross-system impacts instead of only looking at the changed files. The tradeoff is that it’s noticeably more expensive to run. I'm still experimenting on it but I quite like this approach so far.
How do you all handle code review for projects that use specific frameworks or libraries? I write a lot of PySpark and the "flavour" of code is sort of different than traditional Python. AI code reviewers tend to nitpick conventions and common patterns in these libraries, so I find it not very helpful.
How do you all handle code review for projects that use specific frameworks or libraries? I write a lot of PySpark and the 'flavour' of code is different... AI code reviewers tend to nitpick conventions and common patterns in these libraries, so I find it not very helpful.
21 comments
[ 2.6 ms ] story [ 47.4 ms ] threadI think approaches like this don't need to run other than locally. Maybe integrated as pre-push hook. The system is nondeterministic, so it's at odds with the purpose of CI.
The ROI here is so high that I don't mind using the strongest model available for the actual code review. I don't trust Sonnet and such. Just let Opus or GPT 5.5 do the whole thing and pay a bit more for less complexity.
I had the same problem in my recursive agent harness. It would always come back, but it could sometimes take up to 10 minutes depending. I fixed this by adding a required "purpose" argument to every tool and call/return event. As the recursive evaluation proceeds, every single thing that happens streams incremental purpose text to the user's browser (also using the magic of JSONL for this). The incremental progress events contain the purpose and a detail section (tool arg JSON) that the user can expand/collapse.
I’d prefer to have that happen as some sort of pre commit hook, before a merge request is sent. The feedback loop might be a bit faster and the process might produce less noise this way.
we are finding lots of value in self review. its the “imagine you are doing a synchronous paired review with someone - anything that is difficult to explain, has a code smell, doesnt fit the architecture of the system around you, write a comment.” then at the end, agents do a good job of looping over PR comments.
the second thing would be a guided, educational code review tool - there are a few attempts at this, but nothing that has a good enough interface to actually stick. organize hunks by semantic importance, spend some tokens exploring the surrounding systems, showing how new code, public apis and data model flow with the existing design, and allow a human to traverse larger PRs more quickly.
thank you to cloudflare for publishing this.
"Sharing knowledge" is one of the first phrases in the article, and highlighted as a key benefit of code review. But the loss to human-capital from this process is never examined in the post.
> Trivial reviews (typo fixes, small doc changes) cost 20 cents on average
They did around 25,000 of these runs (about 20% of total). So CF spent $5k in the period making language models run through PRs which were <10 lines long. I get that CF engineers are paid well, but the labour cost of having an intern/entry level engineer spend ~30-60s looking through these is likely close to $0.20, and that engineer builds some human-capital while they're at it.
> We also extract a shared context file (shared-mr-context.txt) from the coordinator's prompt and write it to disk. Sub-reviewers read this file instead of having the full MR context duplicated in each of their prompts. This was a deliberate decision, as duplicating even a moderately-sized MR context across seven concurrent reviewers would multiply our token costs by 7x.
No, it would not, because neither is the prompt of the subagent 100% of its token usage, nor will the "shared-mr-context.txt" which is then being read have a size of zero compared to the creation of this shared context.
> You don't need seven concurrent AI agents burning Opus-tier tokens to review a one-line typo fix in a README.
Yeah, well you wouldn't have anyways. Earlier in the post it says that Opus is "exclusively for the Review Coordinator".
They apparently think they need to cash in on AI by serving models and at the same time blocking scrapers. So they need to fuel the hype by pretending to use it.
This shows how the US economy is fundamentally broken: companies that provide a useful service (in theory, if you discount SSL MITM and turnstile gatekeeping) struggle, quasi-religious scams like OpenAI and Anthropic get funded by mentally ill Boomers and Gen-Xers.
Most people I know had the experience that signal to noise was way off, regardless of scale. So it was a burden rather than a help. Code review by AI ended up being a skill before creating the PR so the dev owning the PR addressed everything before the team got bogged down with it in review