With not much more effort you can get a much better review by additionally concatenating the touched files and sending them as context along with the diff. It was the work of about five minutes to make the scaffolding of a very basic bot that does this, and then somewhat more time iterating on the prompt. By the way, I find it's seriously worth sucking up the extra ~four minutes of delay and going up to GPT-5 high rather than using a dumber model; I suspect xhigh is worth the ~5x additional bump in runtime on top of high, but at that point you have to start rearchitecting your workflows around it and I haven't solved that problem yet.
(That's if you don't want to go full Codex and have an agent play around with the PR. Personally I find that GPT-5.2 xhigh is incredibly good at analysing diffs-plus-context without tools.)
Alternative twist on this that I find works very well (and that I posted about a month ago https://news.ycombinator.com/item?id=45959846) - instead of concat&sending touched files, checkout the feature branch and the prompt becomes "help me review this pr, diff attached, we are on the feature branch" with an AI that has access to the codebase (I like Cursor).
I recently started using LLMs to review my code before asking for a more formal review from colleagues. It's actually been surprisingly useful - why waste my colleagues time with small obvious things? But it's also gone much further than that sometimes with deeper reviews points. Even when I don't agree with them it's great having that little bit more food for thought - if anything it helps seed the review
As for PR reviews, assuming you've got linting and static analysis out the way, you'd need to enter a sufficiently reasonable prompt to truly catch problems or surface reviews that match your standard and not generic AI comments.
My company uses some automatic AI PR review bots, and they annoy me more than they help. Lots of useless comments
I have been using Codex as a code review step and it has been magnificent, truly. I don’t like how it writes code, but as a second line of defence I’m getting better code reviews out of it than I’ve ever had from a human.
Hum? I just tell claude to review pr #123 and it uses 'gh' to do everything, including responding to human comments! Feedback from coleagues has been awesome.
while this approach is useful, i think the diff is too small to catch a lot of bugs.
i use https://www.coderabbit.ai/ and it tends to be aware of files that aren't in the diff, and definitely can see the rest of the file your are editing (not just the lines in the diff)
I still dont get the idea about AI code reviews.
A code review (at least in my opinion) is for your peers to check if the changes will have a positive or negative effect on the overall code + architecture.
I have yet to see an LLM being good at this.
Sure, they will leave comments about common made errors (your editor should already warn about this before you even commit it) etc. But to notify about this weird thing that was done to make sure something a lot of customers wanted is made reality.
also, PR's are created to share knowledge. Questions and answers on them are to spread knowledge in the team. AI does not do that.
I work alone (in a medium sized company). No peers, no code review. AI code review is invaluable.
AI is a mixed bag. I'm the type of person who is compelled to have a deep understanding of the code they write. Writing my own code vs fixing AI-generated code is a wash timewise, and the AI generated code is so limited (assuming you pared down the uselessly elaborate code and fixed all the critical runtime bugs) as to restrict further iterations. And I'm talking about uploading an architectural blueprint with every function a documented but otherwise empty stub.
AI is a great bellwether. I bounce ideas off AI for a consensus. The closest equivalent is reading StackOverflow comments. I once offhandedly complained that python had no equivalent for setattr at class scope (as __class__ is not defined until after __new__), but AI showed me how to provide a closure in __prepare__ over the class namespace, which was introduced in 3.3 (?) and to which I paid little attention. What a gem.
AI is great for learning. If you follow a textbook or blog or paper and don't understand, AI can clarify. But be careful with less structured learning - it is important to build a full mental model accounting for every possible outcome and explanation, otherwise you're susceptible to hallucinations. I remember my first derivative in which the end result could be obtained via two separate proofs, one of which would imply an incorrect calculus. You've got to play with it until you're satisfied your mental model accounts for all the facts.
Because AI facilitates learning so easily I feel the best skills for a future generation are those pertaining to memory and retention. Ya know, assuming we don't develop individualized and personalized AI that can model your next word and act as a personal memex.
AI is great as a search assistant. I have much better recall when rereading content. Thus, I prefer to ask AI to search for links to content I vaguely recall, rather than ask AI for it's own summary or recollection.
Despite being terrible at writing decent code, AI provides fantastic code review. It catches everything from subtle high-level errors - even potential errors that haven't yet occured - to api mismatches to syntax errors. I actually wish I was as fluent at namespaces and cgroups as AI, and I'm well versed.
AI is interesting at comments. It can be hard to provide germane comments while wading through the weeds. I feel the best comments are written a few days after the code is complete. AI provides that fresh perspective instantly. And if AI can't understand your code, you had better improve your comments.
I'm about to try AI for unit tests. I prefer hypothesis tests. I took a quick glance at the generated code, and it seemed overly complicated. So I'm not optimistic.
Outside of software, AI is great for things you don't really care about. Yeah it might hallucinate, but my back-and-forth with AI is more about holding up a mirror to myself, revealing inner biases. Especially useful for interior decoration / remodeling.
Of course, take everything I said with a grain of salt as security at my company actively discourages AI. So, everything I said applies to free/cheap plans only. And I haven't tried skills yet.
18 comments
[ 4.2 ms ] story [ 39.2 ms ] thread(That's if you don't want to go full Codex and have an agent play around with the PR. Personally I find that GPT-5.2 xhigh is incredibly good at analysing diffs-plus-context without tools.)
The objective of this initial review is to catch those low-hanging fruit that your colleagues would waste cycles on.
LLMs can catch syntax and basic semantics. Peers can spend time on more interesting things like design and relevant biz context.
As for PR reviews, assuming you've got linting and static analysis out the way, you'd need to enter a sufficiently reasonable prompt to truly catch problems or surface reviews that match your standard and not generic AI comments.
My company uses some automatic AI PR review bots, and they annoy me more than they help. Lots of useless comments
We are sooo gonna get replaced soon...
I didn't see this mentioned, but we've been running bugbot for a while now and it's very good. It catches so many subtle bugs.
i use https://www.coderabbit.ai/ and it tends to be aware of files that aren't in the diff, and definitely can see the rest of the file your are editing (not just the lines in the diff)
Sure, they will leave comments about common made errors (your editor should already warn about this before you even commit it) etc. But to notify about this weird thing that was done to make sure something a lot of customers wanted is made reality.
also, PR's are created to share knowledge. Questions and answers on them are to spread knowledge in the team. AI does not do that.
[edit] Added the part about knowledge sharing
AI is a mixed bag. I'm the type of person who is compelled to have a deep understanding of the code they write. Writing my own code vs fixing AI-generated code is a wash timewise, and the AI generated code is so limited (assuming you pared down the uselessly elaborate code and fixed all the critical runtime bugs) as to restrict further iterations. And I'm talking about uploading an architectural blueprint with every function a documented but otherwise empty stub.
AI is a great bellwether. I bounce ideas off AI for a consensus. The closest equivalent is reading StackOverflow comments. I once offhandedly complained that python had no equivalent for setattr at class scope (as __class__ is not defined until after __new__), but AI showed me how to provide a closure in __prepare__ over the class namespace, which was introduced in 3.3 (?) and to which I paid little attention. What a gem.
AI is great for learning. If you follow a textbook or blog or paper and don't understand, AI can clarify. But be careful with less structured learning - it is important to build a full mental model accounting for every possible outcome and explanation, otherwise you're susceptible to hallucinations. I remember my first derivative in which the end result could be obtained via two separate proofs, one of which would imply an incorrect calculus. You've got to play with it until you're satisfied your mental model accounts for all the facts.
Because AI facilitates learning so easily I feel the best skills for a future generation are those pertaining to memory and retention. Ya know, assuming we don't develop individualized and personalized AI that can model your next word and act as a personal memex.
AI is great as a search assistant. I have much better recall when rereading content. Thus, I prefer to ask AI to search for links to content I vaguely recall, rather than ask AI for it's own summary or recollection.
Despite being terrible at writing decent code, AI provides fantastic code review. It catches everything from subtle high-level errors - even potential errors that haven't yet occured - to api mismatches to syntax errors. I actually wish I was as fluent at namespaces and cgroups as AI, and I'm well versed.
AI is interesting at comments. It can be hard to provide germane comments while wading through the weeds. I feel the best comments are written a few days after the code is complete. AI provides that fresh perspective instantly. And if AI can't understand your code, you had better improve your comments.
I'm about to try AI for unit tests. I prefer hypothesis tests. I took a quick glance at the generated code, and it seemed overly complicated. So I'm not optimistic.
Outside of software, AI is great for things you don't really care about. Yeah it might hallucinate, but my back-and-forth with AI is more about holding up a mirror to myself, revealing inner biases. Especially useful for interior decoration / remodeling.
Of course, take everything I said with a grain of salt as security at my company actively discourages AI. So, everything I said applies to free/cheap plans only. And I haven't tried skills yet.
also works if you have the GitHub cli installed. Would setup an AGENTS.md or SKILL.md to instruct an agent on how to use gh too.