HN Explorer

Ask HN: What's your experience with AI-based code review tools?

19 points by chw9e ↗ HN
I've seen a lot of AI code review tools hitting the market, and I'm curious to hear from people who have tried them out.

Do they speed up reviews? How?

Do you still need human reviewers, or can you directly land code that the AI says passes?

Do they lead to higher quality code? Do they catch important enough things that the review is worth it, or is it mostly minor things?

16 comments

[ 3.3 ms ] story [ 55.9 ms ] thread
vunderba
In my experience, automated AI review can definitely find and identify bad patterns or practices in code or popular SDKs, but our company's tech stack is highly distributed and there is a lot of business domain logic that the AI simply can't really predict (at least not with the current generation) how new code may interact with it on a macro level.
clnq
A lot of value in code reviews comes from knowledge sharing and collaborating on code. It is close to a pair programming exercise that many people in the team can partake in asynchronously.

I’m sorry for not answering many of your questions, but I do think it’s not worth it, at least in a professional context.

For personal use, it might be different.

chw9e
Sure for knowledge sharing about tribal knowledge or undocumented plans I see the value of having a person reviewing, but a lot of other stuff that comes up in pair programming can probably also be managed by AI like what are best practices, a bug you might have missed, evaluating alternative solutions, etc.

People are incredibly varied on code reviews too, I can see where some standardization on what should be blocking and what is best left as a suggestion could accelerate things in some cases and prevent issues in other cases.

isbvhodnvemrwvn
I'd treat it more as a linter. You deal with it before submitting review for a human.
mxer
It helps catch the most obvious bugs. Like the algorithmic bugs. If the comment string says something and the code does not do that, that's caught fine. But it is not so good at catching domain related bugs. I think that's to be expected because these AI-tools don't have the knowledge of the domain and the hidden assumptions that come from experience in a domain. It could improve in future if the AI-based tools are trained on a lot of design documents + code that solve domain problems.
sharemywin
Which begs the question why isn't it flagging a lack of comments on what the function/class does.
thwarted
It helps catch the most obvious bugs. Like the algorithmic bugs.

IME, algorithmic bugs are rarely obvious.

joshuanapoli
It’s more fun to make your own ai linter then to buy a service.
sharemywin
python has an ast library that would be pretty cool for recursing through a code tree.

I also though it would be cool to add a database of class/functions, GPT comments, params, file path.

I thought it might be cool to have it ask questions about the code also. And store answers.

biql
I think it needs to know every conversation that happened in the company to provide meaningful feedback that is above generic coding suggestions. But then it makes onboarding new people and getting them familiar with the project much easier. Businesses will have a lot of incentives to do that.
heisgone
Management wouldn't risk being called on their bullshit and contradictions.