Maybe but I’d argue the frontier corps will self destruct via green annd product management enough to make this impractical. So far here’s how I see that going
1) can’t easily autopilot due to security. Ganna keep on prompting you into running weird shell commands
2) it’ll launch your code on weird Anthropic or ChatGPT sites behind a paywall
3) between the two of them it’ll
Misalign the solution so it’s unusable
I'm noticing AI catching many bugs in code that previously went unnoticed. Some of them are important, and it's great to catch them early.
The other 11 items in the list of 12 would probably never be hit in practice, but now that you know, you have to sort out how to handle each (even if that just means documenting, filing, or ignoring: you have to understand each of the disparate 12 issues first.)
This is from March 2026, but it does match what I've encountered professionally. Code review was always a significant bottleneck and is even more so with agentic coding. We've had discussions about leaning more on automated review, but it hasn't translated into any practical improvements so far. I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
I think if there is a major change in code review culture, it's not going to happen first in established engineering teams. It'll happen in startups that adopt a no-human-review policy from jump and then maintain it as they mature. I'm sure that's happening, but it's likely invisible to most studies and surveys at the moment. If it proves successful, and such companies start shipping significantly faster, the rest of the industry will be forced to adopt.
You have to classify each code review based on what it changes and decide how to handle it, agent review and/or human review. You can have an agent do that.
You don't give up on line-by-line review just because the human is getting better either, and senior engineers want other people to do line by line review on their own code. It's not a matter of how good or bad the person or program generating the code is.
My experience with prototyping games is that you need very strong human driven guidance and review in the beginning, then you can increasingly take your hands off the wheel.
If you take it off too early you end up with shit and will spend weeks fixing all these baked bad assumptions your project is built on top of. I spent weeks fixing some of this shit then the LLM revived a not insignificant portion of it in a later change because it got "confused" (its words) because some of the comments it previously made were left around.
> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
I actually dislike traditional code review. In part this is because most of our code review tools are very cumbersome to use (GitHub's and GitLab's web interfaces in particular).
But also because humans aren't all that great at code review when it comes to correctness and security. Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.
The problem is that purely agentic code review has orthogonal problems, and typically remains subpar when it comes to judging architecture and design considerations, compared to human reviewers. Both false positives and false negatives also remain a concern.
My best results so far have been with agent-assisted code review in a proper IDE, helping you to quickly construct a mental model and to analyze the underlying logic (in addition to doing a first pass for correctness issues).
> But also because humans aren't all that great at code review when it comes to correctness and security. Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.
Code review is not really about correctness and security. We have testing and other verification process for these. Code review is about idea discussion mostly and correcting a few stuff that may have slipped through the author's initial review.
Your PR wants to change the software in some way. My review is to comment on how you've implemented that change, like did you miss some areas where you needed to change, does your description is actually what is happening (either a cursory reading or through testing), or are you doing some unnecessary change.
IMO, the initial reading shouldn't take more than 10 minutes. More than that (so a big diff) it should have been discussed upfront, and the design agreed. If it's an important area, a more careful reading should be done about the tests and relevant stuff.
But generally, if a review takes more than 10 minutes, there's some failure of communication going on (reviewer not apt to understand the change, big diffs that should have been discussed outside of the PR, obtuse code,...)
All good points. Maybe having worked a lot with concurrent and distributed system makes me distrustful of relying on unit/integration tests for correctness; I've seen too many concurrency issues pass CI validation. But unit tests also sometimes miss edge cases in more straightforward code, so I'm still a bit more cautious.
I also agree that merge requests should be short and understandable in 10 minutes, but IME this happy state often does not obtain in practice. It shouldn't, but it often does, whether it's mission creep, whether it's turning out that a change is more complex than originally imagined, or whether it is unavoidably complex and cannot be broken down further.
> IMO, the initial reading shouldn't take more than 10 minutes. More than that (so a big diff) it should have been discussed upfront, and the design agreed
Part of the code review process is to verify that the writer actually did build the agreed design too though.
My main goal in a review is to ensure the design of the system and approach to the change is resilient to errors (by me and the author). You have to assume there will be mistakes you can’t catch (still, catch as many as you can), but other factors heavily influence how bad the mistakes will be.
> Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.
I actually believe the opposite: exactly the fact that a reviewer had to develop a mental model independently is why code review is and remains valuable (and why people use different models when doing agentic reviews).
The biggest value of doing code reviews is in ensuring maintainability of the code (1. someone else can understand it, 2. it is readable and 3. can be evolved in the future) — everything else is downstream from that.
> exactly the fact that a reviewer had to develop a mental model
But do they? In my experience, at most workplaces it's simply a meat linter phase with the occasional useless comment about style (which should probably be handled by proper CI/CD).
Like if you don't even check out the code (generalized 'you'), then they can hardly build up that model. And even checked out, viewed in its proper context, Naur's famous paper comes to mind - the author who wrote it is the only one who has properly built the theory (which might have included alternative implementations that failed but have no visible artifact).
I'm not claiming that there are no places that go absolutely above and beyond with their review sessions, but I would argue it's really rare - and AIs are great at figuring out some of the context the PR description lacks.
What you describe sounds like little more than rubber stamping a PR target than reviewing it.. I see it too, but I don’t thing we should double down on it just because it’s common
People say one thing and do another. It’s tiring. Long before LLMs we had the joke that a PR with 10 lines changes got 10 comments (all nitpicking), a PR with 1000 lines got just one, LGTM. We’ve had several bugs where PR review would almost certainly never catch the issue, people are terrible at reviewing things thoroughly. Guess what can do that much much better?
It's not code review that's slowing things down. It takes time to get the thing you want, and then you have to be reasonably sure that it's right anyway, or you risk shipping a piece of crap to production and tanking your own reputation. Even on teams that rubber stamp PRs I see people pushing drafts and waiting to merge until they feel comfortable with what they've vibed out.
As a senior engineer that is struggling to give up line-by-line reviews, I agree with your take.
I've spent literal decades where one of the most valuable practices has been code reviews, for many dimensions. It is one of the best ways to ensure quality production releases. It is one of the best ways to transfer skill to junior engineers. It increases tribal knowledge throughout the engineering org by exposing others to systems they may not be working in. The number of benefits to a traditional human-centric engineering team is hard to overstate.
But I'm starting to feel that instead of starting at one extreme (heavy handed reviews) and moving towards less review, it probably makes more sense to start at the other extreme (no review at all) and move towards a brand new practice tailored for the new agentic reality.
It is looking increasingly likely that we don't need to modify code review for the agentic era, we need some completely new practice that hasn't yet been invented.
It's really about hamstringing the AIs so they can't write crap. Writing code is not a problem anymore, so anyone with an idea can write a script to do what they want. But if they try to connect two ideas together, now they have a data modeling problem. This is where everything seems to fail with AI coding (largely because the original author had the idea piecemeal).
The problem I have is non technical people now started submitting PRs. They are like the brain damage module on GLaDOS and somehow make opus generate actual broken slop. So I still have to be on my toes to catch that, but they generate such volumes of broken PRs that it takes time away from me doing real work.
Perhaps spec review in the custom of code review - it can even be via PR - so there is a bottleneck before anything is generated, instead of after. Cheap-to-produce code maybe isn't worth understanding line-by-line anymore.
I think spec review, which is definitely valuable, solves a different problem. One question to ask is "are we building the right thing", which a spec review definitely helps to answer.
But another question is "did we build what we planned, and did we build it to a high quality". Code review partially answered that question. Adversarial review is the closest we have to that in the automated LLM world, but I feel we need something better.
As a senior engineer, I think the code quality and review rigour slippage came well before AI. It's cultural, not technical. People just don't want to hear criticism anymore, they're thin skinned and weak minded, and they don't want to do the work to become strong. They want to be handed the answers and told they're doing great work, regardless of whether it's actually true.
Generative AI coding tools have just amplified this problem.
To be perfectly honest, whenever I detect this pathology in a junior my effort reviewing their code drops precipitously. There's just no point investing in an individual who doesn't want to work hard and learn. If my employer wants me to green light their vibeslopped garbage then I'll gladly oblige. Unfortunately this is the norm now.
Agents are fully capable of doing reviews as reliably as senior developers. They just need to be independent and adverserial, and validates the things you actually care about in your QA.
I lean towards a hybrid review process where humans accept their limitations, focus what they're good on(higher level concepts and architecture of the proposed change), and let the LLMs do the rest.
Doing a line by line review as a human is silly now.
If you're right (and you probably are), we're in for a wild ride further down the quality death spiral.
The era of user telemetry and conversion tracking already decoupled product design from customer experience. No need to wait for customer's to articulate their experience or desires or for discerning analysts to amalgamate and ponder their opinions. The numbers tell you everything you need to know! Just throw 3000 pounds of spaghetti at the wall every month and wait to see what sticks.
But now you really were bottlenecked on engineering.
Only 3000 pounds of spaghetti per month? And throwing more bodies at it barely even makes a difference? Oof.
Of course the solution is to just flush pureed spaghetti-slop through as hose as fast as you can. Brilliant!
What could wrong? The users are sure to love it. We'll know because they keep clicking. That's how they tell us the truth, even as every forum fills with louder and louder complaints!
> I think if there is a major change in code review culture, it's not going to happen first in established engineering teams. It'll happen in startups that adopt a no-human-review policy from jump and then maintain it as they mature.
I've been hearing this for more than a year. But if these startups are all moving at superhuman speed, a month instead of a year from idea to a shipping product, where are the products? Every time I ask that, people talk about the amazing velocity gains they experience for the thing they are about to ship. It never does, unless we count throwaway hobby projects at GitHub - by that metric, AI is adding a lot of software. And I guess never-downloaded slop-games on mobile app stores.
But if you actually go shopping for consumer or enterprise software that's camera-ready, the landscape is basically the same as in 2022.
Codex/CC and other OpenAI and Anthropic products like Dots, are examples of how quickly real products can be built today. But to me, they feel more like mediocre products bloated with feature slop. Nothing extraordinary or technologically innovative.
Pi.dev and Cursor's Tab completion are two of the most innovative and extraordinary products I know in AI era, and both were built by humans, at least in their "breakthrough" stages.
> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
Why do you feel like that? Agents have got better but better how? Looking at the frontier model output, I’d say the same myopic approach that causes long term issues without frequent course correction is still there.
And test creation and maintenance. AI-generated tests will often embed existing bugs as testable patterns that fail when fixed or rely on empty asserts to increase coverage. A human author generally knows better. Coverage is increasingly becoming theater and runtime testing eating up CI resources and time with bloated, unnecessary, untargeted testing that hasn't been mutation tested or had test oracles implemented are another drag.
It's really the smaller firms which are able to learn quick adaptations that will win out in the long-run of this thing. Legacy firms with institutionalized bureaucracies are relying on gobbling up competitors and creating vertical monopolies while pushing out the worst slop - it's why GH goes down, why Meta products are buggy as hell now, and everything feels like it's unpolished MVPs these days.
This. The ability to know that a vibe coded PR is all right to merge, is a function of both the thoughtfulness and simplicity of the underlying architecture, and the level of ownership that contributors have of the underlying products. Both benefit more nimble teams.
So far this year, of the defects severe enough to warrant a formal RCA process that I’ve dealt with, “AI generated tests asserted that the defective behavior was correct,” was a contributing factor in every single one.
I haven’t had this happen yet, but I was just thinking about this last night. I would be very interested to see if exercising TDD, as difficult as that practice is to actually do, would have a meaningful positive impact on improving this phenomenon. Usually agents model what most humans do, writing tests to pass and assert that the code you just wrote works as you wrote it.
I’ve had decent luck with it as long as I at least stub out the test cases with a test name and a “given … when … then” comment.
Otherwise it seems the AI has a tendency to write the worst kind of TDD antipattern microtests that simultaneously make refactoring impossible while failing to document or even confirm the most important functional requirements.
Once it’s got those stubs it does do a pretty good job of suggesting behaviors I forgot to specify, and other improvements. It’s just “designing” test fixtures from the ground up that seems to put it into “generate plausible-looking slop” mode.
It has the same failure mode as dealing with TDD by a human engineer, which includes things like mocking the function under test in the test and then testing the mock, asserting only things that aren't actually exercised by the function under test (like asserting the initial user ID is the same as the final user ID for a function that changes name of a group), not validating side effect behavior (or lack thereof) for a function that's only purpose is the side effect (like appending a log line to a log file), and more!
What changes with TDD is that the important part of the review moves from the mainline code to all the TDD tests and the coverage, and you also need to see a test run from before the code changes with all the new/updated TDD tests failing in addition to just a test run after the changes with all the (at least) new/updated TDD tests passing.
This is part of the reason that I've come to view mocks as a code smell. They're too easy to screw up.
But even more than that, I've found that modules that need to be tested with mocking can often be reorganized to be testable without mocking. And doing so tends to also make the code easier to understand and easier to modify without breaking things in the future.
Mocks/stubs definitely have a smell to them, but especially if you want to do more than just unit tests can be necessary. I've found that setting up the mainline architecture so that components that are allowed to be mocked have a single access path in the code makes it easy to mock at a level low enough that the only thing being mocked is that component and then preemptively building out those mocks into the test framework works decently well. For me, 3rd party services are usually fine to mock, but things like DBs and redis are usually not. I'd rather spend the time to make a test container the devs can run locally so that they can't skip over checks/constraints and the integrations with services/components we have full control over.
That way you can tell at a glance of the test file change just how bad it is. Seeing new raw mocks (eg sinon in the test file at all) instead of the existing mocks is an instant no, and seeing an existing mock getting added requires at least investigation for necessity.
Agentic code review can also be employed, but it's worth noting that, as the so-called "Bitcoin Red Team" (which has been testing all of Bitcoin related software in the wake of the July 30 Coldcard attack) has discovered a very useful thing to consider: American models are absolute trash when it comes to reviewing code with an eye on security, due to their "safeguards." This applies even when the code being scanned for vulnerabilities was just written by the same agent.
Kimi K3 seems to be the favorite there, though I believe some of the other open weight/open source models are being employed a bit as well.
And yes, even that requires human oversight -- but it does a quick rough filter through things so you're not wasting time on the little stuff, allowing you more attention to find the glaring holes.
Both Claud and Codex are better than 80% of the developers I’ve met through my 20 year long career. This whole code review thing is a hold over that probably won’t last six months.
> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
Our industry has spent the better part of the last two decades interviewing with brain teasers and leet code questions. Reading code is a skill that we just weren't assessing for, and lots of people aren't really good at it.
I would argue that code review is mostly theater, politics, bike shedding and pedantry.
There is a lot of work to get away from "coding" and "programing" and get down to the actual engineering that really needs to happen in the industry.
IME, the biggest source of rework cost that happens at (or after) code review is when someone notices that the code, correctly or not, implements the wrong thing. For example, the feature, as implemented, doesn’t sufficiently solve the original business problem. Or it solves it in a bad way, like because of overengineering or taking on unnecessary technical debt. I haven’t really seen much evidence of AI code review catching either of these problems.
I’ve heard people complain that human code review doesn’t reliably catch them, either. And I have seen that with my own eyes. Just not universally. Teams that have a strong culture of shared ownership and collaborating on designs regularly catch these kinds of problems in code review. If they’re really good at it, they catch them even earlier. e.g., I know planning poker isn’t fashionable anymore, but when I’ve been on teams that take it seriously they tend to be pretty good at finding minimally expensive, correct designs pretty early, and design flaws that do make it to code review tend to be smaller and less expensive to fix. Also those teams didn't tend to bottleneck on code review in the first place. It tends to go pretty fast if you come to it already knowing what to expect.
I think that this is the kind of thing that led to folks like DORA observing that AI coding is something of a multiplier for the effects of both good and bad software development practice. If you’ve got a cowboy coding culture, throwing AI into the mix will probably just lead to agentic cowboy coding.
> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
Interesting framing that casts senior engineers as the impediment. If the models are now good enough now, perhaps engineering leadership could[1] do away with code review entirely - or maximally automate code reviews. I'm sure seniors would love to get their individual productivity unleashed for performance metrics/bonus reasons.
1. I'm stating this as a hypothetical, but this "vibe-coding your core product" experiment has been repeated many times and I'm yet to hear a single success story
Imagine no code review: poof. You still aren't shipping at lightspeed because it still comes down to knowing where to shoot. Oftentimes this is like pulling teeth from various involved stakeholders.
You say small startups would be faster as they won't be doing code review. There's probably always been startups running without bothering with code review for as long as this has been a field. Bad practice? Sure, but all that matters to the business case a lot of the time is just shipping something that works alright, not the greatest piece of efficient code in the world. Moore's law has bought us a lot of slack in that.
AI is great at reviewing code. It is not great at reviewing intention. If the prompt was bad, not understanding or thinking about how to implement the feature properly, the review usually doesn't point out exactly what's wrong. This can lead to a sort of "polishing a turd" approach to code reviews where you patch some of the negative side effects for an implementation that is completely wrong.
This is a result of AI still being bad at telling the prompter "no". Not only because it will blindly reproduce what you tell it to while writing code without pushing back on assumptions without asking, but because it will often blindly accept an implementation approach while reviewing.
I suspect this is a hard problem, because if AI stops listening at the wrong time it can be a massive inconvenience.
I have noticed that AI is slowly getting better at pushing back on different types of things each time the frontier models get better though. I've also noticed on reviews that it will tend to hallucinate or nitpick around the problem. I assume it sees something wrong there, starts generating output, and points out something different there instead because there is pressure to be both agreeable and helpful, which are not always compatible.
I think it is getting better, but it can't always see the forest for the trees.
I think senior engineers find it very difficult to give up on line-by-line review
Thanks, hearing that renewed some of my hope for the future!
I'm definitely in that camp. The tools need to be as reliable as my compiler at faithfully expressing my intent.
Two inherent headwinds:
Their non-deterministic nature makes this difficult for me to trust.
And the imprecision of the human language.
Elaborating on the second... Source code can be a handy shorthand. It was designed for the purpose. If my level of abstraction (or "programming language") is the English language, it's going to be a lot more "code" to navigate.
That demands highly structured organization, reliance on diagrams, methodical vigilance for edge cases which may not be as readily elicited, etc. Admittedly, design hygiene that should already be the case if you're diligent, but I've seen how those areas can be a weak point in practice - particularly in some corporate environments where the developers are stretched thin, and as a result the culture sacrifices discipline.
In other words: Exactly the place where everyone says AI is supposed to be a revolutionary force multiplier.
I agree with you organizations who leverage the tool most effectively will be the ones who build or reinvent themselves across more than just the engineering teams, to avoid some of the habits human nature tends.
In the meantime I'm very keen to see how long term maintainability pans out for the codebases being churned out today (and fully expect as commodity code becomes cheaper to produce, we'll see it become much more disposable - just throw the whole thing out and start from scratch).
Of course I'd love to hear thoughts on this from people who've spent more time behind the wheel of these tools than I have!
You dont have to review the code, but someone does. If it is a well-tuned adverserial reviewers, that would be more than sufficient. We are building zeroshot to let you customize that as a fixed graph for your repo, feel free to check it out!
All of these model-based approaches are fundamentally nondeterministic systems, and there's no escaping the math of their fundamentals, which is textual representation of natural language.
The "nondeterministic" part is enough to flag these tools in some industries. We can't risk having a different answer to the same question in some applications. Even if the differences are small, differences compound over time, as anyone who's worked in inertial guidance can tell you.
Aside from the "nondeterministic" part, that "Fundamental Math of LLMs" is a very different beast from math that runs much of the rest of the cosmos. It's just human language. Just because you can talk about something does not mean you understand it.
Existing "LLM-type" technology performs beyond all prior preconception because there's pretty obviously some kind of basal mathematical relationship between different kinds of meaning. But that relationship, no one's really grappled with it formally in its entirety. It's probably adjacent to the big new science/mathematics of Emergence. It's also, from a pure science aspect, probably the most interesting aspect of these systems. All the crap we've been trying to wrench - for DECADES - with taxonomies, semantics, a swirling universe of computational linguistics in all its variety - that crap could conceivably have a single " $ E = mc^ 2" moment. But we don't fundamentally understand it, and it's a real mistake to hand critical systems to models that aren't understood. Down that path alchemy lies.
We havent done manual code review for months actually. It is not a problem if you have adverserial review loops that automate your QA and mechanically enforce the orchestration of them. That is basically what the zeroshot framework we are building is for.
Wrong thing to index on. Always bet on the slope, not on points in the slope. Things will only get better and better and factor in that downward slope thought while taking actions and pick ones that are favorable. The folks who focus on a point in time will be left behind, period.
As is standard for academic papers, this is an attempt to describe the state of the world as of when they did the research. But then it takes time to write it up and get it out, after which newspapers report it as current events.
In this case, we're looking at a picture from 2025. The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5. And the paper's design means you're mostly looking at the very initial effects of deployment, before companies developed good review processes.
They've updated it for the agentic code era, and have found basically the same thing. Code can be created faster but that has moved the bottleneck to review.
The data is still mostly 2025 and the 2026 data is mostly from the first few months, and most production deployments (at least in, say, the Fortune 500) lag the frontier by at least several more months (or much more). I suspect it's just not really possible for academic research at its conventional pace to offer an up-to-date perspective on this topic right now.
> I suspect it's just not really possible for academic research at its conventional pace to offer an up-to-date perspective on this topic right now.
If we weren't still in a massive hype-and-investment bubble, the burden of proof would be on the LLM manufacturers to demonstrate what deficiencies that studies like this have revealed have been fixed in their latest products.
I mention this because I've been hearing "Yeah, well, that opinion/study is completely invalid because it used a six month old tool. Today's release of SOMNOS 6.9 (Pinnacle Snoozytime) has totally changed the game!" every six months for what feels like the past five years now.
> I mention this because I've been hearing "Yeah, well, that opinion/study is completely invalid because it used a six month old tool. Today's release of SOMNOS 6.9 (Pinnacle Snoozytime) has totally changed the game!" every six months for what feels like the past five years now.
Given my own conclusion that it's not possible for conventional academic research to accurately snapshot industrial adoption of this technology, I would expect this pattern to continue for a while until either academia becomes much faster or technology progress slows down (or both). This just isn't yet a situation where you can rely on published papers telling you what to think -- you still need to mix in a bit of your own instinct and experience and extrapolation to get an accurate up-to-date read, even if there's some prediction error as a result.
> This just isn't yet a situation where you can rely on published papers telling you what to think...
Well, like I said: If we weren't still in a massive hype-and-investment bubble, the burden of proof would be on the LLM manufacturers to demonstrate what deficiencies that studies like this have revealed have been fixed in their latest products.
Today I can write what I'm looking for, as if I'm writing a design doc, and the models can do the rest. A year ago it was faster for me to just write the code myself. Models today are enormously better than what we had in 2025.
I almost feel like some type of recency fallacy is continuously used by those defending LLMs. You are holding it wrong, or it is not the latest yet to release model.
Then again, I doubt they allow invalidating any person who previously used that defence...
Along these lines: agents will write mountains of super repetitive test code, seemingly uninterested in refactoring it (without explicit instruction) to reduce redundancies.
In any non-trivial integration test scenario, it's easy to end up several times the LoC in tests compared to actual shipped code.
Second problem: different models, agents, and harnesses seem to have different tendencies and they tend to avoid touching code that is not their own unless explicitly instructed. What I mean by this: it will happily duplicate some helper method from another test file rather than pulling it out and reusing it, seemingly avoiding coupling (good kind in this case since model level changes should be propagated atomically).
Tests are where I see agents just spew absolute mountains of code.
When people have gotten excited about having so many tests, I've asked them what their plan is when 500 of their 1000 test suddenly stop passing. I've always favored 50 tests I know are meaningful, where any failure is something to respond to, over 500 tests with a bunch of overlap and edge cases I'm not sure matter. But now people aren't even thinking about that, they're just letting the agents make all those decisions.
I think this approach will continue to work for some people and orgs, but others are going to hit walls where their agents can't manage the mountains anymore. It's going to be interesting to see how people and orgs respond when that happens.
Every senior I've ever worked with at some point embraced the idea of minimalism. Produce minimal changes with a minimal surface. Worked well at startups and especially well on large spaghetti legacy projects. Seeing all of a sudden now how suddenly number of kLocs, PRs, token maxxing, number of tasks in progress is again a metric with which non-engineers brag about on engineering conferences feels like the industry has reverted to some braindead pre-infant state. I guess all mistakes need to be relearned at every hype cycle.
This doesn't solve the issue with "too many tests", but one of my teams addressed this to some extent by writing "blueprints" for test scaffolding that specifically aims to surface the scenario under test.
Was initially hand written as part of our codebase before AI, just extracted it as MIT, OSS since there's no secret sauce here. But really cuts down on test verbosity and makes it much, much easier to read, review, and evaluate tests.
I'm completely uninterested in the opinion of a video game journalist on what does and doesn't work well in writing software. The author knows about as much about writing code as he knows about performing heart surgery
Ars Technica in general has also been catering to their knee-jerk anti-AI audience. I used to find their comments section somewhat interesting but now it's the same old tired, outdated dunks on AI.
Way back in the day, everyone wrote assembly. Then things like C came around, which had to be compiled to assembly, and most people objected: "I don't want to be that far from what the CPU is doing!".
Each time we move up an abstraction layer, people object, saying that nothing will work right if we move up that high in the stack. Give it time. I'll bet in 5-10 years, almost no one will look at code anymore, except in special cases.
We still have people who write in assembly, we still have people who write in C, mostly for performance reasons. Because you still need to be close to the CPU to get that last bit of performance. But in most cases, it doesn't matter. It doesn't matter if the LLM makes bad code if it works and passes all the tests.
Granted, this is the first time we're going from deterministic to non-deterministic, but is it really? Large scale distributed systems are already non-deterministic. They create inputs and situations that programs aren't designed to handle all the time. What's a little more chaos? :)
I’d strongly disagree with one aspect. Homemade personal use software has Definately skyrocketed. Projects I never thought I’d even have the time for are now possible.
Yes, I am literally inundated by cool new tools I've built that I never would have before.
I'm literally building a black-box client + server emulator for one of my favorite old MMOs, remotely, from a coffe shop in SF as I type this, on my laptop at home.
Yes, Claude is building it for me, and now I have the tool to use forever. Point being that there are now many more tools being made, which is contrary to the message of the article linked.
That all aside, it's hilarious you apparently don't think managers or architects are actually contributors to sw dev.
My homelab has many fancy new things like a self-hosted gym app and many more metrics and dashboards in my home assistant instance, both of which I use on a daily basis. It also now runs a linux VM that is an always-on agent workspace, and various emulators & decompilers and their MCPs. I, like apparently everyone else, am also reverse engineering games and mashing them up for my own personal enjoyment. I've modded a few games for fun. I've done a security and utilization audit of my server and saved hundreds of dollars in server upgrades I would have bought if I didn't have AI identifying waste and optimizing things. I could have done all of those things before but I didn't, because of how annoying it is to have to dive so deep into the weeds to do small things.
People who have the technical skills and free time for these hobbies, and aren't having the times of their lives right now, are missing out.
Abstaining from using LLMs isn't doing anything to actually oppose them, it's just denying yourself the stuff you could produce with them.
The game RE stuff is also NUTS right now; the kind of stuff you can do in terms of porting, modding, emulating, etc, is a whole new frontier in personal game ownership.
Like, if you have a copy of a game downloaded, you can prevent it being taken away now. Including by devs turning off remote servers.
I wonder how much of this is "cool new tool I want to try and play with" vs genuine technical capabilities? I'm guessing you know how to write software already. I think part of what might be happening is the inertia of starting with a project seems to be lifted. It is almost like instant gratification. I'd rather have the cup of coffee in front of me vs having to up and make it sort of thing.
But I wonder how much of that putting off was actually a real assessment of how much time and effort a project would take, vs a sort of feeling or assumption that it would be tough, and perhaps avoidant behavior as a result where the task is constantly backburnered?
This is how it is with my personal projects. Hard to decide to sit down and started but once the ball is rolling, I pretty much get the entire thing done in one sitting, learn something and feel satisfied and accomplished. Never takes too much time as I assume. But boy to get started is the real tough part sometimes.
I am not a software engineer, but today with two prompts I made a cute sign/coloring book making application for my kids. It’s got zero test or code reviews and works perfectly. Traditional software for consumers is a dead man walking. Corporate will only be behind until they can figure out how to automate certifications and regulations.
They generate more software too. It's just organizations haven't adapted to remove the bottlenecks yet. I replaced so much software for my self and its better than all the alternatives I used to use. And there are tons of other people doing the same thing.
I read this paper several days ago, when my director shared it. Several things jumped out at me:
- the data is mostly from 2025 and earlier
- the baseline mean lines of code per month was 5017, the estimated increase was 1494 lines per worker-month
- the baseline is 5.34 PRs per month, and increased by 1.22
- the baseline mean for code review is 7.03 days
The second and third bullets are confirmation that this is old data. In late 2026, people aren't delivering 30% more lines of code. If people were just producing 30% more lines, they'd still be reading every line. My point isn't that "more lines is good". I'm worried about a few too many lines of code that no one reads. My point is that this shows the data is old.
For the latter, I'm a bit annoyed by using mean, median would be more valuable, but I think it's safe to say that the engineering culture was already pretty bad. If your average pull request sits for 7 days, you evidently don't care about shipping anything, and agents aren't going to change that.
Aside from other comments mentioning flaws and cutoff of study, anyone in the industry or even working on hobby products can say this conclusion is wrong.
Given what I've been able to accomplish with even just local models at home, and my experience at my day job, I imagine this has almost everything to do with the organizations themselves and less to do with using AI to build software.
But the longer the codebase survives the more hair and cruft it accumulates. My once beautiful logical paths get carved up by compounding requirements and edge cases.
That's what the AI was trained on. The end result - the hairy ball of wax that noone can look at sideways.
I often think of that whenever AI comes up with what I consider to be an irrational but plausible edge case.
Man, I'm probably going to expose my own inexperience here, but like, did anybody actually expect code reviews to catch and prevent bugs on a line by line basis? Like ever?
I never have operated this way. Code reviews were a basic sanity check, a chance to implement best practices, and a chance to teach these things to juniors.
If I was concerned about something breaking, I'd duscuss test coverage.
It was more important for giant C or fortran code bases. You will legit catch pointer errors or memory mgmt stuff, going line by line. Once Sun busted in with Java is when I really saw the shift to skim reviews. So much stuff is caught by compilers now, or SAST, that using dev time to find code smells is mostly a waste of their time. And it's not like the old way caught everything, it's just that the stuff we take for granted being caught by compilers or Sonarqube now, just used to get shipped to prod and cause abends and production support calls at 3am.
I’ve caught bugs in code reviews before, but I would agree that it’s probably not the most crucial thing you should strive to do with code reviews.
That said, I think I’ve done a tremendous amount of PREVENTING bugs from being introduced via code review. Caught all kinds of weird code patterns, duplication, wrong levels of abstraction, etc. Essentially, the kinds of places that bugs like to hide and grow in.
I still think that’s valuable, and it’s the kind of thing that models generally aren’t great at, to this day. It generally relies on having a large amount of context about the project and about what’s being solved, so if you wanted to do this via automated review, you maybe COULD, but it would probably get pretty expensive, especially on larger projects.
At another level, code reviews also give you the time to really think about what is being built and why. I've often come across looming architectural or strategic failures simply by having to stare at a bunch of code and wonder why it works a certain way. Same goes for writing the code by hand too, though it's harder to see past your own biases and ideas sometimes. I find code reviews deeply valuable for myself and my team.
So - I've done this just by reading LLM code. I mean, yes I'm largely looking for high level architectural smells and bad practices, but I feel like "reading the code" now takes on a lot of the previous roles that code reviews may have done.
I just don't do this at commit time. I might make 100 commits and then dig through code and find some bad architecture stuff.
It's very useful, and actually kind of critical for the way I find code reviews valuable. If the person hasn't reviewed their (agent's) code before I get to see it, then I'm not really talking to that person about their architectural or design ideas, they're just the 'meat proxy'. The agent isn't going to develop deeper opinions about the project and product, and neither is the person if they didn't read the code before review.
It's these kinds conversations and constructive challenging of ideas that I worry for the most when orgs talk about reducing code review bottleneck.
> Essentially, the kinds of places that bugs like to hide and grow in.
I like an ecological-framing, especially as systems get more complex.
Yes, I could hunt cockroaches all day, but it would be even better to have caught how the new wall was missing a metal screen over a low ventilation opening.
I agree with this but the point that I'm making is that there's a huge delta between "I have prevented bugs in code review" or "I have implemented best practices in code review" to "Code review is the line by line gate which prevents bugs"
I review code line by line and build a mental model of the software as I review it. I find plenty of important bugs that way. PRs are an opportunity to learn (both committer and reviewer), communicate, and improve the quality of code a team ships. Sadly, many people decline that opportunity for the sake of short-term velocity.
Where I work we have a culture of very careful code reviews, often infuriatingly so. But we catch bugs in code review all the time. Usually with an admonition to add a test for the buggy case.
I think a code review should be just that. A review of the optics of code itself not the solution or anything like that. If that is important it should be done earlier in the process before coding. Even more important when coding with AI. The reviewer is not a tester. I've heard horrible stories from friends who mainly works with fixing AI generated code that junior devs pushes at a rapid pace.
Yes I have often caught stuff like "not using DB transactions in the right way" or "this is not going to be race safe" for instance from juniors just by glancing on a PR.
I find recent LLMs don't do such mistakes though, in my experience. And also LLMs will find such mistakes if asked to review, if not perfect then more predictable than humans.
No. You are on to something here. The utopia I see programmers lamenting the loss of, the one where human code review was perfect and never let any bugs slip through, never existed. I see people on this board and others complaining all the time about LLM generated spaghetti code and bug-ridden codebases. Well, I've been working those code bases for 20 years. We didn't agents to make them. LLMs are a new tool, but the problems that come with them are nothing new.
No. That's just the socially accepted norm and so people fight for it because they don't know any better (and a lot of these guys have egos the size of a small nation that leads them to "exert dominance" due to their insecurity problems—they're also frequently wrong).
Turns out just using the damn thing and testing what you've built on your own can solve a ton of bugs.
This headline being in the present tense, while reporting on a data from 2025, makes it basically dishonest. 2026 AI coding agents bear little resemblance to 2025 AI coding agents, and early-2026 coding agents are not like like-2026 coding agents.
Yes, instead of reviewing 1,000 lines a day I now "review" 10,000 lines a day. My brain is not 10x faster, so either I spend 10x more time reviewing code, or I just opt to accept that AI is probably right based on the complexity of the problem. However I realize that if we just let the agents loose on the codebase, everything would speed up like 100x.
Invariably once certain AI models in certain domains reach "full automation" quality this will happen, until now the human eyeball bottleneck is holding a leash on millions of tireless superhuman coders who have just a few narrow but egregious contextual awareness issues.
My experience is that agents are decent in coding a feature, solving a bug - doing specific well defined task. They are very bad at judging UI/UX and user flows through app. Like really, really bad. They can quickly put together vibe coded app but make it usable takes a lot of extra manual work.
Did anyone solve this problem? Any good approaches?
I don't see how AI coding agents are going to create 'more software' until the PO's and PM's of the world step back and let developer's communicate directly with the stakeholders. Or the PO's and PM's could cross-train on AI coding agents.
We are still being stressed to create all stories with such fidelity that 'any' developer can pick it up and implement it in the timeframe allowed for the size the story was given. When the story is written in that detail, we may as well copy it into an AI agent and have it implement it. But we can't do that, so we have people who have SME write up story's that sit in the backlog for 2 weeks until a developer can bring it in, so that they can spend a day implementing it (really 2 hours with 2 hours of meetings and 4 hours of video games) because the smallest pointing size we can give is 1 point and people aren't allowed to take on more than 8 points a sprint.
How can we expect things to get done faster when the team that reports work up to senior management put's such strict limitations on our throughput?
Any study that wants to say anything about the impact of coding agents on whether more software is being created or not can't limit itself to what firms are doing that create a footprint of "work events — including GitHub coding activity, Jira issues, and Google Calendar events".
This is software that would not existed without coding agents: https://fluximp.org
Sure its a coding agent itself, but I would never have tried to create it manually as I would have expected it to be too much work to see it through as solo dev. There were no issues created anywhere, no calendar events, and there is very little github activity (I primarily used Mercurial on a private server; git is just for distribution).
I started it with Kilo Code and after a couple of months it was purely self-modified and has been since then. I was a bit worried that it wouldn't work on Linux and I'd either have to use another agent or use it running on my Windows box to modify the code on my Linux box to get it working, but it just worked so the streak of being self-modified has continue.
I want to make an indie game and I'm working towards that with the project I'm using to benchmark FluxImp. My first job was as game dev and I took a shot at indie game dev 20 years ago (failed; not fun). Its something I had long written off as too time consuming to even start, but with coding agents I no longer feel that's true.
All of the stories about non-software engineers making small software that is useful to them also seems unlikely to generate the kind of footprint that lends itself to being easily studied.
223 comments
[ 0.25 ms ] story [ 21.5 ms ] threadThe other 11 items in the list of 12 would probably never be hit in practice, but now that you know, you have to sort out how to handle each (even if that just means documenting, filing, or ignoring: you have to understand each of the disparate 12 issues first.)
https://www.youtube.com/watch?v=NnV_cWeoo5Q&list=LL&index=3
I think if there is a major change in code review culture, it's not going to happen first in established engineering teams. It'll happen in startups that adopt a no-human-review policy from jump and then maintain it as they mature. I'm sure that's happening, but it's likely invisible to most studies and surveys at the moment. If it proves successful, and such companies start shipping significantly faster, the rest of the industry will be forced to adopt.
If you take it off too early you end up with shit and will spend weeks fixing all these baked bad assumptions your project is built on top of. I spent weeks fixing some of this shit then the LLM revived a not insignificant portion of it in a later change because it got "confused" (its words) because some of the comments it previously made were left around.
I actually dislike traditional code review. In part this is because most of our code review tools are very cumbersome to use (GitHub's and GitLab's web interfaces in particular).
But also because humans aren't all that great at code review when it comes to correctness and security. Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.
The problem is that purely agentic code review has orthogonal problems, and typically remains subpar when it comes to judging architecture and design considerations, compared to human reviewers. Both false positives and false negatives also remain a concern.
My best results so far have been with agent-assisted code review in a proper IDE, helping you to quickly construct a mental model and to analyze the underlying logic (in addition to doing a first pass for correctness issues).
Code review is not really about correctness and security. We have testing and other verification process for these. Code review is about idea discussion mostly and correcting a few stuff that may have slipped through the author's initial review.
Your PR wants to change the software in some way. My review is to comment on how you've implemented that change, like did you miss some areas where you needed to change, does your description is actually what is happening (either a cursory reading or through testing), or are you doing some unnecessary change.
IMO, the initial reading shouldn't take more than 10 minutes. More than that (so a big diff) it should have been discussed upfront, and the design agreed. If it's an important area, a more careful reading should be done about the tests and relevant stuff.
But generally, if a review takes more than 10 minutes, there's some failure of communication going on (reviewer not apt to understand the change, big diffs that should have been discussed outside of the PR, obtuse code,...)
I also agree that merge requests should be short and understandable in 10 minutes, but IME this happy state often does not obtain in practice. It shouldn't, but it often does, whether it's mission creep, whether it's turning out that a change is more complex than originally imagined, or whether it is unavoidably complex and cannot be broken down further.
Part of the code review process is to verify that the writer actually did build the agreed design too though.
I actually believe the opposite: exactly the fact that a reviewer had to develop a mental model independently is why code review is and remains valuable (and why people use different models when doing agentic reviews).
The biggest value of doing code reviews is in ensuring maintainability of the code (1. someone else can understand it, 2. it is readable and 3. can be evolved in the future) — everything else is downstream from that.
But do they? In my experience, at most workplaces it's simply a meat linter phase with the occasional useless comment about style (which should probably be handled by proper CI/CD).
Like if you don't even check out the code (generalized 'you'), then they can hardly build up that model. And even checked out, viewed in its proper context, Naur's famous paper comes to mind - the author who wrote it is the only one who has properly built the theory (which might have included alternative implementations that failed but have no visible artifact).
I'm not claiming that there are no places that go absolutely above and beyond with their review sessions, but I would argue it's really rare - and AIs are great at figuring out some of the context the PR description lacks.
I've spent literal decades where one of the most valuable practices has been code reviews, for many dimensions. It is one of the best ways to ensure quality production releases. It is one of the best ways to transfer skill to junior engineers. It increases tribal knowledge throughout the engineering org by exposing others to systems they may not be working in. The number of benefits to a traditional human-centric engineering team is hard to overstate.
But I'm starting to feel that instead of starting at one extreme (heavy handed reviews) and moving towards less review, it probably makes more sense to start at the other extreme (no review at all) and move towards a brand new practice tailored for the new agentic reality.
It is looking increasingly likely that we don't need to modify code review for the agentic era, we need some completely new practice that hasn't yet been invented.
But another question is "did we build what we planned, and did we build it to a high quality". Code review partially answered that question. Adversarial review is the closest we have to that in the automated LLM world, but I feel we need something better.
As a senior engineer, I think the code quality and review rigour slippage came well before AI. It's cultural, not technical. People just don't want to hear criticism anymore, they're thin skinned and weak minded, and they don't want to do the work to become strong. They want to be handed the answers and told they're doing great work, regardless of whether it's actually true.
Generative AI coding tools have just amplified this problem.
To be perfectly honest, whenever I detect this pathology in a junior my effort reviewing their code drops precipitously. There's just no point investing in an individual who doesn't want to work hard and learn. If my employer wants me to green light their vibeslopped garbage then I'll gladly oblige. Unfortunately this is the norm now.
I want every other firm to get weak whilst mines stays strong and wipe them out clean.
Doing a line by line review as a human is silly now.
If you're right (and you probably are), we're in for a wild ride further down the quality death spiral.
The era of user telemetry and conversion tracking already decoupled product design from customer experience. No need to wait for customer's to articulate their experience or desires or for discerning analysts to amalgamate and ponder their opinions. The numbers tell you everything you need to know! Just throw 3000 pounds of spaghetti at the wall every month and wait to see what sticks.
But now you really were bottlenecked on engineering.
Only 3000 pounds of spaghetti per month? And throwing more bodies at it barely even makes a difference? Oof.
Of course the solution is to just flush pureed spaghetti-slop through as hose as fast as you can. Brilliant!
What could wrong? The users are sure to love it. We'll know because they keep clicking. That's how they tell us the truth, even as every forum fills with louder and louder complaints!
I've been hearing this for more than a year. But if these startups are all moving at superhuman speed, a month instead of a year from idea to a shipping product, where are the products? Every time I ask that, people talk about the amazing velocity gains they experience for the thing they are about to ship. It never does, unless we count throwaway hobby projects at GitHub - by that metric, AI is adding a lot of software. And I guess never-downloaded slop-games on mobile app stores.
But if you actually go shopping for consumer or enterprise software that's camera-ready, the landscape is basically the same as in 2022.
Pi.dev and Cursor's Tab completion are two of the most innovative and extraordinary products I know in AI era, and both were built by humans, at least in their "breakthrough" stages.
Why do you feel like that? Agents have got better but better how? Looking at the frontier model output, I’d say the same myopic approach that causes long term issues without frequent course correction is still there.
Code review is where many of the design & code quality problems are observed/highlighted.
It's really the smaller firms which are able to learn quick adaptations that will win out in the long-run of this thing. Legacy firms with institutionalized bureaucracies are relying on gobbling up competitors and creating vertical monopolies while pushing out the worst slop - it's why GH goes down, why Meta products are buggy as hell now, and everything feels like it's unpolished MVPs these days.
Otherwise it seems the AI has a tendency to write the worst kind of TDD antipattern microtests that simultaneously make refactoring impossible while failing to document or even confirm the most important functional requirements.
Once it’s got those stubs it does do a pretty good job of suggesting behaviors I forgot to specify, and other improvements. It’s just “designing” test fixtures from the ground up that seems to put it into “generate plausible-looking slop” mode.
What changes with TDD is that the important part of the review moves from the mainline code to all the TDD tests and the coverage, and you also need to see a test run from before the code changes with all the new/updated TDD tests failing in addition to just a test run after the changes with all the (at least) new/updated TDD tests passing.
But even more than that, I've found that modules that need to be tested with mocking can often be reorganized to be testable without mocking. And doing so tends to also make the code easier to understand and easier to modify without breaking things in the future.
That way you can tell at a glance of the test file change just how bad it is. Seeing new raw mocks (eg sinon in the test file at all) instead of the existing mocks is an instant no, and seeing an existing mock getting added requires at least investigation for necessity.
Kimi K3 seems to be the favorite there, though I believe some of the other open weight/open source models are being employed a bit as well.
And yes, even that requires human oversight -- but it does a quick rough filter through things so you're not wasting time on the little stuff, allowing you more attention to find the glaring holes.
Our industry has spent the better part of the last two decades interviewing with brain teasers and leet code questions. Reading code is a skill that we just weren't assessing for, and lots of people aren't really good at it.
I would argue that code review is mostly theater, politics, bike shedding and pedantry.
There is a lot of work to get away from "coding" and "programing" and get down to the actual engineering that really needs to happen in the industry.
I’ve heard people complain that human code review doesn’t reliably catch them, either. And I have seen that with my own eyes. Just not universally. Teams that have a strong culture of shared ownership and collaborating on designs regularly catch these kinds of problems in code review. If they’re really good at it, they catch them even earlier. e.g., I know planning poker isn’t fashionable anymore, but when I’ve been on teams that take it seriously they tend to be pretty good at finding minimally expensive, correct designs pretty early, and design flaws that do make it to code review tend to be smaller and less expensive to fix. Also those teams didn't tend to bottleneck on code review in the first place. It tends to go pretty fast if you come to it already knowing what to expect.
I think that this is the kind of thing that led to folks like DORA observing that AI coding is something of a multiplier for the effects of both good and bad software development practice. If you’ve got a cowboy coding culture, throwing AI into the mix will probably just lead to agentic cowboy coding.
Interesting framing that casts senior engineers as the impediment. If the models are now good enough now, perhaps engineering leadership could[1] do away with code review entirely - or maximally automate code reviews. I'm sure seniors would love to get their individual productivity unleashed for performance metrics/bonus reasons.
1. I'm stating this as a hypothetical, but this "vibe-coding your core product" experiment has been repeated many times and I'm yet to hear a single success story
You say small startups would be faster as they won't be doing code review. There's probably always been startups running without bothering with code review for as long as this has been a field. Bad practice? Sure, but all that matters to the business case a lot of the time is just shipping something that works alright, not the greatest piece of efficient code in the world. Moore's law has bought us a lot of slack in that.
This is a result of AI still being bad at telling the prompter "no". Not only because it will blindly reproduce what you tell it to while writing code without pushing back on assumptions without asking, but because it will often blindly accept an implementation approach while reviewing.
I suspect this is a hard problem, because if AI stops listening at the wrong time it can be a massive inconvenience.
I have noticed that AI is slowly getting better at pushing back on different types of things each time the frontier models get better though. I've also noticed on reviews that it will tend to hallucinate or nitpick around the problem. I assume it sees something wrong there, starts generating output, and points out something different there instead because there is pressure to be both agreeable and helpful, which are not always compatible.
I think it is getting better, but it can't always see the forest for the trees.
Thanks, hearing that renewed some of my hope for the future!
I'm definitely in that camp. The tools need to be as reliable as my compiler at faithfully expressing my intent.
Two inherent headwinds:
Their non-deterministic nature makes this difficult for me to trust.
And the imprecision of the human language.
Elaborating on the second... Source code can be a handy shorthand. It was designed for the purpose. If my level of abstraction (or "programming language") is the English language, it's going to be a lot more "code" to navigate.
That demands highly structured organization, reliance on diagrams, methodical vigilance for edge cases which may not be as readily elicited, etc. Admittedly, design hygiene that should already be the case if you're diligent, but I've seen how those areas can be a weak point in practice - particularly in some corporate environments where the developers are stretched thin, and as a result the culture sacrifices discipline.
In other words: Exactly the place where everyone says AI is supposed to be a revolutionary force multiplier.
I agree with you organizations who leverage the tool most effectively will be the ones who build or reinvent themselves across more than just the engineering teams, to avoid some of the habits human nature tends.
In the meantime I'm very keen to see how long term maintainability pans out for the codebases being churned out today (and fully expect as commodity code becomes cheaper to produce, we'll see it become much more disposable - just throw the whole thing out and start from scratch).
Of course I'd love to hear thoughts on this from people who've spent more time behind the wheel of these tools than I have!
Agents allow me to build cool stuff but atm they're far from good enough to not review the code.
The "nondeterministic" part is enough to flag these tools in some industries. We can't risk having a different answer to the same question in some applications. Even if the differences are small, differences compound over time, as anyone who's worked in inertial guidance can tell you.
Aside from the "nondeterministic" part, that "Fundamental Math of LLMs" is a very different beast from math that runs much of the rest of the cosmos. It's just human language. Just because you can talk about something does not mean you understand it.
Existing "LLM-type" technology performs beyond all prior preconception because there's pretty obviously some kind of basal mathematical relationship between different kinds of meaning. But that relationship, no one's really grappled with it formally in its entirety. It's probably adjacent to the big new science/mathematics of Emergence. It's also, from a pure science aspect, probably the most interesting aspect of these systems. All the crap we've been trying to wrench - for DECADES - with taxonomies, semantics, a swirling universe of computational linguistics in all its variety - that crap could conceivably have a single " $ E = mc^ 2" moment. But we don't fundamentally understand it, and it's a real mistake to hand critical systems to models that aren't understood. Down that path alchemy lies.
In this case, we're looking at a picture from 2025. The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5. And the paper's design means you're mostly looking at the very initial effects of deployment, before companies developed good review processes.
If we weren't still in a massive hype-and-investment bubble, the burden of proof would be on the LLM manufacturers to demonstrate what deficiencies that studies like this have revealed have been fixed in their latest products.
I mention this because I've been hearing "Yeah, well, that opinion/study is completely invalid because it used a six month old tool. Today's release of SOMNOS 6.9 (Pinnacle Snoozytime) has totally changed the game!" every six months for what feels like the past five years now.
Given my own conclusion that it's not possible for conventional academic research to accurately snapshot industrial adoption of this technology, I would expect this pattern to continue for a while until either academia becomes much faster or technology progress slows down (or both). This just isn't yet a situation where you can rely on published papers telling you what to think -- you still need to mix in a bit of your own instinct and experience and extrapolation to get an accurate up-to-date read, even if there's some prediction error as a result.
Well, like I said: If we weren't still in a massive hype-and-investment bubble, the burden of proof would be on the LLM manufacturers to demonstrate what deficiencies that studies like this have revealed have been fixed in their latest products.
What we have today is not meaningfully better than what we had a year ago, so that isn't relevant.
Then again, I doubt they allow invalidating any person who previously used that defence...
In any non-trivial integration test scenario, it's easy to end up several times the LoC in tests compared to actual shipped code.
Second problem: different models, agents, and harnesses seem to have different tendencies and they tend to avoid touching code that is not their own unless explicitly instructed. What I mean by this: it will happily duplicate some helper method from another test file rather than pulling it out and reusing it, seemingly avoiding coupling (good kind in this case since model level changes should be propagated atomically).
Tests are where I see agents just spew absolute mountains of code.
I think this approach will continue to work for some people and orgs, but others are going to hit walls where their agents can't manage the mountains anymore. It's going to be interesting to see how people and orgs respond when that happens.
https://github.com/zeeq-ai/zeeq-biome
Was initially hand written as part of our codebase before AI, just extracted it as MIT, OSS since there's no secret sauce here. But really cuts down on test verbosity and makes it much, much easier to read, review, and evaluate tests.
Each time we move up an abstraction layer, people object, saying that nothing will work right if we move up that high in the stack. Give it time. I'll bet in 5-10 years, almost no one will look at code anymore, except in special cases.
We still have people who write in assembly, we still have people who write in C, mostly for performance reasons. Because you still need to be close to the CPU to get that last bit of performance. But in most cases, it doesn't matter. It doesn't matter if the LLM makes bad code if it works and passes all the tests.
Granted, this is the first time we're going from deterministic to non-deterministic, but is it really? Large scale distributed systems are already non-deterministic. They create inputs and situations that programs aren't designed to handle all the time. What's a little more chaos? :)
I'm literally building a black-box client + server emulator for one of my favorite old MMOs, remotely, from a coffe shop in SF as I type this, on my laptop at home.
Insane.
That all aside, it's hilarious you apparently don't think managers or architects are actually contributors to sw dev.
People who have the technical skills and free time for these hobbies, and aren't having the times of their lives right now, are missing out.
Abstaining from using LLMs isn't doing anything to actually oppose them, it's just denying yourself the stuff you could produce with them.
The game RE stuff is also NUTS right now; the kind of stuff you can do in terms of porting, modding, emulating, etc, is a whole new frontier in personal game ownership.
Like, if you have a copy of a game downloaded, you can prevent it being taken away now. Including by devs turning off remote servers.
But I wonder how much of that putting off was actually a real assessment of how much time and effort a project would take, vs a sort of feeling or assumption that it would be tough, and perhaps avoidant behavior as a result where the task is constantly backburnered?
This is how it is with my personal projects. Hard to decide to sit down and started but once the ball is rolling, I pretty much get the entire thing done in one sitting, learn something and feel satisfied and accomplished. Never takes too much time as I assume. But boy to get started is the real tough part sometimes.
Put another way: "Projects I would previously have considered a waste of time now seem worthwhile."
- the data is mostly from 2025 and earlier
- the baseline mean lines of code per month was 5017, the estimated increase was 1494 lines per worker-month
- the baseline is 5.34 PRs per month, and increased by 1.22
- the baseline mean for code review is 7.03 days
The second and third bullets are confirmation that this is old data. In late 2026, people aren't delivering 30% more lines of code. If people were just producing 30% more lines, they'd still be reading every line. My point isn't that "more lines is good". I'm worried about a few too many lines of code that no one reads. My point is that this shows the data is old.
For the latter, I'm a bit annoyed by using mean, median would be more valuable, but I think it's safe to say that the engineering culture was already pretty bad. If your average pull request sits for 7 days, you evidently don't care about shipping anything, and agents aren't going to change that.
yolo works when I need to focus on higher priority, and make sure things just work (though not "great").
For the latter, I use it for more like spec-driven fashion with HITL. Basically everytime it ffinishes a task, I review and then move on.
Both appraoches work. just make sure your yolo slop isn't bad and passes stuff.
LoC don't mean jack shit.
I generate more beautiful code, yes.
But the longer the codebase survives the more hair and cruft it accumulates. My once beautiful logical paths get carved up by compounding requirements and edge cases.
That's what the AI was trained on. The end result - the hairy ball of wax that noone can look at sideways.
I often think of that whenever AI comes up with what I consider to be an irrational but plausible edge case.
I never have operated this way. Code reviews were a basic sanity check, a chance to implement best practices, and a chance to teach these things to juniors.
If I was concerned about something breaking, I'd duscuss test coverage.
Was I doing it wrong this whole time??
That said, I think I’ve done a tremendous amount of PREVENTING bugs from being introduced via code review. Caught all kinds of weird code patterns, duplication, wrong levels of abstraction, etc. Essentially, the kinds of places that bugs like to hide and grow in.
I still think that’s valuable, and it’s the kind of thing that models generally aren’t great at, to this day. It generally relies on having a large amount of context about the project and about what’s being solved, so if you wanted to do this via automated review, you maybe COULD, but it would probably get pretty expensive, especially on larger projects.
I just don't do this at commit time. I might make 100 commits and then dig through code and find some bad architecture stuff.
It's these kinds conversations and constructive challenging of ideas that I worry for the most when orgs talk about reducing code review bottleneck.
I like an ecological-framing, especially as systems get more complex.
Yes, I could hunt cockroaches all day, but it would be even better to have caught how the new wall was missing a metal screen over a low ventilation opening.
It's a good way to catch thinking errors, which are the insidious errors.
I find recent LLMs don't do such mistakes though, in my experience. And also LLMs will find such mistakes if asked to review, if not perfect then more predictable than humans.
Turns out just using the damn thing and testing what you've built on your own can solve a ton of bugs.
Invariably once certain AI models in certain domains reach "full automation" quality this will happen, until now the human eyeball bottleneck is holding a leash on millions of tireless superhuman coders who have just a few narrow but egregious contextual awareness issues.
Did anyone solve this problem? Any good approaches?
We are still being stressed to create all stories with such fidelity that 'any' developer can pick it up and implement it in the timeframe allowed for the size the story was given. When the story is written in that detail, we may as well copy it into an AI agent and have it implement it. But we can't do that, so we have people who have SME write up story's that sit in the backlog for 2 weeks until a developer can bring it in, so that they can spend a day implementing it (really 2 hours with 2 hours of meetings and 4 hours of video games) because the smallest pointing size we can give is 1 point and people aren't allowed to take on more than 8 points a sprint.
How can we expect things to get done faster when the team that reports work up to senior management put's such strict limitations on our throughput?
Don't take it seriously.
Efficiency is not your job, until you can 1) solve it, 2) present the solved and implemented problem to senior leadership in one fell swoop.
Corporate America is a giant slush fund and this is how it keeps going.
Be grateful!
The true professionals spend those 4 hours making snarky Hacker News comments.
This is software that would not existed without coding agents: https://fluximp.org
Sure its a coding agent itself, but I would never have tried to create it manually as I would have expected it to be too much work to see it through as solo dev. There were no issues created anywhere, no calendar events, and there is very little github activity (I primarily used Mercurial on a private server; git is just for distribution).
I started it with Kilo Code and after a couple of months it was purely self-modified and has been since then. I was a bit worried that it wouldn't work on Linux and I'd either have to use another agent or use it running on my Windows box to modify the code on my Linux box to get it working, but it just worked so the streak of being self-modified has continue.
I want to make an indie game and I'm working towards that with the project I'm using to benchmark FluxImp. My first job was as game dev and I took a shot at indie game dev 20 years ago (failed; not fun). Its something I had long written off as too time consuming to even start, but with coding agents I no longer feel that's true.
All of the stories about non-software engineers making small software that is useful to them also seems unlikely to generate the kind of footprint that lends itself to being easily studied.
> FluxImp is an AI agent harness
That is a tautology
I checked and it looks and works great!