Depends. Back in the day, I worked on development tools for a commericial avionics flight management computer. The headcount for embedded hardware/software verification was greater than for development.
That was never true for software development in other domains I worked in i.e. Fortune 500 enterprise tools, compiler/design tools, networking, and semiconductor.
This LLM as compiler analogy comes up here and there and it’s not hard to poke a hole in it.
If I generate a piece of code with LLM, and verify it to be correct; Can you trust the LLM output, with the same prompt, in a new session, based on my verification of my code with the same certainty that you’d trust LLVM output for the same source code in the similar environment, ran independently? If no, why?
Yea, maybe one day in the future, "the LLM prompts" will be considered the source code. But a lot has to happen first, namely the same prompt needs to deterministically create the same intermediate code. I wouldn't check in source code that compiles a different binary every time, and we should not treat prompts as source code while they produce different code.
I agree with you on this one; the compiler output remains the same if run the next day, but AI code might not. That is why you need a strong, independent reviewer who reviews the AI code and freezes it after verification, so that next time you do not have to ask the AI to regenerate the code.
I was talking to some friends about how CI/CD should be used in $current_year if you have fully committed to agents.
The industry is surprisingly slow to adapt and I'm writing two capabilities I'd like standardised in the industry.
I'd like standardisation of Agentic tests (I know, it sounds terrible) but these are tests that are broader and capture fuzzy intent of the system. Unit tests, e2e tests are focused and hard to maintain. An Agentic test (or a behaviour test) is where the agent understands the underlying behaviour intent and creates scenarios on the fly and executes them.
For example, I'd imagine that Claude Code goes through multiple Agentic tests where Claude itself tries to use the software using desktop control. This would capture a much broader range of bugs and its much easier to maintain because you only need the test to have a description of the intent and not all the hardcore plumbing required in e2e tests and unit tests (like mocks ughh).
I'd like a framework where I could simply have something like SCENARIOS.md and the framework runs the scenarios by spinning up the common environment required for the test. The industry is severely lacking in this kind of capability.
The next thing missing is Agentic deployment verification.
So image that you are deploying your service and you'd need to verify multiple metrics are in shape before you promote it to all servers. Traditionally this was done either by manual checks by the developer (tiresome and prone to errors) or just having alerts on a few key metrics (not extensive).
Concretely, I'd like the agent complete access to all my metrics and logs and have it make the judgement to either promote it or not promote it or notify the developer.
You'd argue this is not required, but in my experience, having this capability gives me a lot of sanity. There are greybeards who love to gatekeep their responsibilities and would argue something like "just have good unit tests!!11!!" but that's silly.
Why these are not becoming standard or even required practices is beyond me. I'm guessing that there's a cohort of developers whose mindset is risk aversion and they are not great at intuiting what an LLM is good at and what they are bad at. Unfortunately, a lot of arguments are just silly like "but hallucination", "but just have deterministic tests", "don't overcomplicate things".
I just don't understand the premise. If we're fully committed to agents, what does it mean to talk about a test being hard to maintain or requiring hardcore plumbing? "Hey Claude, can you please mock out Component X and write a test suite capturing behaviors A, B, C of component Y. Thanks!" When in the process of typing this would I go "ughh", and why would I prefer to receive a SCENARIOS.md as output rather than a test suite?
At some point the agent rewrites Component X, or X+Y, and now many or all of your tests that hit it directly are invalid and need to be rewritten. That introduces bugs, you're changing the tests and code at the same time, so you don't know if the new ones cover the same stuff as the old one.
I won't be "fully committed to agents" re: test-writing until they reach the point where they demonstrate that they're taking contract and component boundary stability into account in placing their tests. Currently I still need to intervene.
And the tests I point it at doing instead of the ones it does by default are currently catching more would-be or latent bugs than ever before, because we're changing stuff so much faster.
Agents won't make you better, quality-wise, unless you were in a lower-percentile of code-writing ability. But they'll make you a lot faster at raw change count. How good or bad that is depends on (a) the situation the product is in and the tolerance for issues/regressions, (b) how much existing judgment and best practices you can wrangle into position on top of those changes to make the path of least resistance for the agent be smoother.
I agree. IN addition I think many businesses/industries are still not aware of the full capability of AI yet, that is why they still do things the old way. I have several friends who work as QA. When they describe the things they do daily, I tell myself quietly that AI agents can do all of them now.
This is all good stuff to verify of course, but why would you not continually add the useful agent-created tests and verifications into a deterministic suite to ensure consistency of coverage?
For instance:
>Traditionally this was done either by manual checks by the developer (tiresome and prone to errors) or just having alerts on a few key metrics (not extensive).
Ignores the places where it's always been important enough to have automated checks on many things.
> Why these are not becoming standard or even required practices is beyond me.
It sounds like you haven't actually tried to do what your proposing then.
> tries to use the software using desktop control
This is not a solved problem yet. I've tried a number of solutions and none of them were actually good enough for what you're proposing. Don't get me wrong, I'm not saying its an unsolvable problem. Only that it isn't sufficiently solved yet.
A not-insignificant portion of HN has full blown AI psychosis. These are the devs churning out Windows 11 and all of the inexplicably broken functionality that has started plaguing software we used to consider reliable in the past year.
Also known as harness engineering and is written about by Martin Fowler, RedHat, and many senior graybeard architects. I was at the DDD conference in Denver a few weeks ago and there were discussions about extracting value from GenAI and harnesses were the happy path.
In my own works, I've been using a home grown harness called DevArch for about 10 months.
Speed replaces pragmatism by allowing design flaws to be refactored N times where in the past we’d design around tech debt.
DDD leads to modularity and smaller bounded contexts has a significantly lower risk of being incorrect and significantly higher ability to refactor quickly.
And no you still have to understand how to construct software. We’re just skipping the coding skills and pushing everyone to be architects.
There's a reason verification and code review shouldn't be done by the person who wrote the code, and those reasons don't magically go away because a code generation tool did it instead.
Of course, since AI will be doing the writing and maintaining, there's really no reason for that any more. Let the AI do it all and wait for management to realize they're paying senior salaries to people that are little more than entering prompts, a job that really only requires someone to be able to read and write English.
At least in the USA, blame (or at least responsibility for damages) will often fall on the customer. We’ve allowed Caveat Emptor to become this awful fundamental basis of the economy. We just cross our fingers and hope that The Invisible Hand of the market stops companies from putting out defective products.
One idea I am thinking through is that I might want to use a linter to restrict the AI to only use a subset of various languages which would not need as much review.
For instance, if I wrote a web front end that only allowed certain HTML tags and certain CSS properties and no Javascript, I could avoid certain classes of errors entirely.
Or for instance, in C, if you disallow dynamic memory allocation you also avoid entire classes of errors.
These are checks I think I can configure with linters and would give me more confidence that AI generated code did not have certain errors.
Unless it's a personal project that you vibe coded, any production code that is the primary asset to a real company should have a person who understands what gets deployed. The best way I know how to do that is to co-design it with the AI. Reading code to match a spec can get the job done but the structure can be all kinds of horrible, and the more it gets iterated upon with features only makes it sprawl.
Two signals I use: do I ever push back and find places where the draft version was incorrect, incomplete, or otherwise not up to par and do I feel comfortable if something goes wrong with in production and I have to debug it at any hour of day or week.
> .... With the same clever tools, skills, and harnesses we use as AI engineers to push code at breakneck speed, we can build elaborate testing and verification automations.
So, in short, AI is going to verify it?
Not picking on ads, I'm sure the tools mentioned are useful, but exactly how effective they are? How many bugs it has discovered/reduced (compare to tests + human verification)? And how much time it saved? Without these info, the article is not that useful.
Yes, these stats will be instrumental to know. Hopefully, some institutions in the public or private sector can run some experiments to answer exactly these questions. I would trust these results more if they came from a research lab as well. Until we are more sure of these facts, we will have to be babysitting the diff :p
If you ask an LLM to review your code, then it tends to write a lot of code for situations that will never happen. You choose the bits you need. It's actually pretty useful as a code reviewer.
If you don't review the review, my assumption is that it will add a lot of code bulk, which won't break anything now but that bulk could soon fill up context windows and/or require far more tokens in future when you want to maintain it. You'll most likely end up with a project that human's can't maintain.
Code bulk and future maintainability really feel like unsolved problems, and we have no idea what tools or costs will be this time next year.
Hey folks,
I'm the co-founder and CEO of Faros, the company behind the Speed Trap report.
Just wanted to add a nuance about the AI-powered code reviews. They definitely help with the speedup, but they still kick the can down the road, and we still don't really know the long-term impact of 100% AI-generated code reviewed only by AI.
Will be happy to answer any questions about the data or the methodology behind the report.
If you write it in a theorem prover you can have the LLM write proofs about the code quite easily. Lean is easily good enough to compete with javascript, so now I think it makes sense to switch to Lean.
55 comments
[ 3.5 ms ] story [ 69.5 ms ] threadIf the future is agentic, human review will be core bottleneck that needs to be streamlined.
That was never true for software development in other domains I worked in i.e. Fortune 500 enterprise tools, compiler/design tools, networking, and semiconductor.
If I generate a piece of code with LLM, and verify it to be correct; Can you trust the LLM output, with the same prompt, in a new session, based on my verification of my code with the same certainty that you’d trust LLVM output for the same source code in the similar environment, ran independently? If no, why?
- writes the code
- reviews the code
- designs it
- productizes it
- advertises it
- consumes it
The industry is surprisingly slow to adapt and I'm writing two capabilities I'd like standardised in the industry.
I'd like standardisation of Agentic tests (I know, it sounds terrible) but these are tests that are broader and capture fuzzy intent of the system. Unit tests, e2e tests are focused and hard to maintain. An Agentic test (or a behaviour test) is where the agent understands the underlying behaviour intent and creates scenarios on the fly and executes them.
For example, I'd imagine that Claude Code goes through multiple Agentic tests where Claude itself tries to use the software using desktop control. This would capture a much broader range of bugs and its much easier to maintain because you only need the test to have a description of the intent and not all the hardcore plumbing required in e2e tests and unit tests (like mocks ughh).
I'd like a framework where I could simply have something like SCENARIOS.md and the framework runs the scenarios by spinning up the common environment required for the test. The industry is severely lacking in this kind of capability.
The next thing missing is Agentic deployment verification.
So image that you are deploying your service and you'd need to verify multiple metrics are in shape before you promote it to all servers. Traditionally this was done either by manual checks by the developer (tiresome and prone to errors) or just having alerts on a few key metrics (not extensive).
Concretely, I'd like the agent complete access to all my metrics and logs and have it make the judgement to either promote it or not promote it or notify the developer.
You'd argue this is not required, but in my experience, having this capability gives me a lot of sanity. There are greybeards who love to gatekeep their responsibilities and would argue something like "just have good unit tests!!11!!" but that's silly.
Why these are not becoming standard or even required practices is beyond me. I'm guessing that there's a cohort of developers whose mindset is risk aversion and they are not great at intuiting what an LLM is good at and what they are bad at. Unfortunately, a lot of arguments are just silly like "but hallucination", "but just have deterministic tests", "don't overcomplicate things".
At some point the agent rewrites Component X, or X+Y, and now many or all of your tests that hit it directly are invalid and need to be rewritten. That introduces bugs, you're changing the tests and code at the same time, so you don't know if the new ones cover the same stuff as the old one.
I won't be "fully committed to agents" re: test-writing until they reach the point where they demonstrate that they're taking contract and component boundary stability into account in placing their tests. Currently I still need to intervene.
And the tests I point it at doing instead of the ones it does by default are currently catching more would-be or latent bugs than ever before, because we're changing stuff so much faster.
Agents won't make you better, quality-wise, unless you were in a lower-percentile of code-writing ability. But they'll make you a lot faster at raw change count. How good or bad that is depends on (a) the situation the product is in and the tolerance for issues/regressions, (b) how much existing judgment and best practices you can wrangle into position on top of those changes to make the path of least resistance for the agent be smoother.
Good tests are so easy to write with these models, and so easy to fix as the code changes it they are too closely tied to the implementation.
For instance:
>Traditionally this was done either by manual checks by the developer (tiresome and prone to errors) or just having alerts on a few key metrics (not extensive).
Ignores the places where it's always been important enough to have automated checks on many things.
It sounds like you haven't actually tried to do what your proposing then.
> tries to use the software using desktop control
This is not a solved problem yet. I've tried a number of solutions and none of them were actually good enough for what you're proposing. Don't get me wrong, I'm not saying its an unsolvable problem. Only that it isn't sufficiently solved yet.
Think embezzlement. Think someone with a political agenda who changes the homepage on the way out the door.
Having a single point of control is asking for trouble.
if there are many of them (e.g. unit tests), who generates those guardrails and who verifies their correctness?
if they are a few, then how can they cover vast product capability surface?
In my own works, I've been using a home grown harness called DevArch for about 10 months.
Visit https://genai.cornelson.net/ if you want a detailed rundown of my experiences.
https://martinfowler.com/articles/harness-engineering.html
https://milvus.io/blog/harness-engineering-ai-agents.md
https://devarch.ai/
I personally follow Domain-Driven Design principles and old enough to have designed software through the old Anderson Consulting Method/1 practices.
Some of the results of my practices are documented at https://genai.cornelson.net
Speed replaces pragmatism by allowing design flaws to be refactored N times where in the past we’d design around tech debt.
DDD leads to modularity and smaller bounded contexts has a significantly lower risk of being incorrect and significantly higher ability to refactor quickly.
And no you still have to understand how to construct software. We’re just skipping the coding skills and pushing everyone to be architects.
If you give up on trying to reduce the number of permutations you need to test, you're gonna also (even if not intentionally) give up on testing them.
We knew this, of course. But everything old is new again.
Of course, since AI will be doing the writing and maintaining, there's really no reason for that any more. Let the AI do it all and wait for management to realize they're paying senior salaries to people that are little more than entering prompts, a job that really only requires someone to be able to read and write English.
For instance, if I wrote a web front end that only allowed certain HTML tags and certain CSS properties and no Javascript, I could avoid certain classes of errors entirely.
Or for instance, in C, if you disallow dynamic memory allocation you also avoid entire classes of errors.
These are checks I think I can configure with linters and would give me more confidence that AI generated code did not have certain errors.
There probably are problems like that but you could come up with a set of imports that are "safe" for the AI.
Two signals I use: do I ever push back and find places where the draft version was incorrect, incomplete, or otherwise not up to par and do I feel comfortable if something goes wrong with in production and I have to debug it at any hour of day or week.
You mean load-bearing
> Simple: automated software verification
> .... With the same clever tools, skills, and harnesses we use as AI engineers to push code at breakneck speed, we can build elaborate testing and verification automations.
So, in short, AI is going to verify it?
Not picking on ads, I'm sure the tools mentioned are useful, but exactly how effective they are? How many bugs it has discovered/reduced (compare to tests + human verification)? And how much time it saved? Without these info, the article is not that useful.
If you don't review the review, my assumption is that it will add a lot of code bulk, which won't break anything now but that bulk could soon fill up context windows and/or require far more tokens in future when you want to maintain it. You'll most likely end up with a project that human's can't maintain.
Code bulk and future maintainability really feel like unsolved problems, and we have no idea what tools or costs will be this time next year.
Just wanted to add a nuance about the AI-powered code reviews. They definitely help with the speedup, but they still kick the can down the road, and we still don't really know the long-term impact of 100% AI-generated code reviewed only by AI.
Will be happy to answer any questions about the data or the methodology behind the report.