Historically, this kind of test optimization was done either with static analysis to understand dependency graphs and/or runtime data collected from executing the app.
However, those methods are tightly bound to programming languages, frameworks, and interpreters so they are difficult to support across technology stacks.
This approach substitutes the intelligence of the LLM to make educated guesses about what tests execute, to achieve the same goal of executing all of the tests that could fail and none of the rest (balancing a precision/recall tradeoff). What’s especially interesting about this to me is that the same technique could be applied to any language or stack with minimal modification.
Has anyone seen LLMs in other contexts being substituted for traditional analysis to achieve language agnostic results?
> The key phrase here is "think deep". This tells Claude Code not to be lazy with its analysis (while spending more thinking tokens). Without it, the output was very inconsistent. I used to joke that without it, Claude runs in “engineering manager mode” by delegating the work.
This section really stood out to me. I knew that asking GPT-5 to think gets better results, but I didn't know Claude had the same behavior. I'd love to see some sort of % success before and after "think deep" was added. Should I be adding "think deep" to all my non-trivial queries to Claude?
This is a hacky joke. No sane engineer would ever sign off on this. Even for a 1-5 person team, why would I want a probabilistic selection of test execution?
The solution to running only e2e tests on affected files has been around long before LLM. This is a bandage on poor CI.
All E2E tests should run before deploy. Probably on every commit on develop branch, even. But there really is no need to run all E2E suite on every PR. In this case, the failure mode of this system, where PR automation failed to flag a breakage, is acceptable if it’s rare enough, so probabilistic solutions are OK.
I don’t think you truly get the “best” of both worlds, because the rate of accidentally omitting a broken test and letting something slip into master is now non-zero (flaky tests aside). This is still a tradeoff. But maybe it’s a good one!
I do wonder if this is as feasible at scale, where breaking master can be extremely costly (although at least it’s not running all tests for all commits, so a broken test won’t break all CI runs). Maybe it could be paired with, say, running all E2E tests post-merge and reporting breakages ASAP.
I can appreciate the effort put into the goal of optimization shared in the post, even if I disagree with the conclusions. All of that effort would be much better directed at doing a manual (or LLM-assisted) audit of the E2E tests and choosing what to prune to reduce CI runtime.
DHH recently described[0] the approach they've taken at BaseCamp, reducing ~180 comprehensive-yet-brittle system tests down to 10 good-enough smoke tests, and it feels much more in spirit with where I would recommend folks invest effort: teams have way more tests than they need for an adequate level of confidence. Code and tests are a liability, and, to paraphrase Kent Beck[1], we should strive to write the minimal amount of tests and code to gain the maximal amount of confidence.
The other wrinkle here is that we're often paying through the nose in costs (complexity, actual dollars spent on CI services) by choosing to run all the tests all the time. It's a noble and worthy goal to figure out how not to do that, _but_, I think the conclusion shouldn't be to throw more $$$ into that money-pit, but rather just use all the power we have in our local dev workstations + trust to verify something is in a shippable state, another idea DHH covers[2] in the Rails World 2025 keynote; the whole thing is worth watching IMO.
As other comments have said, I'd prefer other solutions to get by all the tests to run faster. It would be interesting to see if it could be used to prioritise tests - get the tests more likely to fail to run sooner.
> ...Claude Code strategically examines specific files, searches for patterns, traces dependencies, and incrementally builds up an understanding of your changes.
So, it builds a dependency graph?
I've been playing with graph related things lately and it seems like there might be more efficient ways to do this than asking a daffy robot to do the job instead of a specific (AI crafted?) tool.
One could even get all fancy with said tool and use it to do fun and exciting things with the cross file dependencies like track down unused includes (or whatever) to improve build times.
1. Are you seeing a lower number of production hot fixes?
2. Have you tried other models? Or thought about using Output of different models and using combining the output if they have a delta.
3. Other than time & cost, what benchmarks in terms of software are considered (i.e., less hot fixes, etc.)?
Don't do this. If you have test time issues that are clogging the build pipeline, you can run a merge queue that creates ghost branches to test patches in sets rather than one by one.
> Yes, and I'm not exaggerating. Claude never missed a relevant E2E test.
I was waiting to see this part demonstrated and validated - for a given pr, whether you created an expected set of tests that would run and then compare it to what actually ran. Without a baseline, as any tester would tell you, the LLM output has been trusted without checking.
This is called Test Impact Analysis and it is something worth making deterministic. Like with an algorithm, and without an LLM.
And people have already done this.
For example: SeaLights is a product that does this.
I speak for a more distributed, microservices ecosystem, but the core issue is having a “fat” E2E testing suite in the first place. Since take hours to run, it forces a “batched” testing approach where a bunch of PRs are tested together (nightly etc) and leads to very slow feedback cycles for developers. Ideally I would shift to having a very small suite of E2E tests that can be run for every PR and then have service specific api tests that test the integration of the service being changed with the rest of the system. This way you de-centralize the testing as well and are closer to achieving true continuous delivery of each change to production.
21 comments
[ 2.6 ms ] story [ 31.3 ms ] threadHowever, those methods are tightly bound to programming languages, frameworks, and interpreters so they are difficult to support across technology stacks.
This approach substitutes the intelligence of the LLM to make educated guesses about what tests execute, to achieve the same goal of executing all of the tests that could fail and none of the rest (balancing a precision/recall tradeoff). What’s especially interesting about this to me is that the same technique could be applied to any language or stack with minimal modification.
Has anyone seen LLMs in other contexts being substituted for traditional analysis to achieve language agnostic results?
This section really stood out to me. I knew that asking GPT-5 to think gets better results, but I didn't know Claude had the same behavior. I'd love to see some sort of % success before and after "think deep" was added. Should I be adding "think deep" to all my non-trivial queries to Claude?
The solution to running only e2e tests on affected files has been around long before LLM. This is a bandage on poor CI.
"We" and "win" are both doing a lot of heavy-lifting here, as they are whenever talks about LLM labor destruction.
> what if we could run only the relevant E2E tests
The real title should be "Using Claude Code to Reduce E2E Tests by 84%."
I do wonder if this is as feasible at scale, where breaking master can be extremely costly (although at least it’s not running all tests for all commits, so a broken test won’t break all CI runs). Maybe it could be paired with, say, running all E2E tests post-merge and reporting breakages ASAP.
DHH recently described[0] the approach they've taken at BaseCamp, reducing ~180 comprehensive-yet-brittle system tests down to 10 good-enough smoke tests, and it feels much more in spirit with where I would recommend folks invest effort: teams have way more tests than they need for an adequate level of confidence. Code and tests are a liability, and, to paraphrase Kent Beck[1], we should strive to write the minimal amount of tests and code to gain the maximal amount of confidence.
The other wrinkle here is that we're often paying through the nose in costs (complexity, actual dollars spent on CI services) by choosing to run all the tests all the time. It's a noble and worthy goal to figure out how not to do that, _but_, I think the conclusion shouldn't be to throw more $$$ into that money-pit, but rather just use all the power we have in our local dev workstations + trust to verify something is in a shippable state, another idea DHH covers[2] in the Rails World 2025 keynote; the whole thing is worth watching IMO.
[0] - https://youtu.be/gcwzWzC7gUA?si=buSEYBvxcxNkY6I6&t=1752
[1] - https://stackoverflow.com/questions/153234/how-deep-are-your...
[2] - https://youtu.be/gcwzWzC7gUA?si=9zL-xWG4FUxYZMC5&t=1977
So, it builds a dependency graph?
I've been playing with graph related things lately and it seems like there might be more efficient ways to do this than asking a daffy robot to do the job instead of a specific (AI crafted?) tool.
One could even get all fancy with said tool and use it to do fun and exciting things with the cross file dependencies like track down unused includes (or whatever) to improve build times.
I was waiting to see this part demonstrated and validated - for a given pr, whether you created an expected set of tests that would run and then compare it to what actually ran. Without a baseline, as any tester would tell you, the LLM output has been trusted without checking.
Wait, models will get cheaper?
Coverage for the sake of coverage doesn't solve much, right?
Like... you need setup, act, assertions for it to be worthwhile?