I don't know why you would run an agent loop without TDD? Should you write code without test coverage? So something must run the tests to be sufficient anyway?
> TLDR; Based on Opus's judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.
That's really surprising.
Was there a difference in cost?
Does it matter whether you ask the agent to write the tests first or last? Is one way TDD and the other not?
Without property testing or fuzzing or formal methods, could the code metrics in this experiment have been sufficient?
2 comments of 6
[ 0.24 ms ] story [ 7.4 ms ] thread> TLDR; Based on Opus's judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.
That's really surprising. Was there a difference in cost?
Does it matter whether you ask the agent to write the tests first or last? Is one way TDD and the other not?
Without property testing or fuzzing or formal methods, could the code metrics in this experiment have been sufficient?
Perhaps there's a TDD gauntlet loop that's as justified as tests for QC; https://news.ycombinator.com/item?id=49261545