Taming LLMs: Using Executable Oracles to Prevent Bad Code (john.regehr.org)

[dead]

"Our goal should be to give an LLM coding agent zero degrees of freedom"

Wouldn't that just be called inventing a new language with all the overhead of the languages we already have? Are we getting to the point where getting LLMs to be productive and also write good code is going to require so much overhead and additional procedures and tools that we might as well write the code ourselves. Hmmm...

[dead]

[flagged]

[flagged]

Has anyone had success using 2 agents, with one as the creator and one as an adversarial "reviewer"? Is the output usually better or worse?

[dead]

Now is Haskell's time to shine.

> JustHTML was effectively tested into existence using a large, existing test suite.

I love the phrase “tested into existence”.

The oracle problem is tractable when the output is code: you can compile it, run tests, diff the output. For conversational AI it's much harder. We've seen teams use LLM-as-judge as their validation layer and it works until the judge starts missing the same failure modes as the generator.

[dead]