I’m curious to hear from the HN community how you tamed this probabilistic non-deterministic genius generate a deterministic and repeatable result 100% percent of the time.
Anyone telling you they have tamed LLMs into producing 100% deterministic answers has either scoped the problem space so narrowly as to border on meaningless (e.g. "Is earth flat?" with a structured output schema of a single JSON boolean value), hasn't done robust statistical validation to actually confirm truly deterministic outputs, or both.
LLMs are fundamentally non-deterministic. Trying to use them to solve deterministic problem spaces is selecting the wrong tool for the job, and expecting them to be 100% reliable is the wrong mindset for working with them.
I've wondered if it would be possible (and beneficial) to make LLM's deterministic via a seed. Like how PRNG can specify a seed for repeatable deterministic pseudorandom numbers.
Theoretically, if you could specify a seed and the exact version of the model the output should always be the same. I wonder if this is possible with any open-weight models today?
---
On a more practical level, scripts (small programs) are deterministic so having the coding agent write (and possibly reuse) scripts might help.
I haven't done it with Claude Code itself, but I've built a deterministic + inferential engine that works inside a defined discipline with defined purposes. Broad enough to be practical, niche enough for most of the world to look at me funny.
The short version: let logic decide, and if there are multiple solutions let the model reason within a fixed range grounded by GraphRAG. Test every output against the logic, re-parse on contradiction, and emit 'unsure' after a couple of iterations rather than guessing.
It's no use for general knowledge. But where the judgement is largely codifiable it holds up well. There's an edge case out there that'll turn it to custard, I just haven't found it yet.
I've connected Claude Desktop to it over MCP and the results are good, not great. I designed the thing so I'm working in the sweet spot and there's still the occasional WTF.
the same technique you use to get a human engineering team to generate a deterministic and repeatable result 100% percent of the time. i know that's a flippant (but true!) answer, it's just a pretty tough problem :) setting the bar lower than 100% makes it a lot more tractable
I get Claude to write code and tests and then I review these carefully. I'm not doing any cutting edge SWE work, so this works fine for me. I also enjoy the bonus that once written it's deterministic, costs no tokens, and is no longer coupled to the AI provider or the government where they operate.
16 comments
[ 2.2 ms ] story [ 44.9 ms ] threadLLMs are fundamentally non-deterministic. Trying to use them to solve deterministic problem spaces is selecting the wrong tool for the job, and expecting them to be 100% reliable is the wrong mindset for working with them.
Although obviously I'd rather they stay non-deterministic
Theoretically, if you could specify a seed and the exact version of the model the output should always be the same. I wonder if this is possible with any open-weight models today?
---
On a more practical level, scripts (small programs) are deterministic so having the coding agent write (and possibly reuse) scripts might help.
The short version: let logic decide, and if there are multiple solutions let the model reason within a fixed range grounded by GraphRAG. Test every output against the logic, re-parse on contradiction, and emit 'unsure' after a couple of iterations rather than guessing.
It's no use for general knowledge. But where the judgement is largely codifiable it holds up well. There's an edge case out there that'll turn it to custard, I just haven't found it yet.
I've connected Claude Desktop to it over MCP and the results are good, not great. I designed the thing so I'm working in the sweet spot and there's still the occasional WTF.
not gonna be 100% but reduces most issues significantly and helps with debug