as long as OpenAI and Anthropic keep subsidizing dirt cheap Codex or Claude Code usage, I'll just keep using them as evaluators. The trick is to have a fresh instance doing the reviewing, not the one that did the work.
By "evaluator" (aka "eval”), we did indeed mean frameworks for evaluating agent outputs broadly. The article and experiments center on LLM-as-a-judge, where an LLM is the grader, but the argument is ultimately statistical, so it holds regardless of whether the grader is an LLM, a simple supervised model, a set of regex checks, etc.
We were banking on readers being familiar with evals and left out definitions for conciseness, but as Gregaros points out, we could have been more explicit about what we meant.
5 comments
[ 2.8 ms ] story [ 23.3 ms ] threadFor example:
- You write a heuristic (regex, code, etc.) that assigns a score to an output
- You make another LLM score the output from your system (aka "LLM-as-a-judge")
- You have an automated system that can verify the generated outputs (e.g. does generated code compile or pass tests?)
People often talk about "LLM evals (evaluations)" which will include a set of evaluators i.e. scoring functions.
We'll make this clearer next time!
By "evaluator" (aka "eval”), we did indeed mean frameworks for evaluating agent outputs broadly. The article and experiments center on LLM-as-a-judge, where an LLM is the grader, but the argument is ultimately statistical, so it holds regardless of whether the grader is an LLM, a simple supervised model, a set of regex checks, etc.
We were banking on readers being familiar with evals and left out definitions for conciseness, but as Gregaros points out, we could have been more explicit about what we meant.