We ran a bunch (2,725) of Tessl verifier tests which are part of our test suite for our internal code base and switched the judge model to Jev and GPT Luna 6
The results:
- Jev is 13.6x faster
- Jev is 2.7x cheaper
- The models had a 85.9% agreement on verdicts
Jev finished the suite in 32 seconds. GPT Luna 6 took 436.5 seconds.
The interesting part is where the models disagreed, particularly on rules about comment structure and content. We published the approach, what we tested, and a workflow you can reproduce on your own codebase.
- If it is offline processing (e.g. classifying text snippets stored in a DB (from call transcript, from logs)), you can batch to Luna
- If you batch to Luna and then divide both cost and time by the batch size, you will find that Luna beats Jev.
- Now tune your batch size for your test dataset and see where the batch size causes accuracy falloff.
Tested this approach with GPT-6 Luna. It was 1.6x cheaper, 1.2x slower, within margin of error performance vs Jev with batch size 20 using a CFPB complaint dataset. Tuning batch size up to even double would likely yield similar accuracy while reducing both the per-record run time as well as per-record token cost.
The methodology of comparing single record is only valid for the on-line, real-time use case. For every other case, Luna can match or beat Jev by simply batching.
3 comments
[ 0.23 ms ] story [ 13.4 ms ] threadThe results:
- Jev is 13.6x faster - Jev is 2.7x cheaper - The models had a 85.9% agreement on verdicts
Jev finished the suite in 32 seconds. GPT Luna 6 took 436.5 seconds.
The interesting part is where the models disagreed, particularly on rules about comment structure and content. We published the approach, what we tested, and a workflow you can reproduce on your own codebase.
- If it is offline processing (e.g. classifying text snippets stored in a DB (from call transcript, from logs)), you can batch to Luna
- If you batch to Luna and then divide both cost and time by the batch size, you will find that Luna beats Jev.
- Now tune your batch size for your test dataset and see where the batch size causes accuracy falloff.
Tested this approach with GPT-6 Luna. It was 1.6x cheaper, 1.2x slower, within margin of error performance vs Jev with batch size 20 using a CFPB complaint dataset. Tuning batch size up to even double would likely yield similar accuracy while reducing both the per-record run time as well as per-record token cost.
The methodology of comparing single record is only valid for the on-line, real-time use case. For every other case, Luna can match or beat Jev by simply batching.