Terminal Bench 4 Is Saturated · shard-db HN Explorer

I think ambiguously worded questions have their place, but you have to give up automatic scoring on such tasks.

You don't have to give it up entirely, you could grade on the interpretation they chose. Personally I'm a fan of models which seek clarification on critical ambiguities rather than barging ahead "autonomously" though.

But wow, they found a lot of issues with the benchmark.

scotty79 8d ago ↗

I really like the benchmarks where frontier models score each other's solutions to a problem.

Terminal Bench 4 Is Saturated (preferencemodel.com)

3 comments