HN Explorer

4 comments

[ 2.7 ms ] story [ 17.2 ms ] thread
Reubend
Because the website doesn't seem to show any sample size of runs, I assume they ran it once across the suite.

The models are nondeterministic, and therefore it's pretty normal for different runs to give different results.

I don't see this as evidence that Opus 4.6 has gotten worse.

coldtea
>I don't see this as evidence that Opus 4.6 has gotten worse.

I see it as corroboration evidence of actual everyday experience.

Also, any reason to imply "BridgeBench", apparently dedicated to AI benchmarking, wouldn't have run it more than once across the suite?

ehtbanton
Benchmarks like this one are designed to thoroughly test the model across several iterations. 15% is a MASSIVE discrepancy.

Come on Anthropic, admit what you're doing already and let us access your best models unhindered, even if it costs us more. At the moment we just all feel short-changed.