Hi all! I’m building www.fairbake.dev (the business around Bakeoff) and when the question came up again in one of my calls of what effort level to set on model X for new {feature, bug fix, agent, categorizer, …} type tasks, I figured I’d try to answer it quantitatively using the eval framework and hallenges I’d built alongside my clients.
When I dug into it, I realized that the models shipped at different default effort levels. That seems confusing to users - I wonder why the labs do this. I was surprised to see that increasing the thinking level often didn’t improve correctness and almost always increased (non-test) diff size. I was less surprised that Fable at high and Opus at max did poorly.
Ask me any questions you have; happy to share specific differences I noticed between solutions. What effort level are you running at and why?
1 comment
[ 2.8 ms ] story [ 7.2 ms ] threadWhen I dug into it, I realized that the models shipped at different default effort levels. That seems confusing to users - I wonder why the labs do this. I was surprised to see that increasing the thinking level often didn’t improve correctness and almost always increased (non-test) diff size. I was less surprised that Fable at high and Opus at max did poorly.
Ask me any questions you have; happy to share specific differences I noticed between solutions. What effort level are you running at and why?