2) models have gotten too good at this task to compare them meaningfully (I tried doing a blind test and couldn't reliably tell 6.1 Astra, Sol, and Terra apart at reasoning levels that approximately match in cost)
>> Are the pelicans still a useful benchmark at all?
If you are using these models for software development, then I ask you this question…… when you are interviewing job candidates to join your team, do you ask them to write a program to draw pelicans?
look at the bicycle not the pelican. only astra low made a bicycle that resembles a functional bicycle in the low/none category. luna needs xhigh to make a bike. the legs are still wrong, on the wrong side of the bike, in all but like four of the outputs.
8 comments
[ 0.19 ms ] story [ 10.0 ms ] threadI would guess that
1) they're probably in the training data, and
2) models have gotten too good at this task to compare them meaningfully (I tried doing a blind test and couldn't reliably tell 6.1 Astra, Sol, and Terra apart at reasoning levels that approximately match in cost)
If you are using these models for software development, then I ask you this question…… when you are interviewing job candidates to join your team, do you ask them to write a program to draw pelicans?
Thus you have your answer.
look at the bicycle not the pelican. only astra low made a bicycle that resembles a functional bicycle in the low/none category. luna needs xhigh to make a bike. the legs are still wrong, on the wrong side of the bike, in all but like four of the outputs.