[–] og_kalu 1y ago ↗ New SOTA's on:SWE-Bench - 71.7Competition Code - 2727ARC (Semi Private Eval) - 75.7 on low, 87.5% on high computeFrontier Math (previous SOTA was 2%) - 25% on high compute
[–] macawfish 1y ago ↗ Wow, the demo where the user asks for untraceable payments shows some pretty sophisticated reasoning. The word "crafty" comes to mind.
4 comments
[ 2.9 ms ] story [ 22.1 ms ] threadSWE-Bench - 71.7
Competition Code - 2727
ARC (Semi Private Eval) - 75.7 on low, 87.5% on high compute
Frontier Math (previous SOTA was 2%) - 25% on high compute