>METR measures the longest software engineering task a frontier model can complete successfully half the time
Longest? Yes, they actually use actual wall clock time. I don't think this metric makes any sense. I don't know about the others, I stopped reading at this point.
Author here. In short there are 4 different metrics:
- METR's time horizon
- TrackingAI's offline cognitive test
- Humanity's Last Exam, and
- ARC-AGI-2
that have lasted longer than 2 years (though ARC-AGI-2 is now saturated).
When plotted in the linear domain, they all have an exponential (hockey-shaped) curve, but the interesting thing is that the bend happens right at Q4 of 2025 (right when Gemini 3, Opus 4.5, GPT 5.2 all come out).
4 comments of 6
[ 0.27 ms ] story [ 32.0 ms ] threadLongest? Yes, they actually use actual wall clock time. I don't think this metric makes any sense. I don't know about the others, I stopped reading at this point.
- METR's time horizon
- TrackingAI's offline cognitive test
- Humanity's Last Exam, and
- ARC-AGI-2
that have lasted longer than 2 years (though ARC-AGI-2 is now saturated).
When plotted in the linear domain, they all have an exponential (hockey-shaped) curve, but the interesting thing is that the bend happens right at Q4 of 2025 (right when Gemini 3, Opus 4.5, GPT 5.2 all come out).