Skills and plugins have become less and less important in the ecosystems. Superhuman seems to be the outlier in both this benchmark and empirical evidence.
The challenge is improving performance while decreasing cost. There are many token condensers and model routers that attempt to do decrease cost, but they end up sacrificing performance.
Maybe all the extra features and scaffolding helped when he first started building it... But I'd bet that raw Claude Code and Codex are smart enough now that they make things worse.
They used their own proprietary benchmark called sudo L7, which purports to evaluate staff-level engineering judgment more than the correctness of the code itself.
Not surprising to me that higher thinking will outperform coding harnesses here.
8 comments
[ 5.0 ms ] story [ 27.1 ms ] threadThe challenge is improving performance while decreasing cost. There are many token condensers and model routers that attempt to do decrease cost, but they end up sacrificing performance.
I'm up to ~7m context in a session without spiralling.
I think most of cloud models are like bloated windows.
lol maybe more thinking = higher desire to get promoted or an equity refresh.
Maybe all the extra features and scaffolding helped when he first started building it... But I'd bet that raw Claude Code and Codex are smart enough now that they make things worse.
Not surprising to me that higher thinking will outperform coding harnesses here.