15 comments

[ 0.23 ms ] story [ 22.8 ms ] thread
Nice! It is missing Codex in the agent harnesses comparison IMO.
I'd expect google to do well here, since they were historically strong at multimodal and physics.
I wouldn't trust Gemini anywhere near my code.

However, it's pretty good at regurgitating textbook answers (classic Google...). This makes it an asset for circuit design. Try one of my real queries with Gemini Pro:

"Create a differential current splitter that splits a bias current Ib into a positive current Ipos = Ib/2 + ky*Vy and a negative current Ineg = Ib/2 - ky*Vy where Vy = Vy+ - Vy- is a differential voltage input Y."

Yet another "benchmark to promote their own harness"
Please forgive my naivety, but are world models (once they are in a consumer-ready form) expected to outperform any currently existing LLM on these sorts of tasks (i.e. of the physical world)?
It's kind of interesting this is already out of data as it's missing Kimi 3 and Opus 5.
Seemed strange they would bench Terra and Luna but not Opus, Sonnet, and Haiku
Yeah, things are moving fast. Those benchmarks will be out soonish. Takes a while to run these things.
A really frustrating partial presentation, given an apparent lack of testing with a spread of efforts for each model.

Given that there's no reason to believe that Fable's xhigh is comparable to GPT-sol's xhigh, or Opus xhigh, for that matter, it would be far more useful to see the effort level where these tasks no longer achieved their goals.

Google with apptronic should have good models soon
So Fable "won" but it cost $124.76 for marginal performance benefits over the $22.56 5.6 Sol run.
Honestly, I just hate the term "physical AI". They're robots. It's unfortunate that we had to adopt a term with the words AI in it, just to get investor's attention.
I'm curious how good the terra/luna results are but there's not enough context here for me to judge 0.786 (terra) vs 0.814 (sol) vs 0.889 (fable).

How do the two lower models compare to e.g. gpt 5.5 or 5.4 or various opus models?