My own experience: much less context bloat, and hidden system prompts. Can tune it to your own workflow, instead of dealing with whatever hidden system prompt Claude Code forces on you.
To be fair, Pi only had to be useful. Claude Code had to solve the ancient unsolved problem of making a terminal print text without turning it into a small game engine.
I think it might affect real work if part of it requires a lot of thinking, i.e. something similar in nature to a puzzle. There seems to be something wrong with the "commentary" channel related intermediate updates,…
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide…
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early? Removing…
From some of the numbers I'm seeing in the GitHub issue, the codex desktop app has the same 516 spikes. So most likely it is affected.
Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and…
Codex apparently added this too, though it can only ask questions in plan mode so the damage is limited?
A highly detailed specification is not what I mean here. It's closer to plugging in a few sentence descriptions (or a totally cluttered brain dump) and having the model interview you to help pin down critical details…
Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to.…
Why supply underspecified requirements in the first place? Both models are good at challenging assumptions/edge cases and asking questions to clarify, but seemingly only when explicitly asked (i.e. something like a…
Cost per task is shockingly high. More expensive than Opus 4.8, second in place to Fable. Cost per task data is only available for max effort though, might just be very inefficient at that effort level.
That seems to only be true for the "Agentic Search" benchmark. That benchmark in particular is a bit weird, because Sonnet 4.6 effort levels had a relatively small effect, so Sonnet 5 med is basically comparable to all…
> Don't use it to write code One of the things I've started doing is to write the high level scaffold myself, create stubs (with comments) and have it fill out the details. I haven't done this for long, but I feel like…
This is false. A process needs read permission on the relevant `/dev/input/` device, typically by running as root or as a user in a group like `input`. Normal desktop users generally should not be in the input group.…
Yea these plots are too noisy and dense. Especially that second one, lines all over the place.
From what my own experiences are, and what's on their checkout page, $100 is 5x base usage and $200 is 20x. If $100 was 10x, then I personally would drop down. They want people to go to the highest tier.
I'm really getting sick of reading about safeguards and what I'm not allowed to do on every model release.
To follow up on this, I had it solve a nasty ODE problem that I saw in the recent Mathematica 15 release post: Solve the following first-order ODE for f(x): ((-1 - 2*x)*f(x)*tan(1 + x - exp(-61 - 2*x)*f(x)/x) + exp(61 +…
Lots of confusion about what this model is actually focused on. It is a cheap specialist for closed-world, verifiable reasoning tasks like math, self-contained coding problems, and similar. "Closed-world" means the…
The lack of tool use will hinder it a lot I think, since bug hunting requires collecting context across a code base and stitching it together. It might be good in a more narrow sense, i.e "is there a bug in this block…
This model doesn't support tool calling, was not part of its training. It's focused on Python (and I think C++) competitive programming and mathematics tasks, i.e. tasks with verifiable rewards. So if you have a task…
No one really goes into an interview speaking about politics, or propaganda, or unionizing. So there isn't much signal in telling people to not do that, most already bend over backwards to look like an ideal candidate.…
How does this have anything to do with a sluggish job market? Do you think people are struggling to find jobs because they are too political? One can be a servile dog, and still struggle to land a role, it's not…
Could you provide some details, if possible, like what model & thinking effort, what kinds of tasks? I used to swap between Claude Code and Codex often, and these days use Codex more because of the usage limits.…
My own experience: much less context bloat, and hidden system prompts. Can tune it to your own workflow, instead of dealing with whatever hidden system prompt Claude Code forces on you.
To be fair, Pi only had to be useful. Claude Code had to solve the ancient unsolved problem of making a terminal print text without turning it into a small game engine.
I think it might affect real work if part of it requires a lot of thinking, i.e. something similar in nature to a puzzle. There seems to be something wrong with the "commentary" channel related intermediate updates,…
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide…
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early? Removing…
From some of the numbers I'm seeing in the GitHub issue, the codex desktop app has the same 516 spikes. So most likely it is affected.
Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and…
Codex apparently added this too, though it can only ask questions in plan mode so the damage is limited?
A highly detailed specification is not what I mean here. It's closer to plugging in a few sentence descriptions (or a totally cluttered brain dump) and having the model interview you to help pin down critical details…
Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to.…
Why supply underspecified requirements in the first place? Both models are good at challenging assumptions/edge cases and asking questions to clarify, but seemingly only when explicitly asked (i.e. something like a…
Cost per task is shockingly high. More expensive than Opus 4.8, second in place to Fable. Cost per task data is only available for max effort though, might just be very inefficient at that effort level.
That seems to only be true for the "Agentic Search" benchmark. That benchmark in particular is a bit weird, because Sonnet 4.6 effort levels had a relatively small effect, so Sonnet 5 med is basically comparable to all…
> Don't use it to write code One of the things I've started doing is to write the high level scaffold myself, create stubs (with comments) and have it fill out the details. I haven't done this for long, but I feel like…
This is false. A process needs read permission on the relevant `/dev/input/` device, typically by running as root or as a user in a group like `input`. Normal desktop users generally should not be in the input group.…
Yea these plots are too noisy and dense. Especially that second one, lines all over the place.
From what my own experiences are, and what's on their checkout page, $100 is 5x base usage and $200 is 20x. If $100 was 10x, then I personally would drop down. They want people to go to the highest tier.
I'm really getting sick of reading about safeguards and what I'm not allowed to do on every model release.
To follow up on this, I had it solve a nasty ODE problem that I saw in the recent Mathematica 15 release post: Solve the following first-order ODE for f(x): ((-1 - 2*x)*f(x)*tan(1 + x - exp(-61 - 2*x)*f(x)/x) + exp(61 +…
Lots of confusion about what this model is actually focused on. It is a cheap specialist for closed-world, verifiable reasoning tasks like math, self-contained coding problems, and similar. "Closed-world" means the…
The lack of tool use will hinder it a lot I think, since bug hunting requires collecting context across a code base and stitching it together. It might be good in a more narrow sense, i.e "is there a bug in this block…
This model doesn't support tool calling, was not part of its training. It's focused on Python (and I think C++) competitive programming and mathematics tasks, i.e. tasks with verifiable rewards. So if you have a task…
No one really goes into an interview speaking about politics, or propaganda, or unionizing. So there isn't much signal in telling people to not do that, most already bend over backwards to look like an ideal candidate.…
How does this have anything to do with a sluggish job market? Do you think people are struggling to find jobs because they are too political? One can be a servile dog, and still struggle to land a role, it's not…
Could you provide some details, if possible, like what model & thinking effort, what kinds of tasks? I used to swap between Claude Code and Codex often, and these days use Codex more because of the usage limits.…