What's with slop-factories forgetting what a README is supposed to be? A quick intro to the project, not a gigantic abomination with irrelevant details.
I don't know how good of a marketing ploy it is, tbh. Considering it paints them as incompetent, and that they cannot be trusted with developing this technology safely.
Also note that Opus 5 (High) has an index value of 62, vs Fable 5 (Max) has 61. So some strangeness going on with that index.
I have some doubt about AA-index. For example Opus 5 (High) is at the index value as Fable 5 (Max), I wonder if that's really true.
One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be…
I would expect the benchmark scores to be nonlinear near the top, as the easier tasks get solved and the harder ones are left over. So going from 10 to 15 would be easier than going from 60 to 65. I only take the…
From Artificial Analysis cost per task, it looks like Fable 5.1 is more expensive per task than Fable 5? Cache hit price went down, but the other components still add up to more. Edit: 5.1-xhigh seems to be cheaper than…
All of this stuff seems like a band-aid solution. These things need to be trained ground-up to have some sort of memory, perhaps out-of-context memory. Also seems like a requirement for any sort of continual learning…
What's going on with sol here? The note says it spends a lot of time waiting, did it just not effectively use time (i.e. something like parallel runs) so it's graph ends up being stretched in the time axis?
Also with Qwen 3.8 being more token hungry than Luna, using around 2.3x tokens. Which hurts for local deployment.
One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning. [1]…
I've never found these discussions to be all that useful, because it's hard to define what conditions are sufficient to say something is "thinking" or "reasoning". It just ends up being circular and metaphysical…
You are making two different claims, and then treating them as one. 1. Mimicking a process may not prove that the system is truly preforming the same process. In this case, a "reasoning" process. 2. A system that…
Work doesn't start with a PR description though. I'm assuming most people that are using LLMs start with some sort of document (plan, spec, intent, etc) which captures intent. I guess you could also use all the session…
Their models don't consume encrypted text, it would be absurd to train them to do so. Surely they decrypt the text before feeding it into the LLM, so the contents could get leaked out by asking it.
My own experience: much less context bloat, and hidden system prompts. Can tune it to your own workflow, instead of dealing with whatever hidden system prompt Claude Code forces on you.
To be fair, Pi only had to be useful. Claude Code had to solve the difficult unsolved problem of making a terminal print text, so naturally they built a small game engine [1] [1]…
I think it might affect real work if part of it requires a lot of thinking, i.e. something similar in nature to a puzzle. There seems to be something wrong with the "commentary" channel related intermediate updates,…
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide…
From some of the numbers I'm seeing in the GitHub issue, the codex desktop app has the same 516 spikes. So most likely it is affected.
Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and…
Codex apparently added this too, I only noticed a few days ago: https://github.com/openai/codex/issues/28969 I'm usually at my desk, and get a notification when it stops to ask a question, so I've never accidentally had…
A highly detailed specification is not what I mean here. It's closer to plugging in a few sentence descriptions (or a totally cluttered brain dump) and having the model interview you to help pin down critical details…
Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to.…
What's with slop-factories forgetting what a README is supposed to be? A quick intro to the project, not a gigantic abomination with irrelevant details.
I don't know how good of a marketing ploy it is, tbh. Considering it paints them as incompetent, and that they cannot be trusted with developing this technology safely.
Also note that Opus 5 (High) has an index value of 62, vs Fable 5 (Max) has 61. So some strangeness going on with that index.
I have some doubt about AA-index. For example Opus 5 (High) is at the index value as Fable 5 (Max), I wonder if that's really true.
One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be…
I would expect the benchmark scores to be nonlinear near the top, as the easier tasks get solved and the harder ones are left over. So going from 10 to 15 would be easier than going from 60 to 65. I only take the…
From Artificial Analysis cost per task, it looks like Fable 5.1 is more expensive per task than Fable 5? Cache hit price went down, but the other components still add up to more. Edit: 5.1-xhigh seems to be cheaper than…
All of this stuff seems like a band-aid solution. These things need to be trained ground-up to have some sort of memory, perhaps out-of-context memory. Also seems like a requirement for any sort of continual learning…
What's going on with sol here? The note says it spends a lot of time waiting, did it just not effectively use time (i.e. something like parallel runs) so it's graph ends up being stretched in the time axis?
Also with Qwen 3.8 being more token hungry than Luna, using around 2.3x tokens. Which hurts for local deployment.
One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning. [1]…
I've never found these discussions to be all that useful, because it's hard to define what conditions are sufficient to say something is "thinking" or "reasoning". It just ends up being circular and metaphysical…
You are making two different claims, and then treating them as one. 1. Mimicking a process may not prove that the system is truly preforming the same process. In this case, a "reasoning" process. 2. A system that…
Work doesn't start with a PR description though. I'm assuming most people that are using LLMs start with some sort of document (plan, spec, intent, etc) which captures intent. I guess you could also use all the session…
Their models don't consume encrypted text, it would be absurd to train them to do so. Surely they decrypt the text before feeding it into the LLM, so the contents could get leaked out by asking it.
My own experience: much less context bloat, and hidden system prompts. Can tune it to your own workflow, instead of dealing with whatever hidden system prompt Claude Code forces on you.
To be fair, Pi only had to be useful. Claude Code had to solve the difficult unsolved problem of making a terminal print text, so naturally they built a small game engine [1] [1]…
I think it might affect real work if part of it requires a lot of thinking, i.e. something similar in nature to a puzzle. There seems to be something wrong with the "commentary" channel related intermediate updates,…
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide…
This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide…
From some of the numbers I'm seeing in the GitHub issue, the codex desktop app has the same 516 spikes. So most likely it is affected.
Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and…
Codex apparently added this too, I only noticed a few days ago: https://github.com/openai/codex/issues/28969 I'm usually at my desk, and get a notification when it stops to ask a question, so I've never accidentally had…
A highly detailed specification is not what I mean here. It's closer to plugging in a few sentence descriptions (or a totally cluttered brain dump) and having the model interview you to help pin down critical details…
Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to.…