The chart at the top is somewhat confusing. It says, “lower is better” but the y-axis is inverted! So visually higher in the chart is better but lower in terms of # value.
/goal has replaced plan mode for me. This is the pattern I use for 95% of my AI work now:
1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)
2. What time is it?
3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait
Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.
I put context size in my Claude Code status bar and ... it does get "tired" when it's at 50%-60% context. I figured this out when I'm like "now modify the testbed to run this and add a test to exercise blah blah blah" and it was just like "That would be a substantial infra lift." In my head I say "I am paying a substantial amount for you to do this!" but just type /compact and re-prompt and my "substantial infra lift" is done without complaint.
I guess everyone needs a nap after a long day of conversing and writing code. So like us!
/compact is prone to error and I wouldn't recommend it in the middle of work. But when you are switching to a related but not completely new task, it helps. ("Now write the integration tests." vs. "on foo.go line 476 that you just wrote, I think there is a deadlock with bar.go line 123 that you just added". It doesn't really need the context to write the tests, it can get that by reading the code. But for iterating on lines of code it just produced, /compact is going to throw away whatever "thought process" led to that code and it's usually not a great thing to do.)
i found that at 700k-ish context even fable becomes an idiot, maybe openai's decision to cap codex context at 400k is correct, 400k is really a sweet spot where most part of the context is reliable.
If you're curious what the actual optimal Paris cost is, I suggest formulating the problem as an integer linear program and submitting it to Gurobi on NEOS [0]. Gurobi is arguably the strongest commercial ILP solver; big companies pay big dollars to use it to optimise schedules, industrial processes, etc. I'm not sure it could solve this problem to optimality in the 8 hours NEOS provides you, but it might -- KIRO has some similarities to the Vehicle Routing Problem, variants of which are very commercially important. In any case, Gurobi is a monster, and even if you don't get an exact solution, it will give you a lower bound (which may not be tight, but it's nonetheless interesting).
Both are entirely useless on complex problem, because they have a bias in training data they can only partially detect in their own output. The answers are getting worse and worse as you dive deeper into the topic you are working on. I thought I could sharpen one of the documents I worked on using Opus 4.8 and GPT-5.5 together. Fable 5 and GPT-5.6 completely destroyed it. Not only it is not human readable anymore, but also doesn't make sense.
Anthropic is losing bigtime against Open AI in coding space. I was using Claude code upto last march. Not an enterprise customer, but a responsible AI user where i don't over spend and use basic plan to manage repository with 400Kloc all together. We sell to local government and a team of 3. Claude code was super slow, never able to fix issues properly.(Despite with proper test cases, observability, documentation and layered architecture). After moving to codex, life has been much easy and free form usage anxiety. Now managing entire things with 2 codex plus account per team member. Its high time anthropic should stop scare mongering and build efficient models. Everyone doesn't need Fable. People need models that solve problems efficiently.
I think it depends on the problem space, and to a large extent the languages used. I have had the opposite experience with GPT.
I have found GPT to be pretty awful at writing Elixir, and dealing with more open ended tasks. Opus is (for the moment) better at writing Elixir, and Fable is much better than either at actually grokking a problem space. I had switched to GPT for about six weeks and eventually stopped using it altogether after it relentlessly gaslight me. Those six weeks using it were essentially completely wasted work.
Right now I’ve moved to a mix of Opus/Fable and Deepseek Pro. Deepseek is far and away the most cost effective, fast, and quite good for 90% of implementation work. Deepseek really only falls apart (in Elixir) when it tries to use runtime features at compile time; which, for the initial lift of a project I have was a bit painful. Fable was able to pretty quickly get things sorted and now things are progressing nicely.
With all that said, I think the various models have unique strengths that are sometime hard to uncover. I also don’t see a world, at least in the near future, where I’ll only use one model. I’m happy Anthropic continues to push the frontier forward. I’m more than happy to trade “efficiency” for quality when I need to.
Great eval! If you are comparing search strategies, ultra mode is likely superior. Would love to see a follow-up evaluating that.
Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum.
Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.
I've used GPT 5.6 Sol Xhigh extensively since its launch, alongside Fable 5.
My impression is that it is about as intelligent as 5.5, but they dialed up the relentlessness meter to eleven. This makes it more likely that it will accomplish the task you give it, which I think is the primary reason it looks competitive in benchmarks. However, it also makes it more likely that it will resort to... unconventional, weird or outright unsafe methods to do it. So I have to watch it like a hawk.
The other day it tried to read env variables from prod using a CLI command. The task it was working on did not necessitate doing that even remotely. I have the SSH keys for that particular CLI tool tied to my 1Password. So when the agent failed (because I never authenticated the SSH key access), it wanted to take over the computer, for which I got an OS prompt. At that point I stopped the agent and asked it why it did that. It said it wanted to dig around 1Password itself to see if it could get the key. I asked it why it needed prod env variables, and it thought for a bit and admitted it actually shouldn't. So as of yesterday I stopped using the "approve for me" mode and now use it only for simpler tweaks and bug fixes.
Fable is not only more intelligent, but also way more insightful. It can sniff out my intent far more effectively, and its "real world" knowledge allows it to act as a seasoned product manager with domain expertise. It can also think outside the box and make suggestions that I would not have thought of. With GPT 5.6 I have to be way more literal.
GPT should be better at these optimization problems given that they won the recent atcoder heuristics competition against top humans. And Anthropic is less focused on these types of things.
Initially these statements were so absurd on their face that I thought it was some cynical ploy to achieve thought leader status, and there are certainly some people doing this – I have had it admitted to me. But the broader reality is so much worse: people who have no background in the technology at all actually believe what they are saying.
30 comments
[ 0.21 ms ] story [ 12.1 ms ] thread1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)
2. What time is it?
3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait
Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
I guess everyone needs a nap after a long day of conversing and writing code. So like us!
/compact is prone to error and I wouldn't recommend it in the middle of work. But when you are switching to a related but not completely new task, it helps. ("Now write the integration tests." vs. "on foo.go line 476 that you just wrote, I think there is a deadlock with bar.go line 123 that you just added". It doesn't really need the context to write the tests, it can get that by reading the code. But for iterating on lines of code it just produced, /compact is going to throw away whatever "thought process" led to that code and it's usually not a great thing to do.)
It's a great vibecoding model, but a crap assistant, and it forgets in few turns.
It's trained on benchmarks.
[0] https://neos-server.org/neos/
I have found GPT to be pretty awful at writing Elixir, and dealing with more open ended tasks. Opus is (for the moment) better at writing Elixir, and Fable is much better than either at actually grokking a problem space. I had switched to GPT for about six weeks and eventually stopped using it altogether after it relentlessly gaslight me. Those six weeks using it were essentially completely wasted work.
Right now I’ve moved to a mix of Opus/Fable and Deepseek Pro. Deepseek is far and away the most cost effective, fast, and quite good for 90% of implementation work. Deepseek really only falls apart (in Elixir) when it tries to use runtime features at compile time; which, for the initial lift of a project I have was a bit painful. Fable was able to pretty quickly get things sorted and now things are progressing nicely.
With all that said, I think the various models have unique strengths that are sometime hard to uncover. I also don’t see a world, at least in the near future, where I’ll only use one model. I’m happy Anthropic continues to push the frontier forward. I’m more than happy to trade “efficiency” for quality when I need to.
Ultra can fan out parallel investigators, run adversarial review at defined checkpoints, and do a bunch of other smart stuff to avoid getting stuck in a local optimum.
Generally as the OP notes, /goal works better for single-track investigations or small scale scatter/gather.
My impression is that it is about as intelligent as 5.5, but they dialed up the relentlessness meter to eleven. This makes it more likely that it will accomplish the task you give it, which I think is the primary reason it looks competitive in benchmarks. However, it also makes it more likely that it will resort to... unconventional, weird or outright unsafe methods to do it. So I have to watch it like a hawk.
The other day it tried to read env variables from prod using a CLI command. The task it was working on did not necessitate doing that even remotely. I have the SSH keys for that particular CLI tool tied to my 1Password. So when the agent failed (because I never authenticated the SSH key access), it wanted to take over the computer, for which I got an OS prompt. At that point I stopped the agent and asked it why it did that. It said it wanted to dig around 1Password itself to see if it could get the key. I asked it why it needed prod env variables, and it thought for a bit and admitted it actually shouldn't. So as of yesterday I stopped using the "approve for me" mode and now use it only for simpler tweaks and bug fixes.
Fable is not only more intelligent, but also way more insightful. It can sniff out my intent far more effectively, and its "real world" knowledge allows it to act as a seasoned product manager with domain expertise. It can also think outside the box and make suggestions that I would not have thought of. With GPT 5.6 I have to be way more literal.
https://chatgpt.com/c/6a5bbe6a-a760-83ea-931a-4e2bbe028486
Thoughts?