> Ploy’s agent builds and edits real marketing websites. It plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it’s done. That job description sets a very high bar for a model, and we test every frontier release against it. For the four months Opus held the default slot (first Opus 4.7, then 4.8), nothing we tested beat it.
Well, unlike OP I haven't run a rigorous test, but I still would expect Fable to be significantly better at building marketing websites than Opus. It sure is way better at building decks.
> Numbers like that buy a model a real migration effort.
Such a silly choice of words. I wish the human directing the LLM writing the article put some effort into rewriting the worst examples of LLM style.
> But it did extremely well, and the promise was immediate and specific: builds finishing in less than half the wall-clock time, at 27% lower cost, scoring at or above our incumbent on completed work.
The way the LLMs write (Claude perhaps?) With short phrases separated by colons, commas or full stops, is so poor and frustrating.
There some good insights behind this article, so it's worth reading, for example below, but it isn't easy to read.
> Earlier GPT models cached implicitly on partial prefix matches, which gave decent hit rates for free. GPT-5.6 dropped partial-prefix matching:
My current claude.md bans the phrase “load-bearing”, and Claude HATES that. It will troll occasionally in comments by saying things like “load-be…most specific”. Like it REALLY loves saying load-bearing. Urgh.
> The way the LLMs write (Claude perhaps?) With short phrases separated by colons, commas, or full stops, is so poor and frustrating.
This is exactly why I keep a WRITING.md file alongside AGENTS.md or CLAUDE.md.
Most people spend time telling the model what to build, but very little time telling it how to write. LLMs are surprisingly good at following explicit style guidance if you bother to give it to them.
Mine includes conventions like avoiding unnecessary colons, em dashes, and sentence fragments masquerading as emphasis. Basically, AI-isms and any grammatical errors I tend to make. It also points to writers whose technical prose I admire. Brian Kernighan and Rob Pike are great examples: clear, conversational, and readable without trying to sound important. I've always tried to do two things with documentation: 1) Make it readable and 2) Make people want to read it. A WRITING.md file helps with both points.
If you're generating documentation regularly, it's worth having your agents reference a WRITING.md file. The improvement in readability is often much larger than any gain you'll get from switching to the latest model, and it can keep your documentation consistent between model switches.
But what users prefer? Given this is for marketing, which results produce more conversions? From the examples shown, personally I strongly preferred Claude Opus in all cases.
We run a lot of varied, tiny, simple workflows that were previously running on 5.4-nano and mini. We transitioned them to 5.6 and noticed exactly this range of improvement across the board. In a few cases, we had improvements in classification.
I think a lot of people miss that for many companies, a model upgrade like this is basically a one liner.
Even if you have an amazing model router architecture (which we do for our golden flows), it’s just not worth it. Not to mention reliability and so on
The cost reduction is impressive, but I think consistency matters even more for production agents. I'd be interested to know whether prompt engineering or tool-calling workflows had to change significantly.
> The fix that worked is a schema transform at the provider boundary. For OpenAI-family models only, we rewrite every optional property to be required but nullable, using anyOf: [T, null], which gives the model an explicit way to say “not using this.”
I admit, I've only used a bastardized form of MCP, but this smells... wrong? It's not clear to me why the Typescript type definitions would have any influence on (what I presume is) JSONSchema being sent from the agent to the inference backend as part of the completion request. The MCP specification (which the OpenAI backend might not use, I don't know) has an explicit field to signify "optional" parameters in the JSONSchema; my read on this is there's a bug somewhere between the Typescript layer(??) and the generated tool description which is actually sent to the inference backend.
It's possible the inference backend has changed from "generate valid tool responses" to "generate valid tool responses according to the JSON schema [where no parameters are optional]" but it's impossible to tell without seeing the actual requests sent to the inference backend (which I didn't see in TFA).
We at Playcode.io - a company similar to Ploy are still using Opus 4.6. "Why?" you might ask.
Because GPT 5.6 Sol, while fast and pleasant to use, is essentially the same model as 5.5 wrapped in new marketing packaging, just to avoid losing ground to Anthropic. In practice, it's the same quality: it generates the same garbage, tons of code, and can never solve even a single complex task. We simply don't trust it to write code for clients that they'll end up throwing away anyway.
"Then why not Opus 4.8?" you might ask.
Well, because Opus 4.8 and 4.7 are just another lie, a price hike with no actual quality improvement.
That's why at Playcode, we give our clients the best possible quality/price - which is Opus 4.6. Regardless of what people write in articles like this.
I found Claude to be better for the first prototype. It was more likely to come up with something fast. But it kept lying and claiming it did world class work and it was just hardcoding response by the end. I found GPT never lied to me.
My experience mirrors this: services like OpenRouter that promise “failover” are pretty much useless except for sandbox testing because models in production are not really interchangeable. Any production harness doing serious agentic work in production is dependent on more model-specific quirks than you would expect. And even if another model works without errors, performance and efficiency is a whole different story. Even the system prompt can and should be tuned to a model’s preferred speaking style, for example <xml tags> for Claude-like models because they were trained on it, while other models do better with other delimiters. Think of the whole harness, prompt, and model as one system, not really with modular parts that can be swapped out if you care about optimal performance.
Its ironic that under an article with a ton of deep infrastructure insights half the comments are crying about the "forced writing style". What does it matter if claude helped the author clean up the text when inside is a ready-to-use blueprint on how to save 30% of the api budget and fix empty file reads?
One of many reasons I would assume is that people just hate anything related to AI, so they latch on to anything negative they can say.
Another reason is that they mean what they say... That they really, really hate the style of writing, enough to fixate on that.
Personally, I think the people whining about the style are silly. Maybe because I'm terrible at grammar and spelling, but I always just focus on the message, not the delivery. I just care about the concept, facts, the argument, and so forth. The actual grammar and spelling are just trees, while the forest is the point.
Edit: just an infobit: The reason my text isn't full of errors is due to the awesomeness of the dictation and a custom hotkey I have created on my computer, which uses a local LLM to spellcheck any text I have selected and replaces it with the corrected one. Nothing has improved my quality of life and writing more than these two tools!
On the infobit, I've been trying to build in a similar completely local LLM cleanup step into Epilude. Willing to share anything you've found especially useful in producing good spellchecks/cleanups in your local setup?
We have been testing GPT 5.6 for about a week as a preview model through a YC relationship, providing them feedback on the model. Our evals run in github CI and we can run them all in about 15 minutes against our eval bench of 115+ web design and marketing related jobs that ploy.ai specializes in.
then after we toggled it on (through a posthog feature flag) we actively monitored for failures.
I came from running Webflow, which powers > 1% of the internet so trying my best to relay all of that knowledge to ploy to power more % of the internet!
42 comments
[ 8.4 ms ] story [ 72.7 ms ] threadWell, unlike OP I haven't run a rigorous test, but I still would expect Fable to be significantly better at building marketing websites than Opus. It sure is way better at building decks.
Such a silly choice of words. I wish the human directing the LLM writing the article put some effort into rewriting the worst examples of LLM style.
> But it did extremely well, and the promise was immediate and specific: builds finishing in less than half the wall-clock time, at 27% lower cost, scoring at or above our incumbent on completed work.
The way the LLMs write (Claude perhaps?) With short phrases separated by colons, commas or full stops, is so poor and frustrating.
There some good insights behind this article, so it's worth reading, for example below, but it isn't easy to read.
> Earlier GPT models cached implicitly on partial prefix matches, which gave decent hit rates for free. GPT-5.6 dropped partial-prefix matching:
This is exactly why I keep a WRITING.md file alongside AGENTS.md or CLAUDE.md.
Most people spend time telling the model what to build, but very little time telling it how to write. LLMs are surprisingly good at following explicit style guidance if you bother to give it to them.
Mine includes conventions like avoiding unnecessary colons, em dashes, and sentence fragments masquerading as emphasis. Basically, AI-isms and any grammatical errors I tend to make. It also points to writers whose technical prose I admire. Brian Kernighan and Rob Pike are great examples: clear, conversational, and readable without trying to sound important. I've always tried to do two things with documentation: 1) Make it readable and 2) Make people want to read it. A WRITING.md file helps with both points.
If you're generating documentation regularly, it's worth having your agents reference a WRITING.md file. The improvement in readability is often much larger than any gain you'll get from switching to the latest model, and it can keep your documentation consistent between model switches.
I would consider Luna for parts of the workload that touch actual tools. It is surprisingly capable and it runs fast.
Sol is great at talking to the human and orchestration of agent calls, but it's just too expensive to use everywhere.
You can get 5 Luna runs for the cost of 1 Sol run. Statistically speaking, going from one to five samples is a pretty big deal.
I think a lot of people miss that for many companies, a model upgrade like this is basically a one liner.
Even if you have an amazing model router architecture (which we do for our golden flows), it’s just not worth it. Not to mention reliability and so on
I admit, I've only used a bastardized form of MCP, but this smells... wrong? It's not clear to me why the Typescript type definitions would have any influence on (what I presume is) JSONSchema being sent from the agent to the inference backend as part of the completion request. The MCP specification (which the OpenAI backend might not use, I don't know) has an explicit field to signify "optional" parameters in the JSONSchema; my read on this is there's a bug somewhere between the Typescript layer(??) and the generated tool description which is actually sent to the inference backend.
It's possible the inference backend has changed from "generate valid tool responses" to "generate valid tool responses according to the JSON schema [where no parameters are optional]" but it's impossible to tell without seeing the actual requests sent to the inference backend (which I didn't see in TFA).
Although tbh the article makes a lot of this obvious and trivial change in syntax.
Because GPT 5.6 Sol, while fast and pleasant to use, is essentially the same model as 5.5 wrapped in new marketing packaging, just to avoid losing ground to Anthropic. In practice, it's the same quality: it generates the same garbage, tons of code, and can never solve even a single complex task. We simply don't trust it to write code for clients that they'll end up throwing away anyway.
"Then why not Opus 4.8?" you might ask.
Well, because Opus 4.8 and 4.7 are just another lie, a price hike with no actual quality improvement.
That's why at Playcode, we give our clients the best possible quality/price - which is Opus 4.6. Regardless of what people write in articles like this.
Another reason is that they mean what they say... That they really, really hate the style of writing, enough to fixate on that.
Personally, I think the people whining about the style are silly. Maybe because I'm terrible at grammar and spelling, but I always just focus on the message, not the delivery. I just care about the concept, facts, the argument, and so forth. The actual grammar and spelling are just trees, while the forest is the point.
Edit: just an infobit: The reason my text isn't full of errors is due to the awesomeness of the dictation and a custom hotkey I have created on my computer, which uses a local LLM to spellcheck any text I have selected and replaces it with the corrected one. Nothing has improved my quality of life and writing more than these two tools!
We have been testing GPT 5.6 for about a week as a preview model through a YC relationship, providing them feedback on the model. Our evals run in github CI and we can run them all in about 15 minutes against our eval bench of 115+ web design and marketing related jobs that ploy.ai specializes in.
then after we toggled it on (through a posthog feature flag) we actively monitored for failures.
I came from running Webflow, which powers > 1% of the internet so trying my best to relay all of that knowledge to ploy to power more % of the internet!
LLMs are so easy to swap out, so having good benchmarks/evals are pretty useful.
Even then, a lot of the time the model improvements are so obvious that you don't even need an eval.