I wonder how good it is compared to Claude Sonnet 4, and when it's coming to GitHub Copilot.
I almost exclusively wrote and released https://github.com/andrewmcwattersandco/git-fetch-file yesterday with GPT 4o and Claude Sonnet 4, and the latter's agentic behavior was quite nice. I barely had to guide it, and was able to quickly verify its output.
> GPT‑5 also excels at long-running agentic tasks—achieving SOTA results on τ2-bench telecom (96.7%), a tool-calling benchmark released just 2 months ago.
Yes, but it does worse than o3 on the airline version of that benchmark. The prose is totally cherry picker.
Between Opus aand GPT-5, it's not clear there's a substantial difference in software development expertise. The metric that I can't seem to get past in my attempts to use the systems is context awareness over long-running tasks. Producing a very complex, context-exceeding objective is a daily (maybe hourly) ocurrence for me. All I care about is how these systems manage context and stay on track over extended periods of time.
What eval is tracking that? It seems like it's potentially the most imporatnt metric for real-world software engineering and not one-shot vibe prayers.
I've been testing it against Opus 4.1 the last few hours and it has done better and solved problems Claude kept failing at. I would say it's definitely better, at least so far.
> Producing a very complex, context-exceeding objective is a daily (maybe hourly) ocurrence for me. All I care about is how these systems manage context and stay on track over extended periods of time.
For whatever reason Github's Copilot is treated like the redheaded stepchild of coding assistants. Even through there are Anthropic, OpenAI, and Google models to choose from. And there is a "spaces"[0] website feature that may be close to what you are looking for.
I got better results for testing some larger task using that than I did through the IDE version. But have not used it much. Maybe others have more experience with it. Trying to gather all the context and then review the results was taking longer than doing it myself; having the context gathered already or building it up over time is probably where its value is.
Sorry if this is repetitive but you have to break the problem down just like any complex computing task. The difference is how. You have to break the problems into context windows that you anticipate being able to sow together later. It’s not the same way you would break down a source code authoring task in its absence but the theory is the same.
over the last week or so I have put probably close to 70 hours into playing around with cursor and claude code and a few other tools (its become my new obsession). I've been blown away by how good and reliable it is now. That said the reality is in my experience the only models that actually work in any sort of reliable way are claude models. I dont care what any benchmark says because the only thing that actually matters is actual use. I'm really hoping that this new gpt model actually works for this usecase because competition is great and the price is also great.
I opened up the developer playground and the model selection dropdown showed GPT-5 and then it disappeared. Also I don't see it in ChatGPT Pro. What's up?
The ability to specify a context-free grammar as output constraint? This blows my mind. How do you control the auto regressive sampling to guarantee the correct syntax?
Tried using gpt-5 family with response API and got error "gpt-5 does not exist or you don't have access to it". I guess they are not rolling out in lock step with the live stream and blog article?
It does seem to be doing well compared to Opus 4.1 in my testing the last few hours. I've been on the Claude Code 200 plan for a few months and I've been really frustrated with it's output as of late. GPT-5 seems to be a step forward so far.
Context-free grammar and regex support are exciting. I wonder what, or whether, there are differences from the Lark-like CFG of llguidance, which powers the JSON schema of the OpenAI API [^1].
What the fuck?
Nobody else saw the cursor ceo looking through the gpt5 generated code, mindlessly scrolling saying "this looks roughly correct, i would love to merge that" LOL
It was (attempted to be) solved by a human before, yet not merged...
With all the great coding models OpenAI has access to, their SDK team still feels too small for the needs.
I'm a codeforces guy, and I've benchmarked o3 on several of my favorite problems of various difficulty and concluded that o3 really isn't suitable for true reasoning still. Mostly because it's unable to think from first principles, so if you throw a non-standard problem it will brick. I think this will be a fundamental issue with any LLM.
I will say I would far more appreciate an AI that when it faces these ambiguous problems, either provides sources for further reading, or just admits it doesn't know and is, you know, actually trying to work together to find a solution instead of being trained to 1 shot everything.
When generalizing these skills to say, debugging, I will often just straight up ignore the AI slop output it concluded and instead explore the sources it found. o3 is surprisingly good at this. But for hard niche debugging, the conclusions it comes to are not only wrong, but it phrases it in an arrogant way and when you push back it's actually like talking to a narcissist (phrasing objections as "you feel", being excessively stubborn, word dumping a bunch of phrases that sound correct but don't hold up to scrutiny, etc).
Can anyone explain to me why they've removed parameter controls for temperature and top-p in reasoning models, including gpt-5? It strikes me that it makes it harder to build with these to do small tasks requiring high-levels of consistency, and in the API, I really value the ability to set certain tasks to a low temp.
It's not a hard logic path to follow - If AI becomes a digital necessity for modern society to function, Microsoft's relevance shrinks while OpenAI's relevance grows.
Once OpenAI breaks out of the "App" space and into the "OS" and "Device" space, Microsoft may get absorbed into the ouroboros.
OpenAI's dependence on Microsoft currently is purely financial (investment) and contractual (exclusivity, azure hosting).
eventually traditional operating systems will cease to exist, you'll just have a model creating dynamic UX for you on the fly for whatever experience you want
I understood it as that the economic relationship they have is going to make Microsoft broke somehow, be it dollars and/or just the focus of the company.
75 comments
[ 3.6 ms ] story [ 69.2 ms ] threadI almost exclusively wrote and released https://github.com/andrewmcwattersandco/git-fetch-file yesterday with GPT 4o and Claude Sonnet 4, and the latter's agentic behavior was quite nice. I barely had to guide it, and was able to quickly verify its output.
Yes, but it does worse than o3 on the airline version of that benchmark. The prose is totally cherry picker.
What eval is tracking that? It seems like it's potentially the most imporatnt metric for real-world software engineering and not one-shot vibe prayers.
The power of these models has peaked and simply arn't going to manage the type of awareness being promised.
For whatever reason Github's Copilot is treated like the redheaded stepchild of coding assistants. Even through there are Anthropic, OpenAI, and Google models to choose from. And there is a "spaces"[0] website feature that may be close to what you are looking for.
I got better results for testing some larger task using that than I did through the IDE version. But have not used it much. Maybe others have more experience with it. Trying to gather all the context and then review the results was taking longer than doing it myself; having the context gathered already or building it up over time is probably where its value is.
[0] https://docs.github.com/en/copilot/concepts/spaces
But GPT-5 is substantially cheaper[0].
[0] https://simonwillison.net/2025/Aug/7/gpt-5/#pricing-is-aggre...
EDIT: It's out now
Input: $1.25 / 1M tokens (cached: $0.125/1Mtok) Output: $10 / 1M tokens
For context, Claude Opus 4.1 is $15 / 1M for input tokens and $75/1M for output tokens.
The big question remains: how well does it handle tools? (i.e. compared to Claude Code)
Initial demos look good, but it performs worse than o3 on Tau2-bench airline, so the jury is still out.
The price is what it is today because they are trying to become a dominant platform. It doesn't mean the price reflects what it actually costs to run.
I'd bet a lot of the $40 billion they got in March goes towards loss leaders.
https://platform.openai.com/docs/models/gpt-5
[^1]: https://github.com/guidance-ai/llguidance/blob/f4592cc0c783a...
I used gpt-5-mini with reasoning_effort="minimal", and that model finally resisted a hallucination that every other model generated.
Screenshot in post here: https://bsky.app/profile/pamelafox.bsky.social/post/3lvtdyvb...
I'll run formal evaluations next.
You can't make this up
It was (attempted to be) solved by a human before, yet not merged... With all the great coding models OpenAI has access to, their SDK team still feels too small for the needs.
Looks like they're trying to lock us into using the Responses API for all the good stuff.
I will say I would far more appreciate an AI that when it faces these ambiguous problems, either provides sources for further reading, or just admits it doesn't know and is, you know, actually trying to work together to find a solution instead of being trained to 1 shot everything.
When generalizing these skills to say, debugging, I will often just straight up ignore the AI slop output it concluded and instead explore the sources it found. o3 is surprisingly good at this. But for hard niche debugging, the conclusions it comes to are not only wrong, but it phrases it in an arrogant way and when you push back it's actually like talking to a narcissist (phrasing objections as "you feel", being excessively stubborn, word dumping a bunch of phrases that sound correct but don't hold up to scrutiny, etc).
https://x.com/elonmusk/status/1953509998233104649
Anyone know why he said that?
Once OpenAI breaks out of the "App" space and into the "OS" and "Device" space, Microsoft may get absorbed into the ouroboros.
OpenAI's dependence on Microsoft currently is purely financial (investment) and contractual (exclusivity, azure hosting).