We've been testing it extensively and its performance is no better than prior versions, and in many cases worse than open weight models like GLM. Gemini3.1 Pro is so significantly better.
To me, the play is: open weight on a provider like BaseTen (solid performance, low price point), or pay up for Gemini3.1 Pro if you need it.
But at their high price and low-ish quality, OpenAI models just aren't in the conversation right now without heavy incentives, e.g. via Azure.
Crazy, TBH. Curious if others find the same thing?
Yeah, I've had good experience with Kimi too. Good for the price point, for sure.
Anthropic models are still the best for me -- as long as you don't ask them to do something they don't want to -- but also, way too expensive for bulk pipeline processing. So I keep it to coding and Coworking...
I have building quite complex architecture applications from some time now. So I think I know the answer. For pure Coding no one comes close to Claude. No matter what the benchmark says, no one beats claude in terms of sheer coding skills. Having said that, claude lacks in architecture design decision making, it does not make good decision regarding that, I find ChatGPT more smart in terms of system designs and architectures. And I have experienced this not once but 4 times now. And For mathematical reasoning and formula making Gemini is better than both claude and chatGPT. I have experienced this once, when I had to design formula for calculating scores of different files and functions in a codebase.
I'm using GPT 5.4 solely now, over Claude and any other models. In my usage I find it better than all the other models especially at large and complex codebases.
I love GPT 5.4, and I'm under the impression that many claims of it's good or it's bad are only based on a specific setup + a few shots. I think many model's power and not just GPT 5.4, comes from good setup + good prompting, and some folks just do like "write a game of a spaceship shooting asteroids" and then decide if it's good or not based on that. Full-time GPT 5.4 usage is expensive though, I'm averaging $300-400/month using it all day every day with one or two agents in parallel. After GPT 5.2, I think it was in December last year, the thing that changes is that it went from sometimes getting it right, to almost always getting it right, and I found the times it doesn't get it right, it's because of your context + setup + prompt and not the model itself. I haven't used claude's agents, but I would guess that they're as powerful as GPT 5.4 if used right. Maybe slightly / marginally better or worse but not much.
5.4 is pretty good all things considered, but in some areas of the performance manifold it isn't better (Latency compared to the faster 4 models). Overall still one of the best models available, but all the frontier models have heavily overlapping performance manifolds.
14 comments
[ 3.2 ms ] story [ 59.5 ms ] threadTo me, the play is: open weight on a provider like BaseTen (solid performance, low price point), or pay up for Gemini3.1 Pro if you need it.
But at their high price and low-ish quality, OpenAI models just aren't in the conversation right now without heavy incentives, e.g. via Azure.
Crazy, TBH. Curious if others find the same thing?
We still have to see what Anthropic has cooked though
Anthropic models are still the best for me -- as long as you don't ask them to do something they don't want to -- but also, way too expensive for bulk pipeline processing. So I keep it to coding and Coworking...
So I would agree with you, it is not great.