Has been ongoing for roughly a month now, with a variety of checkpoints along the usual speculation. As it stands, I'd just wait for the official announcement, prior to making any judgement. What their release plans are, whether a checkpoint is a possible replacement for Pro, Flash, Flash Lite, a new category of model, won't be released at all, etc. we cannot know.
More importantly, because of the way AIStudio does A/B testing, the only output we can get is for a single prompt and I personally maintain that outside of getting some basic understanding on speed, latency and prompt adherence, output from one single prompt is not a good measure for performance in the day-to-day. It also, naturally, cannot tell us a thing about handling multi file ingest and tool calls, but hype will be hype.
That there are people who are ranking alleged performance solely by one-prompt A/B testing output says a lot about how unprofessionally some evaluate model performance.
Not saying the Gemini 3.0 models couldn't be competitive, I just want to caution against getting caught up in over-excitement and possible disappointment. Same reason I dislike speculative content in general, it rarely is put into the proper context cause that isn't as eyecatching.
I might be in the minority here but I've consistently found Gemini to be better than ChatGPT, Claude and Deepseek (I get access to all of the pro models through work)
Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with.
I reckon different models are better at different kinds of work, but Gemini is pretty excellent at UI/UX web development, in my experience
I've found it to be excellent but 2.5 seems to experience context collapse around 50k tokens or so. At least that is my findings when using it heavily with Roo Code
I've since switched to Claude Code and I no longer have to spend nearly as much time managing context and scope.
I use it a lot for ideation on things like strategy and creative tasks. I've found Gemini to be much better than Claude, but I almost want to switch back to Claude because of the "Projects" primitive where I can add specific context to the project and ask questions within that project, and switch around to different projects with different context. Gemini just wants to take all context from everything ever asked and use it in the answers, or I can add the context in the individual prompt, which is tedious.
I use the models via Cursor and I prefer the output and speed of Claude Sonnet reasoning mode over Gemini 2.5 Pro. But my work is heavily in ETL/ELT processes and backend business processes. So maybe if I was doing a lot of web stuff it would be different.
> I've consistently found Gemini to be better than ChatGPT [ because ] Google has crawled the internet so they have more data to work with.
This commonly expressed non-sequitur needs to die.
First of all, all of the big AI labs have crawled the internet. That's not a special advantage to Google.
Second, that's not even how modern LLMs are trained. That stopped with GPT-4. Now a lot more attention is paid to the quality of the training data. Intuitively, this makes sense. If you train the model on a lot of garbage examples, it will generate output of similar quality.
So, no, Google's crawling prowess has little to do with how good Gemini can be.
Gemini specifically resets your context after a certain time. I have observed that it will basically clear out your context in a reasonable length session, which neither ChatGPT and Claude do.
Flushing or flattening down context saves costs. For that reason I never trust it with long research sessions. I would not be shocked if after 30 minutes they run a prompt like this:
And now reduce context history by 80%
This can very easily measured too, and would certainly expose the true feature set that differentiates these products.
I mostly use Gemini for everyday Q/A and research type stuff. I find it's pretty accurate and gets straight to the point. I mostly use Claude and very recently Codex for systems software dev. I'm very interested to see what changes.
I'm wondering how these models are getting better at understanding and generating code. Are they being trained on more data because these companies use their free tier customers' data?
I've seen many comments that they are great for OCR stuff, and my usecase of receipt photo processing does have it doing better than ChatGPT , Claude or Grok.
I find the sheer amount of glazing Gemini does unbearably, so I pretty much avoid using it. It’s just an unreal amount compared to GPT-5 or Claude.
Gives it a stack trace or some logs and Gemini treats it like the most amazing thing ever and throws a paragraph in there praising your skills as if you were a god.
you're not in the minority, there's just intense fanboyism on Hacker News to promote OpenAI, because it serves the whole "LLM revolution" schtick better
Gemini has been dominating the field for about a year now, but I suppose Google is bit boring cause they just do things well
So far, I have had a very good experience using Gemini Live with the camera turned on. Just today, I wanted to find out the name of a spare part inside a bathroom faucet. First, Gemini said it was a thermostatic cartridge, but I responded that it couldn't be, as it doesn't control temperature. Then it asked me what it did, and I said it has a button that controls the flow of water between the tap and shower. It correctly guessed that it was a diverter cartridge.
Gemini is the only plan I have not replaced. Claude and ChatGPT I will switch to depending on the ability of the coding agent, but Gemini is still my favorite for general information and especially for writing assistance.
Gemini might be a good model, it is _incredibly_ shit in tool calls and it has this incredibly tendency to multishot itself to death. When using their own gemini-cli tool, it's impossible to take it seriously, it's that bad.
For example:
If it makes a mistake, it'll keep on making the exact same mistake, and it'll act all cute like "Oh no, look at the mess I'm making". Some people say this is just a side effect of long contexts degrading performance, but it can happen even when 98% of the context is unused.
I'm also using a Ghidra MCP server to decompile some binaries.
Claude is great with this. It really gets it and is able to use it properly.
Gemini? Just one or two tool calls, and it'll start repeating the output of the tool calls for some reason.
Gemini also often isn't able to properly call the MCP tools. It just outputs the tool call as JSON text to the user.
Gemini-cli isn't even able to properly resume previous chat sessions. You have to actively save chats in order to resume them. Being able to simply resume the previous conversation using a flag like `--resume` or `--continue` has been a feature request since day one, and similar issues keep popping up weekly on the Github issue list. There are even multiple pull requests for this feature, but it's like nobody over there gives a damn.
In programming accuracy, these past few weeks, chatgpt seem to have improved while Gemini went the other way... or maybe it is just simply relative and only one of them changed... For me on a very custom and complex codebase.
It was also only model that was good with coming with something creative at all, like brainstorming startup ideas etc. for me - they were grounded as in reasonable compared to other I tried
I do very different work, or try to - historical HTR is unfortunately so bad even with the top models that the results aren't useful, but I keep trying new models.
But there's a historian on substack (Mark Humphries) who's also trying new models, and he also thinks he's gotten Gemini 3 output in A/B tests. He's very impressed with it:
Interesting. Not my experience at all. It makes mistakes that GPT-4 used to make: mixing languages (using Python syntax in C++ when I never asked any Python questions), imagining API calls that don’t exist in Google’s own products, writing 50 lines of C++ then inserting pseudo code or completely broken syntax.
Gemini is really good at fact-checking blog articles too, and suggesting edits/improvements. Other models will just post summaries but Gemini will walk you through the whole process
https://x.com/chetaslua is experimenting a lot with Gemini 3 and posting its results (various web desktops, a vampire survivor clone which is actually very playable, voxel 3d models, other game clones, SVG etc). They look really good, specially when they are one-shot.
I hope they are going to solve the looping problem. It’s real and it’s awful. It’s so bad that the CLI has a loop detection which I promptly ran into after a minute of use.
In the Gemini app 2.5 Pro also regularly repeats itself VERBATIM after explicitly being told not to multiple times to the point of uselessness.
This is super exciting. Gemini 2.5 pro was starting to feel like it's lagging behind a little bit; or at least it's still near the best but 3.0 had to be coming along.
It's my goto coder; it just jives better with me than claude or gpt. Better than my home hardware can handle.
What I really hope for 3.0. Their context length is real 1 million. In my experience 256k is the real limit.
Gemini2.5 Pro has assisted me better in every aspect of AI as compared to ChatGPT5. I hope they don't screw up Gemini 3 like OpenAI screwed ChatGPT with GPT5.
My strange observation is that Gemini 2.5 Pro is maybe the best model overall for many use cases, but starting from the first chat. In other words, if it has all the context it needs and produces one output, it's excellent. The longer a chat goes, it gets worse very quickly. Which is strange because it has a much longer context window than other models. I have found a good way to use it is to drop the entire huge context of a while project (200k-ish tokens) into the chat window and ask one well formed question, then kill the chat.
Hey, this has been my experience, too! I like Gemini because I’ve told it the tone and style I like my answers in and the first answer is very, very on point with that. But several times I’ve noticed that if I ask follow-up questions, the style immediately changes for the worse, often no longer following my preferences. I’ve also noticed that in follow-ups it makes really bad analogies that are not suitable at all for the kind of audience that the first response is catered to. I’ve been clicking the thumbs-down button every time I’ve seen this and commenting on the change in style and quality, so hopefully the training process will ingest that at some point.
There are a lot more of these Gemini 3 examples out on twitter right now.
After seeing them, I bought Google stock. What shocks me about its output is it actually feels like it's producing net new creative designs, not just regurgitated template output. Its extremely hard to design in code in a way that produces consistent, beautiful output, but it seems to be achieving it.
That combined with Google being the only one in the core model space that is fully vertically integrated with their own hardware makes me feel extremely bullish on their success in the AI race.
After looking at the Gemini 2.5 iterations under Appendix: “Gemini 3.0” A/B result versus the Gemini 2.5 Pro model, I couldn't help but think:
It's like a child who's given up on their homework out of frustration. Iteration 1 is way off, 2-3 seem to be improvements, then it starts to veer wildly off-track until essentially everything is changed in iteration 10. E.g. "HERE, IS THIS WHAT YOU WANT?!"
Which led me to hypothesize that context pollution could be viewed as a defense mechanism of sorts. Pollute the context until the prompter (perturber) stops perturbing.
The sentiment in this thread surprises me a great deal. For me, Gemini 2.5 Pro is markedly worse than GPT-5 Thinking along every axis of hallucinations, rigidity in its self-assured correctness and sycophancy. Claude Opus used to be marginally better but now Claude Sonnet 4.5 is far better, although not quite on par with GPT-5 Thinking.
I frequently ask the same question side-by-side to all 3 and the only situation in which I sometimes prefer Gemini 2.5 Pro is when making lifestyle choices, like explaining item descriptions on Doordash that aren't in English.
edit: It's more of a system prompt issue but I despise the verbosity of Gemini 2.5 Pro's responses.
For writing code at least this has been exactly my experience. GPT5 is the best but slow. Sonnet 4.5 is a few notches below but significantly faster and good enough for a lot of things. I have yet to get a single useful result from Gemini.
it is wild to me that people will see that invisible change in output they have zero insight, opinion, let alone control... and say "perfect! let's build a business on top of it!"
It's very interesting, and also quite frustrating that no two AI experiences are the same. Scrolling through the threads here and they're all seemingly contradictory.
I've had the Gemini 3.0 (presumably) A/B test and been unimpressed. It's usually on fairly novel questions. I've also gotten to the point where I often don't bother with getting Gemini's opinion on something because it's usually the worst of the bunch. I have a Claude Pro and OpenAI Pro sub and use Gemini 2.5 Pro via key.
The most glaring difference is the very low quality of web search it performs. It's the fastest of the three by far but never goes deep. Claude and Gemini seemingly take a problem apart and perform queries as they walk through it and then branch from those. Gemini feels very "last year" in this regard.
I do find it to be top notch when it comes to writing oriented tasks and sounding natural. I also find it to be fairly good about "keeping the plot" when it comes to creative writing. Claude is a great writer but makes a bit too many assumptions or changes. OpenAI is just flat out poor at creative writing currently due to the issues with "metaphorical language".
On speculative tasks -- e.g., "let's rank these polearms and swords in a tier list based on these 5 dimensions" -- Gemini does well.
On code work, Gemini is GOOD so long as it's not recent APIs. It tends to do poorly for APIs that have changed. For instance, "do XYZ in Stripe now that the API surface has changed, lookup the docs for the most recent version". GPT-5 has consistently amazed me with its ability to do this -- though taking an eternity to research. It's generally performed great with single-shot code questions (analyze this large amount of code and resolve X or fix Y).
On the Agentic front - it's a nonstarter. Both the CLI toolset and every integration I've used as recently as Monday have been sub-par when compared to Codex CLI and Claude Code.
On troubleshooting issues (PC/Software but not code), it tends to give me very generic and non-useful answers. "update your drivers, reset your PC". GPT-5 was willing to go more speculative dive deeper, given the same prompt.
On factual questions, Gemini is top notch. "Why were medieval armies smaller than Roman era armies" and that sort of thing.
On product/purchase type questions, Gemini does great. These are questions like "help me find a 25" stone vanity counter top with sink that has great reviews and from a reputable company, price cap $1000, prefer quality where possible". Unfortunately, like all of the other AI models, there's a non-zero chance that you'll walk through links and find that the product is not as described, not in-stock, or just plain wrong.
One last thing I'll note is that -- while I can't put my finger on it -- I feel like the quality of Gemini 2.5 Pro has declined over time while the model has also sped up dramatically. As a pay-per-token user, I do not like this. I'd rather pay more to get higher quality.
This is my subjective set of experiences as one person who uses AI everyday as a developer and entrepreneur. You'll notice that I'm not asking math questions or typical homework style questions. If you're using Gemini for college homework, perhaps it's the best model.
1. I find Gemini 2.5 Pro's text very easy and smooth to read. Whereas GPT5 thinking is often too terse, and has a weird writing style.
2. GPT5 thinking tends to do better with i) trick questions ii) puzzles iii) queries that involve search plus citations.
3. Gemini deep research is pretty good -- somewhat long reports, but almost always quite informative with unique insights.
4. Gemini 2.5 pro is favored in side by side comparisons (LMsys) whereas trick question benchmarks slightly favor GPT5 Thinking (livebench.ai).
5. Overall, I use both, usually simulatenously in two separate tabs. Then pick and choose the better response.
If I were forced to choose one model only, that'd be GPT5 today. But the choice was Gemini 2.5 Pro when it first came out. Next week it might go back to Gemini 3.0 Pro.
All I can hope for is that the “effective context window” (some level before competency plummets) is like 1m+ tokens. I would give a finger to just put my entire codebase into a model every time I want to talk to it. For now I’m still only talking to parts of the codebase, so to speak.
68 comments
[ 3.9 ms ] story [ 72.4 ms ] threadMore importantly, because of the way AIStudio does A/B testing, the only output we can get is for a single prompt and I personally maintain that outside of getting some basic understanding on speed, latency and prompt adherence, output from one single prompt is not a good measure for performance in the day-to-day. It also, naturally, cannot tell us a thing about handling multi file ingest and tool calls, but hype will be hype.
That there are people who are ranking alleged performance solely by one-prompt A/B testing output says a lot about how unprofessionally some evaluate model performance.
Not saying the Gemini 3.0 models couldn't be competitive, I just want to caution against getting caught up in over-excitement and possible disappointment. Same reason I dislike speculative content in general, it rarely is put into the proper context cause that isn't as eyecatching.
Maybe it's just the kind of work I'm doing, a lot of web development with html/scss, and Google has crawled the internet so they have more data to work with.
I reckon different models are better at different kinds of work, but Gemini is pretty excellent at UI/UX web development, in my experience
Very excited to see what 3.0 is like
I've since switched to Claude Code and I no longer have to spend nearly as much time managing context and scope.
This commonly expressed non-sequitur needs to die.
First of all, all of the big AI labs have crawled the internet. That's not a special advantage to Google.
Second, that's not even how modern LLMs are trained. That stopped with GPT-4. Now a lot more attention is paid to the quality of the training data. Intuitively, this makes sense. If you train the model on a lot of garbage examples, it will generate output of similar quality.
So, no, Google's crawling prowess has little to do with how good Gemini can be.
Flushing or flattening down context saves costs. For that reason I never trust it with long research sessions. I would not be shocked if after 30 minutes they run a prompt like this:
And now reduce context history by 80%
This can very easily measured too, and would certainly expose the true feature set that differentiates these products.
I'm wondering how these models are getting better at understanding and generating code. Are they being trained on more data because these companies use their free tier customers' data?
Gives it a stack trace or some logs and Gemini treats it like the most amazing thing ever and throws a paragraph in there praising your skills as if you were a god.
My results with Gemini are consistently better and usually also more reliable than other LLMs.
But tbh I prefer the UI of ChatGPT.
Gemini has been dominating the field for about a year now, but I suppose Google is bit boring cause they just do things well
For example:
If it makes a mistake, it'll keep on making the exact same mistake, and it'll act all cute like "Oh no, look at the mess I'm making". Some people say this is just a side effect of long contexts degrading performance, but it can happen even when 98% of the context is unused.
I'm also using a Ghidra MCP server to decompile some binaries. Claude is great with this. It really gets it and is able to use it properly. Gemini? Just one or two tool calls, and it'll start repeating the output of the tool calls for some reason.
Gemini also often isn't able to properly call the MCP tools. It just outputs the tool call as JSON text to the user.
Gemini-cli isn't even able to properly resume previous chat sessions. You have to actively save chats in order to resume them. Being able to simply resume the previous conversation using a flag like `--resume` or `--continue` has been a feature request since day one, and similar issues keep popping up weekly on the Github issue list. There are even multiple pull requests for this feature, but it's like nobody over there gives a damn.
Can't believe I am paying for multiple llms...
https://generativehistory.substack.com/p/has-google-quietly-...
I can attest to what he's saying that existing models are especially useless on tabular handwritten data such as ledgers.
In the Gemini app 2.5 Pro also regularly repeats itself VERBATIM after explicitly being told not to multiple times to the point of uselessness.
It's my goto coder; it just jives better with me than claude or gpt. Better than my home hardware can handle.
What I really hope for 3.0. Their context length is real 1 million. In my experience 256k is the real limit.
Based on what I'm hearing from friends who work at Google and are using it for coding, we're all going to be very disappointed.
Edit: It sound like they don't actually have Gemini 3 access, which would explain why they aren't happy with it.
So I get ChatGPT to spec out the work as a developer brief including suggested code then I give it to Gemini to implement.
After seeing them, I bought Google stock. What shocks me about its output is it actually feels like it's producing net new creative designs, not just regurgitated template output. Its extremely hard to design in code in a way that produces consistent, beautiful output, but it seems to be achieving it.
That combined with Google being the only one in the core model space that is fully vertically integrated with their own hardware makes me feel extremely bullish on their success in the AI race.
But you do you if you have "fun money" to throw around!
It's like a child who's given up on their homework out of frustration. Iteration 1 is way off, 2-3 seem to be improvements, then it starts to veer wildly off-track until essentially everything is changed in iteration 10. E.g. "HERE, IS THIS WHAT YOU WANT?!"
Which led me to hypothesize that context pollution could be viewed as a defense mechanism of sorts. Pollute the context until the prompter (perturber) stops perturbing.
With more work https://x.com/cannn064/status/1977882763832201643 https://codepen.io/jules064/pen/PwZKMQq
I frequently ask the same question side-by-side to all 3 and the only situation in which I sometimes prefer Gemini 2.5 Pro is when making lifestyle choices, like explaining item descriptions on Doordash that aren't in English.
edit: It's more of a system prompt issue but I despise the verbosity of Gemini 2.5 Pro's responses.
I've had the Gemini 3.0 (presumably) A/B test and been unimpressed. It's usually on fairly novel questions. I've also gotten to the point where I often don't bother with getting Gemini's opinion on something because it's usually the worst of the bunch. I have a Claude Pro and OpenAI Pro sub and use Gemini 2.5 Pro via key.
The most glaring difference is the very low quality of web search it performs. It's the fastest of the three by far but never goes deep. Claude and Gemini seemingly take a problem apart and perform queries as they walk through it and then branch from those. Gemini feels very "last year" in this regard.
I do find it to be top notch when it comes to writing oriented tasks and sounding natural. I also find it to be fairly good about "keeping the plot" when it comes to creative writing. Claude is a great writer but makes a bit too many assumptions or changes. OpenAI is just flat out poor at creative writing currently due to the issues with "metaphorical language".
On speculative tasks -- e.g., "let's rank these polearms and swords in a tier list based on these 5 dimensions" -- Gemini does well.
On code work, Gemini is GOOD so long as it's not recent APIs. It tends to do poorly for APIs that have changed. For instance, "do XYZ in Stripe now that the API surface has changed, lookup the docs for the most recent version". GPT-5 has consistently amazed me with its ability to do this -- though taking an eternity to research. It's generally performed great with single-shot code questions (analyze this large amount of code and resolve X or fix Y).
On the Agentic front - it's a nonstarter. Both the CLI toolset and every integration I've used as recently as Monday have been sub-par when compared to Codex CLI and Claude Code.
On troubleshooting issues (PC/Software but not code), it tends to give me very generic and non-useful answers. "update your drivers, reset your PC". GPT-5 was willing to go more speculative dive deeper, given the same prompt.
On factual questions, Gemini is top notch. "Why were medieval armies smaller than Roman era armies" and that sort of thing.
On product/purchase type questions, Gemini does great. These are questions like "help me find a 25" stone vanity counter top with sink that has great reviews and from a reputable company, price cap $1000, prefer quality where possible". Unfortunately, like all of the other AI models, there's a non-zero chance that you'll walk through links and find that the product is not as described, not in-stock, or just plain wrong.
One last thing I'll note is that -- while I can't put my finger on it -- I feel like the quality of Gemini 2.5 Pro has declined over time while the model has also sped up dramatically. As a pay-per-token user, I do not like this. I'd rather pay more to get higher quality.
This is my subjective set of experiences as one person who uses AI everyday as a developer and entrepreneur. You'll notice that I'm not asking math questions or typical homework style questions. If you're using Gemini for college homework, perhaps it's the best model.
2. GPT5 thinking tends to do better with i) trick questions ii) puzzles iii) queries that involve search plus citations.
3. Gemini deep research is pretty good -- somewhat long reports, but almost always quite informative with unique insights.
4. Gemini 2.5 pro is favored in side by side comparisons (LMsys) whereas trick question benchmarks slightly favor GPT5 Thinking (livebench.ai).
5. Overall, I use both, usually simulatenously in two separate tabs. Then pick and choose the better response.
If I were forced to choose one model only, that'd be GPT5 today. But the choice was Gemini 2.5 Pro when it first came out. Next week it might go back to Gemini 3.0 Pro.