A month is not enough time for any meaningful change in an organization the size of Deepmind/Google. These models were surely the result of work streams and teams that started under Demis. I think Demis can safely feel proud Deepmind is getting back on track.
Popular claim by IBM PR, but the issue was that OS2fW component (which reused locally-installed Windows 3.x) had binary patches it applied to windows core component to turn it from DPMI host (which owns 32bit pagetable etc.) into DPMI client so that Windows would call to OS/2 for handling paging setup, and other details of interop.
The patches were made for Windows 3.1, but not for Windows 3.11 - since the changes resulted in binaries with different offsets, Windows 3.11 used under OS2fW would fail.
They're quite selective in benchmarks, c.f. only notably based one is 10% on TerminalBench. It's a really addled model, one time I said "Hi" and it built out a 4 panel hello world app with (fake) weather, a todo list, and a couple other things I forgot. I wouldn't be comfortable saying "ignore the #s!" except when I complained it was trash and way overcooked on agentic coding yet not good at it, and a couple DeepMind ML people liked the tweet.
Gemini 3.7 Flash was already smashing more expensive models on my Redactle benchmark https://redactle.net/llm-leaderboard which mostly tests omniscience.
I know everyone is benchmaxxing but this one feels one step too far. Doesn't DeepSWE have both public and private tasks? I'd love to see the diff here.
It looks more like Google execs losing their mind and pressuring researchers to put DeepSWE directly into the training set.
Will look forward to the "feel" of the model in real testing. But I agree that these benchmarks do get "dealt with" rapidly. That's a shame, but I guess it's the times we live in.
We'll see about that. I suspect benchmaxxing as all the labs do as I haven't found Gemini models to be nearly as good in agentic engineering compared to Claude or GPT models.
Accordit to reddit talk, Fable 5.1 is worse than Opus 4.6 and 8B models are smarter than Qwen 3.8 Max, I wouldn't take anything said there with any more reliability than an instagram short.
They are all much larger and more expensive models. Google does not have a frontier model right now, but for cheap ones, they are better than event the chinese models now.
Stop lying. mattlondon said "gemini-3-8-flash shows an intelligence score of 59" which is undeniably correct. You can't say that number is false. You're literally lying.
All you had to do is go hover your mouse over "Models" in the top bar, hover over Claude Opus 5 and and click on medium: https://imgur.com/mlRCrt1
You have to be an incredibly dishonest person to see a 59 on both pages and say "the initial reported numbers were false and this was simply pointed out. You're changing the subject".
Better than even the Chinese models? That's a difficult-to-quantify, extremely rapidly moving target. Just today, Qwen 3.8 Max 0902 came out with a huge improvement over the previous Qwen 3.8 Max.
Wait a week with your judgement - most likely, Google is just bench-maxing very hard.
If you look at the previous Flash models and the announcement on Google I/O, it was an absolute disaster. Reality diverged very much from the marketing (supposedly great benchmarks).
anthropic really needs something to address the cheaper end of the market before they get left behind. Sonnet 5 sucks, and Haiku hasn't been updated in a year. meanwhile we've got gemini flash, luna, and GLM5.3 all delivering 90% of the performance for a small fraction of the cost. paying $25/mTok is going to start looking pretty silly soon.
I've been trying this Gemini 3.8 Flash for a day. Looks not much different than Gemini 3.7 Flash in my use case: I have Codex (gpt-5.6 sol) write up a design plan to implement a feature or refactor a portion of a system I am building, and have Claude (Opus-5) and Gemini (3.8 Flash) review and critique the plan, until all problems are addressed by Codex and approved by the reviewers; then have a cheaper model of Codex (gpt-5.6 luna) implement the plan, and still have Claude (Opus-5) and Gemini (3.8 Flash) review and critique the implementation, until all problems are addressed by Codex and approved by the reviewers.
The result is the same as the previous Gemini 3.6/3.7 Flash days: Claude could always note much more problems in Codex's plan and implementation than Gemini could - the ratio is like 10:1.
I occasionally switch the roles between Codex and Claude, and result is the same, Codex could always catch much more problems in Claude's plan and implementation, than Gemini could.
So I am guessing in a relatedly complex codebase, Gemini is much less effective in acting as a guardrail (or a senior engineer/team lead) than the other SOTA models.
I find this very interesting, I wonder if there is a public benchmark that reflects this “red team coding critique” aspect of the current SOTA model that reflects what you have observed.
It would be really useful to observe this in a benchmark vs. the more common “go implement this, or fix this bug” type benchmarks that seem to be prevalent.
Yeah, my tool to automate these review loops is https://github.com/wwind123/coding-review-agent-loop . It's basically a script calling Claude, Codex and Antigravity CLI's. The benefit of using CLI's is, the tool uses quota in your subscription plan of these AI providers, which is much cheaper than using extra tokens from the same providers to do the same thing.
A couple of months ago (before opus-5 and gpt-5.6 sol), The ratio of problems caught by codex/claude vs gemini was more like 2:1 to 3:1. But now it seems codex and claude have made huge leaps and gemini is more or less staying put.
Amazingly, these few days the Gemini 3.8 Flash (High) has been catching much more problems in code reviews than before. I think it started from the second day since I posted the observation above. Maybe somebody from Google saw my posts and tuned some knobs in the model to allow more critical thinking?
Another observation, Gemini's review on code is more critical now, but its review on design plans is still quite agreeable - it tends to approve Codex's design plan immediately, while Claude could often pick out a bunch of problems in the design plan in the first round of reviews.
It's just another mid-tier flash model, nothing exciting, but Antigravity has very generous quotas so it's a good workhorse model when your Claude/OpenAI subs run out.
And whilst it's a fast model, having to baby sit through and approve prompts every few seconds ends up making it slower than the Auto approve modes of Claude/ChatGPT - they definitely need an auto approve mode.
not sure why you are being downvoted, but that has been my experience with 3.7 flash and sol/fable comparisons
i think luna-max has the best cost value offer when it comes to coding, but i note the multi modality of gemini flash as a win
i might consider 3.8 flash for simple side hobby projects or quick scaffolding but would not trust it for long agentic tasks, that really is the realm of sol/fable
agy cli still has a lot of issues not sure if its due to the underlying model hallucinating or the harness or both
Gemini is getting less useful with each update. I could edit a pdf with the 3.1-pro model before but 3.5-pro couldn't edit the given pdf nor it could generate one for me.
If you want to "edit" a PDF, then Claude Sonnet works well, although what it's going to do is regenerate it from scratch trying to retain overall formatting. It can even do this for scanned PDFs and foreign language ones that need translating.
If you just need to create PDFs, not edit them, then Gemini notebook (notebook.google) works well and has Google's usual very high free usage limits.
AFAIK in general you can't really edit PDFs since it's not a reflowable format - even with Adobe tools all that editing does is modify the text within a text box - not reflow the document to adjust to any change in size of the text box.
It is still going to be better at text work, skills, document review, deep reasoning, architecture review, etc. It is only 6 months old, it isn’t like its world knowledge and software knowledge is really out of date. Use it to churn on harder design problems.
In my experience? No. 3.7 is faster and it just seems to get things right more often. Only big architecture tasks and analysis make sense with 3.1, perhaps, but honestly just use the Opus 4.6 to generate a plan and then switch back to flash for the implementation
IME 3.1 Pro still has better system-instruction following than Flash 3.7, esp. when there're many conditions and clauses. 3.1 also writes better prose for technical material than Flash 3.7.
Once the system prompt complexity goes up, Flash starts to write very dense english. it might be fine for tasks like coding, but not for user-facing text meant to be digested by the average person.
3.7-flash has been useless many times, specially when context gets bigger. 3.1 is the only Google model that has seen use from me. With extended thinking, 3.7-flash is kinda usable but not without many problems. I find myself falling back to 3.1 often. I don't believe in any benchmarks because whatever they are doing to award 85% to 3.7 on anything, they should seriously reconsider that test for anything.
I see benchmarks beating sol terra and sonnet. But is actually better? Has someone used it? I don't see actually much people that use Gemini for coding.
Is the Gemini CLI still terrible compared to Claude Code and Codex? The harness the main thing holding back Google models as they could've been the best given all the advantages in compute capacity and training data they initially had, where now even the Google CEO said they're falling behind in agentic tasks, which is sort of a vicious cycle because RLHF relies on human usage.
Honestly, it's platform dependent and "OK" at best, "Mediocre" at worst (Agy on Windows).
Gemini is great via the Chat interface and decent via Github Copilot.
I honestly hate it via Antigravity CLI because their sandboxing system frankly doesn't work. Every other harness has mastered "don't ask me if you're working in this one directory and using common commands". Agy instead either tries to pull a global elevation or wants every tedious variation of a command string whitelisted. Madness - circa 2023.
Agy _really_ needs to make the out-of-the-box experience cleaner and hassle-free. Heck, even Grok CLI "just works".
This may reflect a global mind-shift from "approve and validate everything" to "just do the stuff and only ask permission if it's outside the folder or a command that actually requires elevation". Maybe that's not for everyone, but for those that do want to perform unattended agentic work -- Agy is painful.
Is the google infra stable enough right now? At the start of the year, the flash model was unusable for a whole month via gemini CLI. They could not fix it for a whole month and I was a paid customer.
The flash models, for coding are reckless in my experience. I have a Ultimate subscription, get good quota, but still use Opus 4.6 as it's much more reliable if you manage the context window carefully.
And yet again another failed launch from Google. I pay for their AI plus Google one package to get more cloud storage (have no interest in their AI bundle but you have to pay). and all I see in the Gemini app is 3.6-flash
472 comments
[ 0.24 ms ] story [ 47.2 ms ] threadI eagerly wait more info but sounds like Deepmind without Demis calling the shots has been unleashed and are operating at full speed? Shocker!
At this point it is a meme of course, but where is 3.5 Pro :)
https://x.com/AndrewCurran_/status/2094937419615502370
Sometimes it is hard for a scientist by nature to build and iterate and lead revenue generating products.
The patches were made for Windows 3.1, but not for Windows 3.11 - since the changes resulted in binaries with different offsets, Windows 3.11 used under OS2fW would fail.
EDIT: Source with some discussion of disassembled OS2fW code: https://jacobfilipp.com/DrDobbs/articles/DDJ/1994/9406/9406m...
experience tells me that those people simply have not used models for a long period of time specifically on coding and have run their own comparisons
to someone who uses all vendors, the differences are very palpable and drives purchase decisions.
also keep in mind Gemini and other labs have repeatedly done benchmaxxing, you must have your own benchmarks to evaluate these models.
https://artificialanalysis.ai/models/gemini-3-8-flash shows an intelligence score of 59, the same as Opus 5!
Wow - for a flash model this seems to benchmark powerfully. Remains to be seen what it is like to use.
It looks more like Google execs losing their mind and pressuring researchers to put DeepSWE directly into the training set.
It's clearly been "dealt with" already. When it launched we had interesting gaps and definitely differences. Now every new release is "crushing it".
...on Medium reasoning. Claude Opus 5 (high) is the default in e.g. Claude Code and scores 61. Still very impressive.
Further, opus 5 medium outputs 4x fewer tokens to achieve the same result, negating a lot of the speed difference.
These folks must laugh themselves to sleep. This whole industry hoodwinked the masses. It’s impressive.
With a score of 59, Gemini 3.8 Flash is in eighth place, falling behind even Grok 4.6, Kimi k3, and GLM 5.3.
https://imgur.com/a/BMOJBED
Are you implying Google or Artificial Analysis are reporting false numbers? What's your source?
Nothing here is false, you are simply confused. You either didn't read what they wrote in its entirety or decided to reinterpret what they did write.
All you had to do is go hover your mouse over "Models" in the top bar, hover over Claude Opus 5 and and click on medium: https://imgur.com/mlRCrt1
When you do that you arrive on this page: https://artificialanalysis.ai/models/claude-opus-5-medium
The gemini flash page for reference: https://artificialanalysis.ai/models/gemini-3-8-flash
You have to be an incredibly dishonest person to see a 59 on both pages and say "the initial reported numbers were false and this was simply pointed out. You're changing the subject".
Just wow. Someone actually said this.
anthropic really needs something to address the cheaper end of the market before they get left behind. Sonnet 5 sucks, and Haiku hasn't been updated in a year. meanwhile we've got gemini flash, luna, and GLM5.3 all delivering 90% of the performance for a small fraction of the cost. paying $25/mTok is going to start looking pretty silly soon.
I find GLM5.3 so much better than Sonnet it is not even funny.
Sonnet behaves like a cheap model while being very expensive.
For example, it's not even close to Opus 5 on Terminal-bench 4.0, 19.1% vs. 51.8%.
The result is the same as the previous Gemini 3.6/3.7 Flash days: Claude could always note much more problems in Codex's plan and implementation than Gemini could - the ratio is like 10:1.
I occasionally switch the roles between Codex and Claude, and result is the same, Codex could always catch much more problems in Claude's plan and implementation, than Gemini could.
So I am guessing in a relatedly complex codebase, Gemini is much less effective in acting as a guardrail (or a senior engineer/team lead) than the other SOTA models.
It would be really useful to observe this in a benchmark vs. the more common “go implement this, or fix this bug” type benchmarks that seem to be prevalent.
A couple of months ago (before opus-5 and gpt-5.6 sol), The ratio of problems caught by codex/claude vs gemini was more like 2:1 to 3:1. But now it seems codex and claude have made huge leaps and gemini is more or less staying put.
Another observation, Gemini's review on code is more critical now, but its review on design plans is still quite agreeable - it tends to approve Codex's design plan immediately, while Claude could often pick out a bunch of problems in the design plan in the first round of reviews.
https://x.com/OfficialLoganK/status/2079594867161022817
And whilst it's a fast model, having to baby sit through and approve prompts every few seconds ends up making it slower than the Auto approve modes of Claude/ChatGPT - they definitely need an auto approve mode.
i think luna-max has the best cost value offer when it comes to coding, but i note the multi modality of gemini flash as a win
i might consider 3.8 flash for simple side hobby projects or quick scaffolding but would not trust it for long agentic tasks, that really is the realm of sol/fable
agy cli still has a lot of issues not sure if its due to the underlying model hallucinating or the harness or both
If you just need to create PDFs, not edit them, then Gemini notebook (notebook.google) works well and has Google's usual very high free usage limits.
AFAIK in general you can't really edit PDFs since it's not a reflowable format - even with Adobe tools all that editing does is modify the text within a text box - not reflow the document to adjust to any change in size of the text box.
Once the system prompt complexity goes up, Flash starts to write very dense english. it might be fine for tasks like coding, but not for user-facing text meant to be digested by the average person.
I haven't tested 3.8 on my workload yet.
Gemini is great via the Chat interface and decent via Github Copilot.
I honestly hate it via Antigravity CLI because their sandboxing system frankly doesn't work. Every other harness has mastered "don't ask me if you're working in this one directory and using common commands". Agy instead either tries to pull a global elevation or wants every tedious variation of a command string whitelisted. Madness - circa 2023.
Agy _really_ needs to make the out-of-the-box experience cleaner and hassle-free. Heck, even Grok CLI "just works".
This may reflect a global mind-shift from "approve and validate everything" to "just do the stuff and only ask permission if it's outside the folder or a command that actually requires elevation". Maybe that's not for everyone, but for those that do want to perform unattended agentic work -- Agy is painful.