I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?
You can use it for free with Google AI studio (free tier or paid tier accounts with different limits). Or use the paid version from Vertex AI which is around 3x cheaper than Gemini 3 Flash.
I'm using Gemma 4 31B in my app with 5 agents, 1.5k requests per day, each.
nice, will run it later agains qwen3.6 27b, the speed was one of the reasons why in was running qwen and not gemma. the difference was big, there is some magic that happpens when you have more then 100tps.
Watching the computer write text sort of reminds me of using a modem to call a BBS in the old days. This seems like going from 300 baud to 1200 - a significant improvement, but still pretty slow, and someday we will wonder how we put up with it.
I recently set up the 26B A4B model up on vLLM on an RTX3090 (4-bit) after a hiatus from local models. Just completely blown away by the speed and quality you can get now for sub-$1k investment.
I tried first with Qwen but it was unstable and had ridiculously long thinning traces!
Really excited to try this once it is merged into llama.cpp.
Gemma 4 26B-A4B is much quicker on my setup vs Qwen3.6-35B-A3B (by about 3x), so the thought of a 1.5 speedup is tantalizing.
Have tried draft models to limited success (the smaller 3B draft model in addition to a dense 14B Ministral model introduced too much overhead already)
Google is singlehandedly carrying western open source models. Gemma 4 31B is fantastic.
However, it is a little painful to try to fit the best possible version into 24GB vram with vision + this drafter soon. My build doesn't support any more GPUs and I believe I would want another 4090 (overpriced) for best performance or otherwise just replace it altogether.
I found that Gemma 4:26b makes way more mistakes compared to Qwen and Gemma 3. Gemma3 27b QAT was my goto for some time as this was quite fast. Qwen is still king for a balance of accuracy and inference speed.
Gemma:31b was more accurate but speed was horrendous.
In my testing the Gemma 4 31b model had the biggest speed boost in Ollama w/ the MLX runner for coding tasks (at about 2x). Unfortunately you'll need a pretty beefy Mac to run it because quantization really hurts the acceptance rate. The three other smaller models didn't perform as well because the validation time of the draft model ate up most of the performance gains. I'm still trying to tune things to see if I can get better performance.
You can try it out with Ollama 0.23.1 by running `ollama run gemma4:31b-coding-mtp-bf16`.
I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance.
It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now under performing leading open models by 5-10%, but doing it in 1/10th the time.
I think you can see this one of two ways: you could also consider it a miracle that the qwen models are able to perform so well when being trained on inefficient wrapper code data.
GPT 5.5/5.4 are the smartest models, but at great token / code bloat cost. Qwen 3.6 Max strikes a good balance. But Gemma 4 26B writes some really efficient code, with great results considering the model size. Things do start falling apart under higher contexts.
Gemini models, even if not so good at coding, are also competitive with GPT-5.5 and Claude Opus 4.7 in a lot of tasks while having considerably less parameters.
> It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt.
Yes, Gemma 4 is very promising for its strong performance and token efficiency, but it's unfortunate that it's sliding window attention has a fatal flaw that makes me seriously hesitate to rely on it. See the series of videos on this channel:
On top of Qwen3.5/3.6's superior recall, it's attention mechanism dramatically reduces KV cache requirements, so you can fit longer sessions in the same VRAM (or more concurrent sessions if you have agents running), which is critical for local hosting.
At this point Qwen3.6 with thinking mode disabled seems like the best balance.
I like Google’s business model more than the other frontier model providers: sustainable. One thing I don’t like with Gemini Ultra is no visibility into token use or what the cost would be. I have been planning on letting my Ultra subscription expire and go with OpenCode with a fast inference provider to get this visibility, but this discussion thread gave me the idea of also trying the paid APIs with AntiGravity instead of a subscription. When I sit down to do a specific task I want accurate token usage and $$ data as I work.
60 comments
[ 3.6 ms ] story [ 62.3 ms ] threadI'm using Gemma 4 31B in my app with 5 agents, 1.5k requests per day, each.
The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.
https://docs.nvidia.com/megatron-core/developer-guide/0.15.0...
I tried first with Qwen but it was unstable and had ridiculously long thinning traces!
They serve gemma-4-26b-a4b-it.
Gemma 4 26B-A4B is much quicker on my setup vs Qwen3.6-35B-A3B (by about 3x), so the thought of a 1.5 speedup is tantalizing.
Have tried draft models to limited success (the smaller 3B draft model in addition to a dense 14B Ministral model introduced too much overhead already)
However, it is a little painful to try to fit the best possible version into 24GB vram with vision + this drafter soon. My build doesn't support any more GPUs and I believe I would want another 4090 (overpriced) for best performance or otherwise just replace it altogether.
google/gemma-4-31B-it-assistant
google/gemma-4-26B-A4B-it-assistant
google/gemma-4-E4B-it-assistant
google/gemma-4-E2B-it-assistant
Gemma:31b was more accurate but speed was horrendous.
You can try it out with Ollama 0.23.1 by running `ollama run gemma4:31b-coding-mtp-bf16`.
It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now under performing leading open models by 5-10%, but doing it in 1/10th the time.
GPT 5.5/5.4 are the smartest models, but at great token / code bloat cost. Qwen 3.6 Max strikes a good balance. But Gemma 4 26B writes some really efficient code, with great results considering the model size. Things do start falling apart under higher contexts.
Yes, Gemma 4 is very promising for its strong performance and token efficiency, but it's unfortunate that it's sliding window attention has a fatal flaw that makes me seriously hesitate to rely on it. See the series of videos on this channel:
https://youtu.be/ONQcX9s6_co?si=Yt55_N4DcNLstnGS
On top of Qwen3.5/3.6's superior recall, it's attention mechanism dramatically reduces KV cache requirements, so you can fit longer sessions in the same VRAM (or more concurrent sessions if you have agents running), which is critical for local hosting.
At this point Qwen3.6 with thinking mode disabled seems like the best balance.
I'm not seeing any update to the app on my android phone... maybe later today?
>We’ve published an in-depth technical explainer
I was expected a pdf link, but this goes to a brief article on twitter/X. lol, okay...
Beta but useable
Focusing more on performance to compute efficiency over pure performance. And maybe that’s why Gemini is (seemingly) lagging behind?
Other providers hitting capacity and hitting the limits subsidising their inference.
Google strategy seems to be about scaling and distributing these models to their existing billions of users.
Yeah, part of that is installing a model in chrome to millions of users without consent.