3.8 actually performs slightly worse than 3.6 on AA-Omniscience Accuracy, which could imply that they traded out world knowledge for capability in other areas.
It also produces nearly twice as many tokens per task as 3.6 (and by extension, time), which may be a tradeoff required to achieve correctness at this parameter size.
Imo it makes sense for things to move in the direction of small, focused models that excel in one area. I use LLMs for technical work 99% of the time, I could care less about general world knowledge, or if the model is good at creative writing.
With good orchestration and delegation you can get surprisingly far with small models running on consumer hardware.
Agreed. Luckily, this model also scores high in AA non-hallucination, so it knows what it doesn't know -- perfect for situations where it can just tool call a web search.
The biggest untapped market is pure agentic models that are built for tool calling and non hallucination instead of memorizing facts. You need some world knowledge (as in common sense) to build a useful model, but I don't think perfect recall on general QA is a good use of space when you have web search and structured knowledge in Wikidata or Wolfram Alpha.
Training should focus on tasks that require real intelligence instead of memory. Creative writing is actually good for this if you score it on coherence instead of getting random real life details right. Basic level of coding (simple prompt to code, don't need to one shot complex projects) is also great because writing a small script is more efficient than 20 separate tool calls.
With respect to Wolfram Alpha, it's worth noting that VibeThinker-3B is basically a match for the larger frontier models -- hundreds of times larger -- in the narrow domain of mathematical and logical reasoning problems. It doesn't seem necessary to resort to external models or tools for that, at least in principle.
Yep, exactly. I keep saying that I want the “coding expert” extracted from these multi-T parameter models to run locally on reasonable hardware (large laptops, not servers). Yea, I know there’s no single “coding expert” that you can actually extract in these models, but you get what I mean. Like you said, when I’m coding, I don’t care about world knowledge, and I’m fine with consulting another model when I need that.
I'm torn on this - on the one hand performance matters, on the other so does capability.
I could run Qwen 3.6 27B on my laptop, but at 5 tok/s it was too slow even without overthinking - I never used it. OTOH, Qwen 3.6 35B A3B ran at 20 tok/s but it just could not get done what I asked of it. It sort of got close but you had to repeat and retry so much that it might have been faster to run 27B dense... maybe?
So that said, I might take a much better model that runs 2-3x slower (total time per task) but that's more capable over a faster, less capable one.
I'd also like to try a proper "plan-then-execute" type execution where thinking is entirely disabled (or low) during the execution stage but enabled/max during the planning stage.
I will definitely give 3.8 27B a better shot than 3.6 though.
Depends on your use-case. Over the past couple days, I've found 192k context more than enough for coding. There's more thinking for sure compared to comparably sized models (running on xhigh), but I've found the results are so much better that the entire session consumes less tokens on average since weaker models need more review passes.
Well, ackshually. E do know exactly how big glm 5.2 is. And there's more than enough data to draw conclusions about luna. Or are you one of the guys who says "big bang is just a theory"?
A lot of the benchmarks seem often near meaningless these days - really bench-maxxed to the hilt. I tend to still look at the Artificial Analysis rankings to get at least an idea on relative performance of models, is that still warranted?
What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?
Do you have any evidence that this model is bench-maxxed? I know that's particularly difficult to quantify. If there is an indicator of bench-maxxing, that just becomes the new benchmark to benchmax.
Here, filtered down for you. [0] Look at the individual benchmarks, not the combined one. You can tell that this model is much more benchmaxxed as its relative ranking swings between benchmarks is much larger. This is a hallmark.
Sounds obvious but just try using the models for anything outside the evals. Take something arcane from Greek history, use it to create a masked linguistic puzzle, which you then ask the model to solve mathematically, all wrapped as an ask to generate ASCII art. Yes, all these elements exist in some form in the evals but the key is in how utterly unconventional the elements are that you pick and in how you combine them.
I have consistently noticed Opus 4.8 and GPT-5.6 far outshine the Chinese models. Gemini is sort of middle of the road, Grok is better than Gemini but not really close to Opus/GPT. OAI & Anthropic still remain unbeaten by a wide margin in my eyes.
Certainly, real-life is the ultimate benchmark. But for various reasons that isn't always immediately possible to go evaluate a model on.
Maybe my Greek idea sounded too high falutin' or simply seemingly clever (I give an example below -- try it out!).
So here's something way simpler that Qwen3.8 27B does not get; only GLM-5.2 and K3 do.
**
Analyze the 2 structural (not semantic) patterns in this text:
Morning revient.
Birds saluent Morgenlicht.
We suivons Waldwege toward maison.
Rain tombe plötzlich; we cherchons Schutz beneath sapins.
Night vient langsam; we trouvons Wärme near le Feuer, sharing quelques Geschichten together.
**
The answer should get not just the obvious E-F-G pattern but also the word counts being Fibonacci. Surprisingly few models get this. The only way I got Qwen to do this was on the 2.4T model, with extensive prompt scaffolding.
This also seems like quite an esoteric use-case to me, but I guess some people might need to know when text follows a fibonacci sequence in terms of word count.
That problem sounds reminiscent of one I like to use as a benchmark, which is to request that the model create an .SVG of a logarithmic spiral of 50 numbered stones. Qwen 3.8 27B absolutely knocked that one out of the park, where a lot of larger models have failed outright or otherwise performed suboptimally.
Can you share an example of the Greek-history puzzle prompts you're talking about?
This one is very benchmaxxed, and you can tell from this page alone. Look at the huge variance in ranking per benchmark. Most models, including at that size, are much more consistent.
IMO, ELO rating from The Intelligence company and arena.ai are more representative of rankings since they use humans to judge a head-to-head comparison between a couple models at a time.
https://www.intelligence.ai/http://arena.ai
That is actually insane. In my opinion, the future is local AI: for most daily tasks, you absolutely don't need Fable level intelligence - you need Fable level agentic capabilities. And this model has (almost) just that. If we get a similarly capable MoE model in a few months (yes we will), it's going to be an utterly wild ride.
Thank you for writing this; I have delayed 3.8:27b because my ollama needs to be updated (and I'm too lazy); but if I can save thousands of dollars not buying GPU hardware – for Top-10 performance – yes I will unlazy myself and update local ollama software.
Simon Willison's post about this gives a good context on why exactly this is happening. While it doesn't mention this in the Artificial Analysis page, this is likely with Max reasoning, which has extremely long reasoning traces:
I'm curious if they can make up for this with insanely high tokens-per-second especially when served from hosted providers, though, given how tiny it is (37B!)
Qwen models are slower in tokens/s, compared to similarly sized gemma4 and others, and they use more tokens per task, in part thanks to that xhigh default.
On the other hand, there are some of us who are stuck with hardware that has plenty of compute, but limited (V)RAM. The new 27B is just perfect for that.
What configuration are you using? On both vllm and llama-cpp, I get significantly higher speeds from gemma4 than qwen3.6 (with their respective speculative decoding methods).
Output TPS in vllm for instance:
- Gemma4 26B-A4B: 200-300TPS
- Qwen3.6 35B-A3B: 120-180TPS
- Gemma4 31B: 80-120TPS
- Qwen3.6 27B: 60-80TPS
This is for a first request on a dual 5090 setup, with their respective speculative decoding methods.
> with their respective speculative decoding methods
You're benchmarking drafter acceptance rate, then. Which is real life values, yes, but attributing worse drafter performance to the other 95% of the model being inherently slower.
Yes, I mentioned the setup, but on vllm you can only use TP with speculative decoding or pipeline parallelism without, so there's tradeoff to both.
I gave general numbers of what I'm getting above, the performance ratios seemed similar regardless of setup (eg. getting a AWQ-in4 quant on a single GPU vs PP without speculative decoding vs TP with speculative decoding).
Overall single GPU is fastest, and TP+speculative decoding is still faster than PP, but for fp8 models you need dual GPUs whether you want it or not.
Use claude/codex/whatever with /goal to optimize params for you.
IMHO draft model support on dense models is great alternative to MoE on GPUs (high bandwidth, less memory) – more intelligence, speed somewhere mid way there which is usually sufficient.
I took the liberty of adding your reasoning effort chat template to my setup. You can play around with the last few parameters. In generall VLLM will be better in higher concurrency scenarios, so if you only use it for a personal vibe coding assistant and less as a general home model for task execution llama.cpp may be better.
personal use doesn't necessary mean no concurrency.
it's good to play with harness setup where you fan out multiple concurrent branches that share non trivial amount of prefix then reduce their output/summary back into main agent.
ie. instead of serially reading further skills/relevant source files for planning/thinking, you can branch and read them in parallel reusing prefix / or use to to approach request from different angles in parallel - to map-reduce result onto main context of what's actually relevant. branching subagents has benefits of not polluting main context, shared prefix prefill is close to free on a cache hit and with concurrent decoding/continuous batching you can utilize gpu well to get good speedups.
ie. what's relevant is number of active concurrent sequences (and their shape, ie. shared prefix), not so much number of users.
i'm not sure with llama.cpp vs vllm regarding concurrency – llama server has multiple server slots, continuous/dynamic batching enabled by default, prompt caching (also on by default), ram prompt cache, context checkpoints, unified kv buffer across sequences etc. so shouldn't be bad, i guess would be good to actually benchmark. personally i'm happy with llama.cpp.
It’s still going to chew up context quickly. Surely, some of the added tokens are helping the model, but does it require as many as it generates? What happens on long, multi step tasks as it pushes old tokens out of context? I’m not sure we know the answers to those.
> insanely high tokens-per-second especially when served from hosted providers, though, given how tiny it is (37B!)
It's a dense model so it will use all of its parameters per token. 37B active parameters isn't tiny at all, it's almost what Deepseek R1 had, and it's 2/3 of what Kimi k3 uses, so it's not going to be “insanely high” tps: it's going to be three times slower than Deepseek Flash (Prefil speed is going to be quite high though, but not token generation).
Its 27B not 37B and having just 27B in total and 3T and like 30B active of those is still totally different. A 120B with 5B active is still much slower than a proper 5B. Just like the new Ling 3.0 Tiny with 8B and 1B active only gets around 120tk/s compared to 250tk/s which a real 1B one gets on my hardware.
On the other hand, it used about the same tokens as GLM 5.2 and got 1 point lower score.
The fact that we have a GLM 5.2-class model that can run on two 3090's comfortably at Q8 is absolutely insane. It wasn't long ago that GLM 5.2 was considered amazing for open weight models.
i feel like Simon omitted an important part of how Qwen's "reasoning" levels work. they are just one sentence additions/omissions to the system prompt
xhigh -> "Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer."
medium -> no mention of effort (sentence omitted)
low -> "Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration."
in my testing this doesn't seem to produce exactly deterministic thinking levels, because it's just a system prompt nudge. i had instances where medium thought longer than xhigh
The models are post-trained on these prompt additions so they’re more structural than thinking of them as “system prompts” suggests. (All LLMs ever see is tokens going in, so even the concept of a system prompt is just formatting they’ve seen in post-training.)
You can also apply fixed token budgets for the reasoning blocks, though it will decrease quality in some cases.
yes of course, I understand that. but I feel like it would've been nice to include in the article because the main point of it is the effort and overthinking
Why not invent a few magic token values for reasoning level instead? It would be like 4 out of a vocabulary of 200k and save like 30 tokens in every prompt
Perhaps model size and reasoning length trade off to some extent, similar to CPU vs. RAM. A smaller model with a longer reasoning trace has more intermediate structure to latch onto and build on.
Interesting to see where local models are going to be in the coming days. I am already starting to believe open source models are the way to go in the coming days. With Qwen 3.8 Max, Kimi K3 etx already delivering at part perf with frontier models, the future is going to be exciting.
I purchased a 5070Ti (16GB nVIDIA GPU) a few months ago, and it is absolutely incredible what can be accomplished on local hardware (whether offline or not).
Don't forget `mistral-small` (Apache's LLM), which to me is equally as impressive as qwen3.5 (only benefit of qwen is seeing the pre-text reasoning is often more helpful than the actual text output).
To my surprise, providers on OpenRouter (io/akash/chutes) are serving Qwen3.8 27B at ~ $0.4 (in) / $3 (out) / $0.25 (cache), more expensive than DeepSeek v4 Flash.
i think there definitely is some truth to this in terms of embeddings spaces, which is why i believe they are implemented by OpenAI/Anthropic in roughly highest import => least import bit order-- an overwhelming majority of the variance is in the first few hundred vector bits. i haven't actually tested this myself by manually truncating vectors, but it is my understanding that they generally speaking have this property.
It would be a very nice model at 200-300 tps and if it was dirt cheap.
What's the limiting factor of optimizing speed and price for inference providers?
That might actually compensate for the overthinking, if it can think really fast. Dense models are easier than MoE to put on silicon. https://chatjimmy.ai/ is getting 16k tps with an 8B model. Extrapolating that gives nearly 5k tps for 27B. And we're still early in this technology.
If tps is so high, a compaction step could be performed over every thinking turn to keep context size down.
I have an internal automated benchmark, which roughly follows my workflow, and I've been testing various models on it, local and cloud. Qwen 3.8 27B did awesome. Its understanding is correct, research is better than e.g. glm's (and I like glm), and implementation is good and careful.
Qwen 3.8 27B doesn't look benchmaxxed. These "52 AA score" numbers feel real, which is surprising. I've been using it locally for a few days for other tasks as well. If not the speed, I'd be totally happy to use it as a daily driver instead of cloud models, it is that good.
---
(benchmark, to get an idea):
1. First, initial prompt which is not super precise - similar to how I'd write a task when talking e.g. to Opus. I'm describing an idea, and asking model to come up with some plan, and also to criticize the approach. Task is about implementing a particular pi extension. I'm checking if a model actually understands what I'm asking.
2. Then, as a follow-up, I ask to research alternative implementations, research UX of similar extensions, etc. It needs to do web searches, inspect open source codebases, read articles and papers, etc. I don't prompt to do this exactly, but I expect good models to figure out they need to do it.
3. Then, implementation.
Also, one finding: Q4 and Q8 seem to have very different behavior in this benchmark. Q4 produces 2-3x thinking in the end, and makes more turns - it seems it makes more mistakes, and needs effort to recover from them, while Q8 gets more things right in a first try. In the end, quality is roughly similar, but Q8 gets there much faster, especially the implementation (tried it several times). Could be a difference between concrete artifacts, or between runtimes, I don't know, but be careful - it seems the real-world experience with qwen 3.8 27B can be vastly different, depending on how it's set up.
Regarding DeepSeek 0731 vs Qwen 3.8 27B. On this benchmark, Qwen understand my intent better, it's better at research, and I also liked its implementation more. But: if you're more precise in what you ask, 0731 is also very good, and it's quite a lot faster on mac; raw speed is better, and it needs less thinking to get there. So, I'd say it's a tie in practice, both are awesome :)
yeah but using the rule of thumb (I see floating in the Internet) which is sqrt(#params * #active params) which would give sqrt(284B*13B) = 60B, so deepseek should perform better.
I’ve been using v4 Flash 0731 a lot lately and you can’t beat the price performance. That said, it sometimes takes my prompts as more of a suggestion than a directive. I’ve found that introducing a reviewer subagent (even with the same model) helps push it back to what I’ve asked for. But makes every coding session a back and forth: “do X” -> “use a reviewer subagent to analyze whether you really did X as I asked”.
For cost reasons, I’ve been using Flash for the reviewer, too, but I plan on trying to use Pro for that. Thus far, however, Flash has been doing well at reviewing. I’m cheap as I’m paying for all the tokens myself.
deepseek-v4-flash-0731 has been awfully prone to infinite looping output in reasoning for me, and once it hallucinated in the middle of going in circles that I had instructed it to start using Yoda-speak, which… I don’t have any idea where that came from.
I have had it loop once. I had to abort the turn, clear the context (maybe I just compacted it?) and restarted. That seemed to clear it up. In general, I have not found that V4 Flash 0731 loops a lot or even any more than other models.
How were you accessing the model? I was using ollama.com’s cloud hosting the first few times I ran into it, and then it happened again the first time I tried Fireworks.ai before I made it even 10% of the way through the context window.
How is the general knowledge of Qwen 3.6?
Do you need to explain things outside of algorithms to it?
Since the size is so small, I guess you need more explanations to it.
General knowledge helps with coding when your don't specify a lot of details and ask for big changes.
> It probably could only happen in China, not in current USA or EU (for different reasons)
Per Artificial Analysis benchmarks, Meta's Muse Glimmer 30b (open weight) holds its own (for agentic code workloads) against models 5x to 10x its size, too.
I found Glimmer underwhelming in terms of coding - I tried it as a drop-in replacement for 3.6, and the output was noticeably worse. 3.8 has been a significant step up so far from early testing.
163 comments
[ 0.27 ms ] story [ 26.3 ms ] threadSame score as the latest DeepSeek Flash 0731 which has 284B parameters! (13B active)
Its also the second best Qwen model, much better than Qwen 3.7 Max, but significantly below Qwen 3.8 Max.
3.8 actually performs slightly worse than 3.6 on AA-Omniscience Accuracy, which could imply that they traded out world knowledge for capability in other areas.
It also produces nearly twice as many tokens per task as 3.6 (and by extension, time), which may be a tradeoff required to achieve correctness at this parameter size.
With good orchestration and delegation you can get surprisingly far with small models running on consumer hardware.
Training should focus on tasks that require real intelligence instead of memory. Creative writing is actually good for this if you score it on coherence instead of getting random real life details right. Basic level of coding (simple prompt to code, don't need to one shot complex projects) is also great because writing a small script is more efficient than 20 separate tool calls.
I could run Qwen 3.6 27B on my laptop, but at 5 tok/s it was too slow even without overthinking - I never used it. OTOH, Qwen 3.6 35B A3B ran at 20 tok/s but it just could not get done what I asked of it. It sort of got close but you had to repeat and retry so much that it might have been faster to run 27B dense... maybe?
So that said, I might take a much better model that runs 2-3x slower (total time per task) but that's more capable over a faster, less capable one.
I'd also like to try a proper "plan-then-execute" type execution where thinking is entirely disabled (or low) during the execution stage but enabled/max during the planning stage.
I will definitely give 3.8 27B a better shot than 3.6 though.
What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?
[0] https://artificialanalysis.ai/models/qwen3-8-27b?models=deep...
They have simply decided to not train the model in some areas such as world physics
I have consistently noticed Opus 4.8 and GPT-5.6 far outshine the Chinese models. Gemini is sort of middle of the road, Grok is better than Gemini but not really close to Opus/GPT. OAI & Anthropic still remain unbeaten by a wide margin in my eyes.
Surely most of your use-cases are not novel tasks that combine obscure domains.
It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.
Maybe my Greek idea sounded too high falutin' or simply seemingly clever (I give an example below -- try it out!).
So here's something way simpler that Qwen3.8 27B does not get; only GLM-5.2 and K3 do.
**
Analyze the 2 structural (not semantic) patterns in this text:
Morning revient. Birds saluent Morgenlicht. We suivons Waldwege toward maison. Rain tombe plötzlich; we cherchons Schutz beneath sapins. Night vient langsam; we trouvons Wärme near le Feuer, sharing quelques Geschichten together.
**
The answer should get not just the obvious E-F-G pattern but also the word counts being Fibonacci. Surprisingly few models get this. The only way I got Qwen to do this was on the 2.4T model, with extensive prompt scaffolding.
Can you share an example of the Greek-history puzzle prompts you're talking about?
https://benchlm.ai/models/qwen3-5-27b#ledger
https://simonwillison.net/2026/Aug/16/qwen-38-27b/
It seems like the token usage is 2.3x GPT Luna Max and almost 2x Kimi K3!
https://imgur.com/a/dDSyhr2
I'm curious if they can make up for this with insanely high tokens-per-second especially when served from hosted providers, though, given how tiny it is (37B!)
On the other hand, there are some of us who are stuck with hardware that has plenty of compute, but limited (V)RAM. The new 27B is just perfect for that.
No? Gemma 31B and Qwen 27B are about the same speed. Gemma 26B-A4B and Qwen 35B-A3B are about the same speed.
Output TPS in vllm for instance:
- Gemma4 26B-A4B: 200-300TPS
- Qwen3.6 35B-A3B: 120-180TPS
- Gemma4 31B: 80-120TPS
- Qwen3.6 27B: 60-80TPS
This is for a first request on a dual 5090 setup, with their respective speculative decoding methods.
You're benchmarking drafter acceptance rate, then. Which is real life values, yes, but attributing worse drafter performance to the other 95% of the model being inherently slower.
I gave general numbers of what I'm getting above, the performance ratios seemed similar regardless of setup (eg. getting a AWQ-in4 quant on a single GPU vs PP without speculative decoding vs TP with speculative decoding).
Overall single GPU is fastest, and TP+speculative decoding is still faster than PP, but for fp8 models you need dual GPUs whether you want it or not.
IMHO draft model support on dense models is great alternative to MoE on GPUs (high bandwidth, less memory) – more intelligence, speed somewhere mid way there which is usually sufficient.
Here's a VLLM command for 3.6 (I'll update to 3.8 today) to test out:
```
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
vllm serve Qwen/Qwen3.6-27B-FP8 \
--dtype auto \
--kv-cache-dtype fp8 \
--enable-chunked-prefill \
--enable-prefix-caching \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' \
--default-chat-template-kwargs '{
```I took the liberty of adding your reasoning effort chat template to my setup. You can play around with the last few parameters. In generall VLLM will be better in higher concurrency scenarios, so if you only use it for a personal vibe coding assistant and less as a general home model for task execution llama.cpp may be better.
it's good to play with harness setup where you fan out multiple concurrent branches that share non trivial amount of prefix then reduce their output/summary back into main agent.
ie. instead of serially reading further skills/relevant source files for planning/thinking, you can branch and read them in parallel reusing prefix / or use to to approach request from different angles in parallel - to map-reduce result onto main context of what's actually relevant. branching subagents has benefits of not polluting main context, shared prefix prefill is close to free on a cache hit and with concurrent decoding/continuous batching you can utilize gpu well to get good speedups.
ie. what's relevant is number of active concurrent sequences (and their shape, ie. shared prefix), not so much number of users.
i'm not sure with llama.cpp vs vllm regarding concurrency – llama server has multiple server slots, continuous/dynamic batching enabled by default, prompt caching (also on by default), ram prompt cache, context checkpoints, unified kv buffer across sequences etc. so shouldn't be bad, i guess would be good to actually benchmark. personally i'm happy with llama.cpp.
It's a dense model so it will use all of its parameters per token. 37B active parameters isn't tiny at all, it's almost what Deepseek R1 had, and it's 2/3 of what Kimi k3 uses, so it's not going to be “insanely high” tps: it's going to be three times slower than Deepseek Flash (Prefil speed is going to be quite high though, but not token generation).
The fact that we have a GLM 5.2-class model that can run on two 3090's comfortably at Q8 is absolutely insane. It wasn't long ago that GLM 5.2 was considered amazing for open weight models.
xhigh -> "Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer."
medium -> no mention of effort (sentence omitted)
low -> "Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration."
in my testing this doesn't seem to produce exactly deterministic thinking levels, because it's just a system prompt nudge. i had instances where medium thought longer than xhigh
You can also apply fixed token budgets for the reasoning blocks, though it will decrease quality in some cases.
But I think the next step will be even more thinking on smaller models. Maybe fine-tuned and we get really crazy stuff
Don't forget `mistral-small` (Apache's LLM), which to me is equally as impressive as qwen3.5 (only benefit of qwen is seeing the pre-text reasoning is often more helpful than the actual text output).
To my surprise, providers on OpenRouter (io/akash/chutes) are serving Qwen3.8 27B at ~ $0.4 (in) / $3 (out) / $0.25 (cache), more expensive than DeepSeek v4 Flash.
https://openrouter.ai/qwen/qwen3.8-27b / https://archive.vn/RrDGO
Open Router
Input /M $0.45
Output /M $3.20
Cache read /M $0.05
Throughput 27 tps
It would be a very nice model at 200-300 tps and if it was dirt cheap. What's the limiting factor of optimizing speed and price for inference providers?
If tps is so high, a compaction step could be performed over every thinking turn to keep context size down.
https://news.ycombinator.com/item?id=49308715
Qwen 3.8 27B doesn't look benchmaxxed. These "52 AA score" numbers feel real, which is surprising. I've been using it locally for a few days for other tasks as well. If not the speed, I'd be totally happy to use it as a daily driver instead of cloud models, it is that good.
--- (benchmark, to get an idea):
1. First, initial prompt which is not super precise - similar to how I'd write a task when talking e.g. to Opus. I'm describing an idea, and asking model to come up with some plan, and also to criticize the approach. Task is about implementing a particular pi extension. I'm checking if a model actually understands what I'm asking.
2. Then, as a follow-up, I ask to research alternative implementations, research UX of similar extensions, etc. It needs to do web searches, inspect open source codebases, read articles and papers, etc. I don't prompt to do this exactly, but I expect good models to figure out they need to do it.
3. Then, implementation.
Also, one finding: Q4 and Q8 seem to have very different behavior in this benchmark. Q4 produces 2-3x thinking in the end, and makes more turns - it seems it makes more mistakes, and needs effort to recover from them, while Q8 gets more things right in a first try. In the end, quality is roughly similar, but Q8 gets there much faster, especially the implementation (tried it several times). Could be a difference between concrete artifacts, or between runtimes, I don't know, but be careful - it seems the real-world experience with qwen 3.8 27B can be vastly different, depending on how it's set up.
Regarding DeepSeek 0731 vs Qwen 3.8 27B. On this benchmark, Qwen understand my intent better, it's better at research, and I also liked its implementation more. But: if you're more precise in what you ask, 0731 is also very good, and it's quite a lot faster on mac; raw speed is better, and it needs less thinking to get there. So, I'd say it's a tie in practice, both are awesome :)
Which harness for the benchmark ?
You have previously commented on using OC/GLM. R u going to stock with it?
pi, with a plugin to do web search / web fetch.
> You have previously commented on using OC/GLM. R u going to stock with it?
For personal use - probably yes, z.ai + kimi + opencode go subscriptions, with some share of local models now. For work - claude code, codex.
Once you send your benchmark to "cloud", I don't think you can rely on it being secret/private any longer.
And I just can't fathom that the new 3.8 beats the new DeepSeek V4 Flash (which, in my eyes, is one of the best everyday coding models).
What an insane release, and convenient size to use every day/locally.
but i will test this model extensivly.
Were your opinions effected by the harness ?
DS is an amazing combo. It probably could only happen in China, not in current USA or EU (for different reasons)
Per Artificial Analysis benchmarks, Meta's Muse Glimmer 30b (open weight) holds its own (for agentic code workloads) against models 5x to 10x its size, too.