Can you please try and see how many tokens you get with some form of concurrency. Pretty much ALL the benchmarks I've seen on the more accessible cards are just single request.
Because concurrency with a single "accessible" card quickly diminishes. I have dual 3090s, and on Qwen 3.6 35B A3B at 80k a single card with max concurrent set to 4 will get 70-80tps single request, 50-60 TOTAL tps with 2, 45-50 with 3, and around 40tps with all 4 going.
I know this doesn't exactly match your request, but I'm happy with my 3.6's performance. I've seen people claim they can get 120+tps on a 3090, but I'm not impatient when it comes to streaming text faster than I can read it.
Your configuration is broken or wrong. What are you using? Hopefully not llama.cpp?
I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.
I haven’t tried any larger models, but a 12B model on my Ampere A5000 gets around 4-5x the aggregate throughput with concurrency. I have the maximum context configured to 32k, but my actual requests are usually around 2-4k tokens.
No idea how that compares to running a larger model and context though.
I guess it's due to testing on MoE. Different completions activate different experts, thus very little cache reuse and completions "steal" memory bandwidth from each other.
As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass.
With dense Qwen 27B on 3090 I get 1x40, 2x35 and 4x30 t/s.
Its egregious the quant level isnt disclosed along the "Qwen" string. Everyone knows theres huge difference in speed/quality along the quant axis, I now attribute the ommision of such to deliberate choice to not curb the hype of the tittle.
Which model/quant/command line did you use? I can barely get 100 token/secs and for sure clearly not a full context. With vllm, I am limited to 130k tokens with vllm + nvfp4.
yeah, I was confused during the whole thing, i get 70 t/s on a 3090, which evens out around 50 t/s at 128k+ , have been running 3.6 and now 3.8 (both iq4_nl at 256k q4 kv) on the 3090 for months. I am confused as to what we 'discovered' here, it's a common config. and at less than 1/2 the price of the gpu (and double the bandwidth, though no fp4 cores to be fair).
A 3090 has 936 GB/s of bandwidth vs 432 GB/s on the 70W RTX PRO 4000 SFF, so 70 t/s there is not surprising. The interesting constraint here was fitting a workload-tuned 5.01 BPW quant + 256K + MTP into 24 GB while working with less than half the memory bandwidth
they talk about the quants they tried in the article and settle on a Qwen3.8-27B-Hermes-iMatrix-NVFP4-Balanced.gguf which they calibrated on their own session traces and they pulled in 5 different llama.cpp pull requests to their local llama-server.
5.01 BPW custom hybrid: bulk NVFP4, selected Q5_K/Q6_K tensors from an iMatrix, Q6_K embeddings and Q8_0 lm_head. The iMatrix was built from 5,472 messages across 296 real Hermes sessions
The whole site looks like and reads like AI slop. The outcomes also don't make any sense and don't feel rigorously tested (no, having claude test for you doesn't count as rigorous).
Please stop making this comment. The war is lost. Instead, you should be commenting that it looks like a human wrote this when you come across the rare brain-produced writing
"Combining them into one heroic speedup would make a better headline and a worse benchmark."
"The machine immediately taught me that capacity estimates are just admission tickets."
"Useful in production, poison in a kernel comparison."
Please don't publish writing like this, it's exhausting to read. You can edit that stuff out.
The best part of this is the "Five xhigh artifacts from the finished model" section at the bottom, I suggest either moving that up or at least prominently promoting it at the top of the article.
Apparently we've blown way past the Turing test and approaching AGI and yet LLM-generated text still sticks out like a sore thumb. Maybe LLMs aren't that good at writing after all.
Its not quite that simple here. The iMatrix-guided hybrid uses different quantization levels per tensor/layer, so there isnt one honest Q4/Q5/NVFP4 label I can put in the title
31 comments
[ 0.23 ms ] story [ 55.9 ms ] threadI know this doesn't exactly match your request, but I'm happy with my 3.6's performance. I've seen people claim they can get 120+tps on a 3090, but I'm not impatient when it comes to streaming text faster than I can read it.
I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.
No idea how that compares to running a larger model and context though.
As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass.
With dense Qwen 27B on 3090 I get 1x40, 2x35 and 4x30 t/s.
In my testing I got 150 tokens/sec with a single 5090 RTX.
"The machine immediately taught me that capacity estimates are just admission tickets."
"Useful in production, poison in a kernel comparison."
Please don't publish writing like this, it's exhausting to read. You can edit that stuff out.
The best part of this is the "Five xhigh artifacts from the finished model" section at the bottom, I suggest either moving that up or at least prominently promoting it at the top of the article.
llm gets them some of that sweet_llm_padding and readers smell the slop and attack the writing :)
people, go write "boring stuff" - its more interesting!