This looks very interesting. Possible to get those rates without exotic hardware.
But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison.
We need to see the comparison with this framework and useful models, which at present seems to mean ~30 B.
NVIDIA H200 Is not a standard GPU. 8 of them in a box with a cpu and ram costs close to the same as a house.
I am 100% all about using local models instead of sending someone else all my data and paying for the privilege of doing so, this article is misleading.
I can get a 27b model to kick out 40 tok/s on 16 gb vram. This is the area ripe for development.
If you can’t connect a monitor, it isn’t a standard GPU, at least not in the way people have spoken about GPUs until a few years ago.
This blog post clearly targets VCs, but what they are doing is legit and can improve the performance of local models on low-end hardware as well, especially since their priority is to optimize non-batched inference.
For new open weights models, will you need to adapt model code and optimization for your inference engine by hand?
It's true that BS=1 is king when it comes to agentic workflows, however these kinds of system serve multiple requests concurrently with dynamic batching. Do you think it will scale as well ?
Making these claims on a 2B parameter model seems a bit like seeing linear scalability from 1 to 4 cores and then assuming 256 cores will give you a 256x speedup. Or demonstrating massive improvement on datasets that fit in cache and then assuming the same improvements will be present on problem sizes that span the memory of multiple machines. Something tells me that scaling to larger models will be more difficult than assumed.
disclaimer: I've known the founder for a while, as legitimate as it gets in deep tech, real years of research and engineering behind this, not vaporware
I have been lamenting for a while that the memory-bandwidth <-> tps relationship was pretty much working for small models on consumer cards, but not at all on datacenter hardware.
It's great to see that with proper care on the inference engine implementation the relationship can be restored.
Huh, interesting. Some parts of this do generalize even to an RTX 6000 Pro Blackwell, I imagine, though we're going to be solidly bottlenecked then on inter-card throughput through the PCIe interface.
An article with a title saying tokens per second throughput without any qualifier e.g. what size the model is should immediately be classified as spam.
38 comments
[ 5.3 ms ] story [ 52.4 ms ] threadBut I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison.
We need to see the comparison with this framework and useful models, which at present seems to mean ~30 B.
Benched at 96 input tokens, 4000 output tokens.
> 8× NVIDIA H200
For instant code generatio, 400-500 tok/s should be sufficient, though most frontier models give us closer to 70 tok/s.
I am 100% all about using local models instead of sending someone else all my data and paying for the privilege of doing so, this article is misleading.
I can get a 27b model to kick out 40 tok/s on 16 gb vram. This is the area ripe for development.
If you can’t connect a monitor, it isn’t a standard GPU, at least not in the way people have spoken about GPUs until a few years ago.
Monokernel deep dive (GPU Engineering): http://blog.kog.ai/building-a-single-kernel-latency-optimize...
Delayed Tensor Parallelism (research): http://blog.kog.ai/delayed-tensor-parallelism-for-faster-tra...
To try the speed on the playground: http://playground.kog.ai
For new open weights models, will you need to adapt model code and optimization for your inference engine by hand?
It's true that BS=1 is king when it comes to agentic workflows, however these kinds of system serve multiple requests concurrently with dynamic batching. Do you think it will scale as well ?
Any plans to release it open source?
Congratz again for the release
That means Jensen can add another 30 times faster when comparing Rubin to Blackwell without having to actually do anything.
Hopefully that means he won't have any problem to make another 150 billion in profit in the next year.
Sorry for the sarcasm. Looks like interesting work.
each time getting 3300+ tps.
The demo is very impressive!
disclaimer: I've known the founder for a while, as legitimate as it gets in deep tech, real years of research and engineering behind this, not vaporware
Feels like a preview of the future
I have been lamenting for a while that the memory-bandwidth <-> tps relationship was pretty much working for small models on consumer cards, but not at all on datacenter hardware.
It's great to see that with proper care on the inference engine implementation the relationship can be restored.