True, and for third-party models we'll just re-use their public open weights. There is a time-consuming part, though, that is performed manually by our (human) team: implement the logic of the model in C++ and assembly…
It also matters for thinking models and for agentic workflows, especially in software engineering, where a lot of tokens need to be output in iterative loops before the user sees any result. This is our main use case.
In theory yes, although not in a linearly proportional way, because in practice our memory streaming is not yet perfect. There are still some fixed costs that we did not fully optimize (for now).
I'm sure there are, and I really hope we can work on consumer-grade GPUs at some point. It should be possible to apply the same methodology (digging deep into the hardware details to understand all its little…
Note that this coding model is trained on programming use cases, and is also not tuned for multi-turn chat. You can ask it to implement an algorithm; we provide suggested prompts you can test. Also, this tech preview is…
Why not, it's one way to look at it! Although I have yet to see other work with speculative decoding higher than ~1,000 tokens/s., because the other bottlenecks start to matter at that point, and they need to be solved…
it's also a coding model
thank you deflator, I understand this now! much appreciated
Token generation speed matters for sequential agentic workflows, like software engineering / vibe coding, where a lot of reasoning tokens, code generation, refactoring, testing, etc. happen in a loop before an actual…
will do - we are a small team and it takes time to implement and optimize a new model, whatever the size.
Totally, though DTP is not required for these kind of speeds. Standard TP works also. DTP is something we built for our roadmap in order to get to extremely high speeds (like 10k+ tokens/s). When the budget is under 10…
Thanks for the comment and the question! The last section of the article lays out the scaling laws that apply when porting this approach to another model. In a nutshell, DeepSeek V4 Pro with 49B active params is close…
Thanks a lot! Much appreciated. To answer your questions: - yes, we rewrite the whole model code (while keeping the same logic) in CUDA/HIP and assembly, in order to optimize by hand for each GPU type. It's quite…
Yeah, I agree: I'm actually not expecting it to be easy, and there will certainly be several unknown unknowns we'll discover along the way. Our process has been, and will continue to be, a sequence of (tedious) R&D…
Great points, let me clarify: - model size: 2B is just for this preview (it was faster to implement), our article explains how we expect to support large frontier MoE at 1,000 to 5,000 tokens/s - reaching 500 tok/s, or…
I updated the article title accordingly
YES - I just updated the title of our article according to your suggestion.
Follow-up reading the most technical and research people here: Monokernel deep dive (GPU Engineering): http://blog.kog.ai/building-a-single-kernel-latency-optimize... Delayed Tensor Parallelism (research):…
I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed. Sorry for the confusion
I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed.
thanks! we explain how it scales to larger models in the last section the OP blog post
Thanks. To be fair, this number is what we expect to get once we port DeepSeek V4 in our engine on the upcoming generation of GPUs!
Great points. We strived to be fair as possible in the benchmark, but it's indeed not perfect. Taalas should have been added in the dedicated hardware section, even though they use 3-bit quantization when we are on FP16…
Fair point - this tech preview is about the speed (hence the small dense model, it was easier to implement). The math checks out though to allow support for large frontier MoE models at similar speeds. At batch size 1,…
True, and for third-party models we'll just re-use their public open weights. There is a time-consuming part, though, that is performed manually by our (human) team: implement the logic of the model in C++ and assembly…
It also matters for thinking models and for agentic workflows, especially in software engineering, where a lot of tokens need to be output in iterative loops before the user sees any result. This is our main use case.
In theory yes, although not in a linearly proportional way, because in practice our memory streaming is not yet perfect. There are still some fixed costs that we did not fully optimize (for now).
I'm sure there are, and I really hope we can work on consumer-grade GPUs at some point. It should be possible to apply the same methodology (digging deep into the hardware details to understand all its little…
Note that this coding model is trained on programming use cases, and is also not tuned for multi-turn chat. You can ask it to implement an algorithm; we provide suggested prompts you can test. Also, this tech preview is…
Why not, it's one way to look at it! Although I have yet to see other work with speculative decoding higher than ~1,000 tokens/s., because the other bottlenecks start to matter at that point, and they need to be solved…
it's also a coding model
thank you deflator, I understand this now! much appreciated
Token generation speed matters for sequential agentic workflows, like software engineering / vibe coding, where a lot of reasoning tokens, code generation, refactoring, testing, etc. happen in a loop before an actual…
will do - we are a small team and it takes time to implement and optimize a new model, whatever the size.
Totally, though DTP is not required for these kind of speeds. Standard TP works also. DTP is something we built for our roadmap in order to get to extremely high speeds (like 10k+ tokens/s). When the budget is under 10…
Thanks for the comment and the question! The last section of the article lays out the scaling laws that apply when porting this approach to another model. In a nutshell, DeepSeek V4 Pro with 49B active params is close…
Thanks a lot! Much appreciated. To answer your questions: - yes, we rewrite the whole model code (while keeping the same logic) in CUDA/HIP and assembly, in order to optimize by hand for each GPU type. It's quite…
Yeah, I agree: I'm actually not expecting it to be easy, and there will certainly be several unknown unknowns we'll discover along the way. Our process has been, and will continue to be, a sequence of (tedious) R&D…
Great points, let me clarify: - model size: 2B is just for this preview (it was faster to implement), our article explains how we expect to support large frontier MoE at 1,000 to 5,000 tokens/s - reaching 500 tok/s, or…
I updated the article title accordingly
YES - I just updated the title of our article according to your suggestion.
Follow-up reading the most technical and research people here: Monokernel deep dive (GPU Engineering): http://blog.kog.ai/building-a-single-kernel-latency-optimize... Delayed Tensor Parallelism (research):…
I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed. Sorry for the confusion
I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed.
thanks! we explain how it scales to larger models in the last section the OP blog post
Thanks. To be fair, this number is what we expect to get once we port DeepSeek V4 in our engine on the upcoming generation of GPUs!
Great points. We strived to be fair as possible in the benchmark, but it's indeed not perfect. Taalas should have been added in the dedicated hardware section, even though they use 3-bit quantization when we are on FP16…
Fair point - this tech preview is about the speed (hence the small dense model, it was easier to implement). The math checks out though to allow support for large frontier MoE models at similar speeds. At batch size 1,…