Whilst this is an excellent post from vLLM, one of the truly baffling things from either their team or AMDs team, is how much the workstation grade AMD r9700 has been ignored.
Stock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s
Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.
> I have had direct contact with members of the AMD RTG team and I was disgusted to find that AMD doesn't even provide them with hardware to work on. The developer I was working with had to buy the GPU he was writing drivers for.
Thanks! Didn't expect to see this here. Exactly what I needed to run Qwen3.8-27B-Quark-AWQ-MXFP4-native.gguf as well as other experiments on one or 2x R9700's (I hope).
Similar here peak ~250 and down to ~120 as it gets close to 128k (which is where I set DSH compaction) though it can readily do 256k.
I just got DeepSeek Harness (DSH) set up with 2x R9700 and it's rather mind blowing that these can do actual work and quickly. Up until now I've always been evaluating and searching for better hardware/model/tweaks. This is much more than I even hoped for and considered getting extra 3090/4090. Now I can stop looking/tweaking and start using it for all the different things I've yet to discover it's good for. I do plan to also try/use Hermes and Pi. DSH is annoying that every plugin install/remove requires a restart--given that "everything's a plugin".
GCN was such a promising compute architecture, AMD even pioneered stuff like async compute and compute shader heavy rendering pipelines, only to never seriously go beyond that on consumer gear.
I agree with your assessment that the story of supporting the competition, only to get burned, has repeated many times with AMD outside the public eye. It's why I don't put much stock in claims that things work great as long as specific flags are used.
Lemonade is one of the worst performing options. Run any modern Linux distro and ask your current LLM to setup llama.cpp with dflash2 for you as an unprivileged container running from a systemd user unit.
Obviously only on a system you do not trust at all.
You can get those numbers with https://codeberg.org/ggz14/radiance-vllm-mxfp4. I also can get it on a single R9700 but the 75 ~ 80 t/s is only peak acceptance of very predictable tokens like coding or json, and averages lower for prose. It's still much faster than regular llama.cpp.
I'd rather buy two used rtx3090 than a single r9700 AI pro. More VRAM (some wasted due to it being non continuous), more RAM bandwidth, more aggregate compute.
Only if AMD made a card like this with 48G+ I'd consider it.
Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.
I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc b70.
I was actually very interested in this so I checked the details. Turns out it was 200 simultaneous users running the same 1024 token prompt :D so all the experts got maximum parallelism.
How often are you going to run 200 parallel sessions with a tiny context and same prompt running at 7tok/s.
Based on how much my rtx3090 is getting on a single user (150tok/s) I'm estimating b70 to probably get less than that.
Sadly nvidia is king now.
Also, most of us already have nvidia cards and no inference software supports mixing let's say nvidia, Intel and amd cards in inference of one model.
2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4).
And that is tight already.
You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access.
So surprisingly, 2x r9700 starts be a nice deal.
> Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.
Well, luckily these are not vendor numbers. Prefill also scales almost linearly with the amount of GPUs.
I don't see how. Even on 32GB you can run Q6_K_XL quant with MTP at 200k context k=q8_0, v=q5_1. So 48GB VRAM is good enough to run Q8 at long context. Also with tings like ninfer and it's various forks I'm seeing people get very good performance out of Qwen3.8 models on all sorts of NVIDIA cards.
You also want prompt/prefix cache. Otherwise there is a lot of duplicate prefill processing if you fork conversation, have a different chat window or anything like that. It therefore also makes subagents much faster.
This is way outside my expertise so might be a dumb question, but how does the target model verify candidate tokens? Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with. But obviously that would defeat the purpose of speculative decoding so there must be some other way.
Also, what is the difference between “target model” and “target-model,” if any? I feel like half the instances of that phrase included the hyphen and half didn’t.
You're correct that it needs to run the full model to "verify" a token, but LLM inference benefits from batching - it's much faster to run twice in parallel than sequentially. So the draft model runs ~2-5 tokens ahead, and the full model then runs ~3-6 batches in parallel using those tokens, and can skip ahead by however many results match.
If you have some other source of parallel data (lots of users, many separate tasks) then speculative decoding might not provide any benefit.
> it must perform it's normal autoregressive decoding to know what is the correct token in order to have something to compare with
Correct except for the word "autoregressive". When you have to verify a sequence of tokens (which are autoregressively generated by the cheap model), you can do each token in parallel. This amortizes the cost of loading the weights from vram to the processors (the primary cost in LLM serving) across those tokens. Cost here is wall clock time, as well as power.
The autoregressive decoding that generates this batch of tokens is delegated to the cheaper model where the cost of loading the weights is lower and so not amortizing it is fine.
Verification means, how close is this sequence to the one I would have output.
> Naively, I would assume it must perform its normal auto regressive decoding to know what the “correct” token is in order to have something to compare the candidate token with.
Yes, but you can do it in parallel.
Suppose you predicted the tokens "D E F" in the sequence "A B C D E F". To "generate" the last token (F), it must know all preceding tokens (A B C D E). To "generate" the next-to-last token (E), it must know all preceding tokens (A B C D). And so on.
Assuming the prediction is correct, it can then run the "generation" for tokens D, E, and F at the same time. At the end, after all these tokens were "generated", it compares each token with the prediction; if the "generation" result was "D H F" it knows it has to discard the last two predicted tokens (and output "D H"), if the "generation" was "D E H" it knows it has to discard the last predicted token (and output "D E H"), etc.
And the most important part is that you can do it in parallel for each layer of the model. That is, you run "A B C D E F" through the first layer, then through the second layer, and so on; you only have to load the model weights from memory once for each layer. Instead of reading the full weights for all layers once for D, then once for E, then once for F, you only read them once for "D E F", and if the prediction was correct, you output three tokens by the (memory read) price of one (you still had to do the same amount of compute, but AFAIK LLMs tend to be more memory-bound than compute-bound).
I just finished overhauling our speculative decoding implementation for Mixlayer, so maybe I can help.
I think the piece of information that might make this click for you is the model outputs the probability distribution for all intermediate tokens even during prefill.
So for example, let's say you prefill the prompt "The quick brown fox" (and for the sake of simplicity, let's say each word is a single token). The model outputs a tensor that is [4, $vocabulary_size]. The first dimension is a token index into the input and the 2nd dimension assigns a probability to each token in the vocabulary. So even during prefill, we can look at the prediction logits for all of the intermediate tokens. That is, we can look at what the model would have predicted after "quick" and "brown", not just the tail token "fox".
In the single token autoregressive case, we just look at the next token prediction for "fox". But in the speculative decoding case we can use this information to compare the distribution of the draft model against the target model. In the greedy decoding case (ie, no sampling) we just make sure the highest probability token matches in draft and target. If we have sampling params like temperature and top-P, we have to apply something called Leviathan rejection sampling to the distribution. This basically allows us make sure the distribution is the same even if the exact probabilities are not and accept or reject draft tokens on that.
An intuition is to think of prefill (prompt processing) and decoding (text generation) phases.
When you send a request with a 50K tokens document, there's a huge amount of work happening: all tokens must go through the entire target model to be "understood" fully. But since the tokens are known in advance it can process them in large batches, so this is much faster than text generation.
Speculative decoding leverages this: small batches of ~5 tokens are proposed and processed by the target model in one pass like during prefill. The extra step to verify/discard them is relatively cheap.
So you can think of the target model's work as continuously pushing ahead in small batches, averaging maybe 2-3 accepted tokens instead of 1 at a time.
You need the next token to begin the next decode. So if you can get what might be the next token in half the time, you can kick off the next decode before the true decode for this token has finished.
If the draft was wrong you can kill the speculative decode, and you haven’t lost anything except for idle time
It’s true we can’t show the user the token until we have the true decode finished, but we can launch more work internally before we’re certain
In my opinion, LLMs are one of the most fascinating result coming from machine learning in recent years. Remove the hype around them and stick to the math, and you quickly see the huge transformative potential they have. It's great to see a lot of research being done in that direction, I wish it would mainly come from academia though..
Very true, I see it for example when my co-workers apply DRY without actually thinking about the complexity it might cause. It objectively makes code smaller, but often comes with complexity cost.
46 comments
[ 0.27 ms ] story [ 9.9 ms ] threadStock vLLM runs so slowly on these cards compared with vLLM forks like Radiance. Going from say 20-30t/s gen, to 150-200t/s
Most of AMD/vLLM work seems to be around their data centre cards, or the AMD AI Halo/Ryzen and ignores the R9700 AI Pro.
Really wish this would change.
They might in the future but future is in the future ofc
Edit: to be clear I think it's ridiculous they don't but from a company's stand point it doesn't make much sense
> I have had direct contact with members of the AMD RTG team and I was disgusted to find that AMD doesn't even provide them with hardware to work on. The developer I was working with had to buy the GPU he was writing drivers for.
Most companies self host OSS models but AMD doesn't even have the hardware for that. It errors out due to capacity when trying to use them.
The MXFP4 fork is excellent too. Its my daily driver right now. https://codeberg.org/ggz14/radiance-vllm-mxfp4
Also has PARO quant support there too (early stage)
Also speedups in both repos for 4x R9700s
What kind of performance are you getting with 4x R9700s--what do you do with all the VRAM (batching, concurrent requests, etc)?
Yeah the R4D Kernel rules imho.
I just got DeepSeek Harness (DSH) set up with 2x R9700 and it's rather mind blowing that these can do actual work and quickly. Up until now I've always been evaluating and searching for better hardware/model/tweaks. This is much more than I even hoped for and considered getting extra 3090/4090. Now I can stop looking/tweaking and start using it for all the different things I've yet to discover it's good for. I do plan to also try/use Hermes and Pi. DSH is annoying that every plugin install/remove requires a restart--given that "everything's a plugin".
I agree with your assessment that the story of supporting the competition, only to get burned, has repeated many times with AMD outside the public eye. It's why I don't put much stock in claims that things work great as long as specific flags are used.
Obviously only on a system you do not trust at all.
For a single r9700 you have 637 GB/s and for qwen 3.8 27b q4_k_xl the maximum tg/s is 33 before mtp
Now if you meant 4xr9700 tensor parallelism with mtp, 80 tg/s starts to make sense
Only if AMD made a card like this with 48G+ I'd consider it.
Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.
I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc b70.
I was actually very interested in this so I checked the details. Turns out it was 200 simultaneous users running the same 1024 token prompt :D so all the experts got maximum parallelism.
How often are you going to run 200 parallel sessions with a tiny context and same prompt running at 7tok/s.
Based on how much my rtx3090 is getting on a single user (150tok/s) I'm estimating b70 to probably get less than that.
Sadly nvidia is king now.
Also, most of us already have nvidia cards and no inference software supports mixing let's say nvidia, Intel and amd cards in inference of one model.
You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access.
So surprisingly, 2x r9700 starts be a nice deal.
> Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.
Well, luckily these are not vendor numbers. Prefill also scales almost linearly with the amount of GPUs.
I don't see how. Even on 32GB you can run Q6_K_XL quant with MTP at 200k context k=q8_0, v=q5_1. So 48GB VRAM is good enough to run Q8 at long context. Also with tings like ninfer and it's various forks I'm seeing people get very good performance out of Qwen3.8 models on all sorts of NVIDIA cards.
Also, what is the difference between “target model” and “target-model,” if any? I feel like half the instances of that phrase included the hyphen and half didn’t.
If you have some other source of parallel data (lots of users, many separate tasks) then speculative decoding might not provide any benefit.
Correct except for the word "autoregressive". When you have to verify a sequence of tokens (which are autoregressively generated by the cheap model), you can do each token in parallel. This amortizes the cost of loading the weights from vram to the processors (the primary cost in LLM serving) across those tokens. Cost here is wall clock time, as well as power.
The autoregressive decoding that generates this batch of tokens is delegated to the cheaper model where the cost of loading the weights is lower and so not amortizing it is fine.
Verification means, how close is this sequence to the one I would have output.
Yes, but you can do it in parallel.
Suppose you predicted the tokens "D E F" in the sequence "A B C D E F". To "generate" the last token (F), it must know all preceding tokens (A B C D E). To "generate" the next-to-last token (E), it must know all preceding tokens (A B C D). And so on.
Assuming the prediction is correct, it can then run the "generation" for tokens D, E, and F at the same time. At the end, after all these tokens were "generated", it compares each token with the prediction; if the "generation" result was "D H F" it knows it has to discard the last two predicted tokens (and output "D H"), if the "generation" was "D E H" it knows it has to discard the last predicted token (and output "D E H"), etc.
And the most important part is that you can do it in parallel for each layer of the model. That is, you run "A B C D E F" through the first layer, then through the second layer, and so on; you only have to load the model weights from memory once for each layer. Instead of reading the full weights for all layers once for D, then once for E, then once for F, you only read them once for "D E F", and if the prediction was correct, you output three tokens by the (memory read) price of one (you still had to do the same amount of compute, but AFAIK LLMs tend to be more memory-bound than compute-bound).
I think the piece of information that might make this click for you is the model outputs the probability distribution for all intermediate tokens even during prefill.
So for example, let's say you prefill the prompt "The quick brown fox" (and for the sake of simplicity, let's say each word is a single token). The model outputs a tensor that is [4, $vocabulary_size]. The first dimension is a token index into the input and the 2nd dimension assigns a probability to each token in the vocabulary. So even during prefill, we can look at the prediction logits for all of the intermediate tokens. That is, we can look at what the model would have predicted after "quick" and "brown", not just the tail token "fox".
In the single token autoregressive case, we just look at the next token prediction for "fox". But in the speculative decoding case we can use this information to compare the distribution of the draft model against the target model. In the greedy decoding case (ie, no sampling) we just make sure the highest probability token matches in draft and target. If we have sampling params like temperature and top-P, we have to apply something called Leviathan rejection sampling to the distribution. This basically allows us make sure the distribution is the same even if the exact probabilities are not and accept or reject draft tokens on that.
When you send a request with a 50K tokens document, there's a huge amount of work happening: all tokens must go through the entire target model to be "understood" fully. But since the tokens are known in advance it can process them in large batches, so this is much faster than text generation.
Speculative decoding leverages this: small batches of ~5 tokens are proposed and processed by the target model in one pass like during prefill. The extra step to verify/discard them is relatively cheap.
So you can think of the target model's work as continuously pushing ahead in small batches, averaging maybe 2-3 accepted tokens instead of 1 at a time.
If the draft was wrong you can kill the speculative decode, and you haven’t lost anything except for idle time
It’s true we can’t show the user the token until we have the true decode finished, but we can launch more work internally before we’re certain
"If you remove the hype about how transformative they are, they really are transformative".
No. If you remove the hype about how transformative they are, you're left with what they actually are: a sometimes mildly useful tinker toy.
The "mildly useful tinker toy" was smart enough to fool you into thinking you were responding to a human. I'd re-examine a few priors if I were you.