> For decode, (...) it is difficult to batch multiple output tokens together. I think it's the other way around? The GPU has to stream gigabytes of active layer weights to compute the next token, so having a batch of…
[dead]
> For decode, (...) it is difficult to batch multiple output tokens together. I think it's the other way around? The GPU has to stream gigabytes of active layer weights to compute the next token, so having a batch of…
[dead]
[dead]