Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (lmsys.org)

New parallel decoding algorithm that trades flops for latency reduction

Oh sweet, this looks like a nice boost. Pretty simple too. Surprised it hasn’t been tried before that I’m aware of at least.