Cutting LLM Batch Inference Time in Half: Dynamic Prefix Bucketing at Scale (daft.ai)

Part of the Daft team here! Happy to answer any questions