Cutting LLM Batch Inference Time in Half: Dynamic Prefix Bucketing at Scale (daft.ai) 5 points by ykev 11mo ago ↗ HN
1 comment
[ 3.1 ms ] story [ 13.3 ms ] thread