LLM continuous batching host scheduler in Rust (191k tok/s on 1 vCPU) (github.com) 1 points by bmartin-systems 8d ago ↗ HN
1 comment
[ 0.35 ms ] story [ 7.2 ms ] thread