hi!
open sourced a serving config for Kimi K2.6 from 90 tok/s 508 tok/s on 8xMI300X. Same weights / 0 quality loss.
Scaling is linear @15.8 tok/s per slot latency is constant. REpo has command launcher, Dockerfile, benchmark tool. Known limitations: BF16 KV only (FP8 crashes due to an AITER 384-expert constraint)
2 comments
[ 3.0 ms ] story [ 10.4 ms ] threadScaling is linear @15.8 tok/s per slot latency is constant. REpo has command launcher, Dockerfile, benchmark tool. Known limitations: BF16 KV only (FP8 crashes due to an AITER 384-expert constraint)