1 comment

[ 0.19 ms ] story [ 3.7 ms ] thread
I am very impressed with the KV Cache Compression work as well as the prefix cacheing making queries converge on practically free.