2 comments

[ 6.9 ms ] story [ 16.2 ms ] thread
Previously, the "Executor" runtime was shipped as binary blobs. This is the bit that schedules requests and manages KV cache (similar to vLLM or SGLang server).
(comment deleted)