[–] mmoskal 1y ago ↗ Previously, the "Executor" runtime was shipped as binary blobs. This is the bit that schedules requests and manages KV cache (similar to vLLM or SGLang server).
2 comments
[ 199 ms ] story [ 948 ms ] thread