Reverse-engineering Nvidia's CUDA-checkpoint for faster cold starts (blog.doubleword.ai) 2 points by ilreb 5d ago ↗ HN
[–] zoobab 5d ago ↗ Would it be able to swap models on demand with this? [–] fergus00 5d ago ↗ https://blog.doubleword.ai/fast-sglang-starts yeah, this is part of the idea. If you get cold starts down to seconds or subseconds, then you can run many models multiplexed on the same GPUs
[–] fergus00 5d ago ↗ https://blog.doubleword.ai/fast-sglang-starts yeah, this is part of the idea. If you get cold starts down to seconds or subseconds, then you can run many models multiplexed on the same GPUs
2 comments
[ 2.5 ms ] story [ 18.0 ms ] thread