Ask HN: Are there any ready-to-use Docker images for running LLMs locally?
I've been having a lot of trouble spinning up the various stacks for running open LLMs like Alpaca or Vicuna because they often require specific CUDA versions, specific gcc toolchains, etc...
Has anyone got a dockerfile or published container image that "just works" and can run 4-bit quantized models on CPUs and/or GPUs? Ideally something that will run StableLM.
I've tried to build such a thing myself, but I found that the vague instructions in blogs aren't sufficient for reproducible build. Too many instances of "clone this (every changing) Git repo" or "just curl & execute this", leading to very rapid bit-rot where even instructions from a month ago can't be reproduced!
6 comments
[ 5.9 ms ] story [ 28.8 ms ] thread[0] https://github.com/nsarrazin/serge
- 4 bit cuda
- 4 bit triton
- 4 bit cpu
https://github.com/oobabooga/text-generation-webui/blob/main...
Models should be quantized specifically for each, and both branches are under heavy (daily) development... you really want to git pull them all the time.
What OS are you running? A linux distro I presume?
While I have 64 GB of memory and an okay NVIDIA GPU, what I would like is a Docker image I can experiment with locally, but run on a cloud-hosted "spot" instance if I want to do something heavyweight, without having to run through the install process from scratch each time.
Similarly, with the quantisation process, for some reason few such files are being published. It apparently takes lots of CPU and RAM to produce the 4-bit versions. With a container image, I could do this in the cloud, then download the result to run on a more constrained device, etc...
I havent explored beyond that.
Actually there are tons of quantizatons on hugging face, just search for "4bit". 128g usually (but not always) means a Triton variant, ggml means a cpu variant. I know that vicuna 7/13B and the "uncensored" vicuna 7B are quantized on sf, but there are already some newer, supposedly better instruction following mixes.
It's a really shocking experience coming form the Rust and C# ecosystems where everything is strongly versioned, packages have "lock" files, and there's even SHA hashes to really pin things down. Meanwhile in this world it's just: "Get the latest version of this thing. It won't work, of course, because it only worked for a week in 2023 when I did it, but good luck!"
Bingo! Welcome to ML land, you are truly starting to understand.
Honestly the oobabooga repo is 100x better than average. Most ML repos are one off research papers or demonstrations that barely (or don't) work on precise python environments and then are completely abandoned. Others live in dependency hell, with CUDA hell in particular being very common.