I keep ending up with the same problem when running llama.cpp: multiple models, multiple instances, different GPU allocations, ports, flags, and a growing pile of scripts to keep it all manageable.
LlamaRack is my attempt at making that less annoying.
It manages llama.cpp instances, models, GPU assignment and runtime settings, with an OpenAI-compatible API on top.
It works well enough on my own setup now, but I’d really like to find a few other llama.cpp users willing to try it on different hardware and workflows.
Not looking for promotion or reviews - mostly bugs, rough edges, “why does it work like this?” and things I’ve completely overlooked.
Worth trying on a CPU-only host as well as GPU boxes. A lot of LlamaRack users will hit memory-bound cases where a Seattle Xeon with decent RAM is enough for GGUF batch runs. If you want a short metered window without buying hardware, CPU hours around $19/hr: https://ai.westnet.ca/
3 comments
[ 0.20 ms ] story [ 8.0 ms ] threadLlamaRack is my attempt at making that less annoying.
It manages llama.cpp instances, models, GPU assignment and runtime settings, with an OpenAI-compatible API on top.
It works well enough on my own setup now, but I’d really like to find a few other llama.cpp users willing to try it on different hardware and workflows.
Not looking for promotion or reviews - mostly bugs, rough edges, “why does it work like this?” and things I’ve completely overlooked.
https://github.com/brantje/LlamaRack