Those cards should be able to do a lot more. They support int8 and they have very good processing speed. For reference I have a single V100 running around 900t/s prefill and 100 t/s decode during DeepSWE runs on 27B (4 bit quant). Those cards have half the bandwidth so a realistic decode is going to stop around 50 t/s, but those cards can support NVFP4 more easily than the V100. Combined with a TP build you should be able to get 100 t/s. And you should be able to 2x the prefill of a V100. I have had Claude optimizing llama.cpp for about a week and am at around 300 t/s decode so far on two ancient V100s @ 64 GB RAM.
Also for the 3 people that ever read this and are curious about local models still, Qwen 27B 3.8 matched Sonnet 5 in the 17 DeepSWE tasks I have run so far, solving the exact same 7 it has. Caveat: datacurve combined low/medium/high Sonnet 5 data.
Yes and no. I tested llamacpp on my intel gpu with both vulkan and sycl. I measured them to be in the same ballpark even if i have se en pepole claim marger differences than i observed.
In the end i took the sligth slowdown of vulkan to have a more stable and higher development velocity backend
While being able to use identical setups on both intel and and gpus.
This seems to be a thin wrapper around libllama, what's the point? Llama.cpp already ships a web server. I don't see anything in the README that llama.cpp doesn't already support.
Usually people advertise "Go binary" when it's pure Go, not just a CGO wrapper. The project seems to be the result of 5 minutes of running Claude Code.
> Local inference — llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback
so this is just a go binary that exec's `llama-server --model $MODEL_PATH ...`? there's room for llama wrappers, sure, but the readme only shows features that are already exposed directly from llama-server. it doesn't seem to do anything besides rename the CLI arguments: i don't get it.
20 comments
[ 0.27 ms ] story [ 39.5 ms ] threadI got excited about someone paying attention to intel. Oh well.
https://github.com/ggml-org/llama.cpp/blob/master/docs/backe...
Also for the 3 people that ever read this and are curious about local models still, Qwen 27B 3.8 matched Sonnet 5 in the 17 DeepSWE tasks I have run so far, solving the exact same 7 it has. Caveat: datacurve combined low/medium/high Sonnet 5 data.
In the end i took the sligth slowdown of vulkan to have a more stable and higher development velocity backend While being able to use identical setups on both intel and and gpus.
Usually people advertise "Go binary" when it's pure Go, not just a CGO wrapper. The project seems to be the result of 5 minutes of running Claude Code.
so this is just a go binary that exec's `llama-server --model $MODEL_PATH ...`? there's room for llama wrappers, sure, but the readme only shows features that are already exposed directly from llama-server. it doesn't seem to do anything besides rename the CLI arguments: i don't get it.