Long time llama.cpp user (and user of other ggml based AI programs) running on my crappy AMD card that only supports vulkan and it's usable thanks to GGML and llama.cpp, great program great project.
There's ample precedent for this. When you're building something to scratch an itch, the name is secondary. You cast about for one. Ah, you have a name. That'll do.
For example, DJ Delorie's djgpp project, built in 1989, was how I got into "real" programming on MS DOS.
What is really impressive about this project and llama.cpp as well is that they keep finding new ways to speed up the code on the same hardware. Every time I upgrade (once a month or so) there is usually a 5 to 10% speed boost.
While I think that llama.cpp is an outstanding project - a one-stop-shop to get almost any model to run on almost any hardware, you get a lot more speed from specialised inference engines with kernels made for your hardware.
But if you don't have time to evaluate other engines and you simply want a rock solid inference engine, then llama.cpp is the one of the best ones to go to.
> you get a lot more speed from specialised inference engines with kernels made for your hardware.
Of course you will. But that's not the point. The point is that Llama runs just about every model on just about every hardware. You can always improve on any metric by dropping that constraint. But there are a lot of such metrics:
- ease of use
- range of hardware
- accuracy of results
- range of models
Probably others. Llama.cpp may not be best at any one of those. But across all of them it has been consistently good. They've shown they're willing to stay the course as well. This is a foundation level project much like Linux or Apache.
I run vllm and llama.cpp; llama.cpp is pretty hackable by a local agent and is fast to start up, while vllm takes and age to build its optimized cuda graph.
The result is I run my big model (glm 5.3 flash) on llama.cpp and my small utility model (Qwen3 vl 8b) on vllm. Qwen often has better vision than glm (despite the tiny size) and can respond to tool calls from glm and importantly it doesn't use gobs of prefill+kv and is super fast.
Totally, I’ve been fiddling with local models on my machine (16GB AMD GPU + 64GB DDR4) for a few years and it’s remarkable the pace of improvement in what I can run. It has gone from complete novelty/useless to legitimately capable models running fast enough and with a big enough context to be useful running in a coding harness on a real codebase. A lot of that is down to the models themselves and the people making quants, but it feels like every time I pull llama.cpp changes and rebuild I get more performance and capability.
> Claude, design a landing page with beige background color in a gradient, 3 fonts in UI, a blinking led at top right, and enough animation and javascript that makes laptops built before 2025 stutter in scrolling.
> We like simplicity and aim to keep the codebase as small and as simple as possible
Spawning the cli to server routing everything to http it complicates it instead of keeping it simple. This is pure telemetry or in plain english Spying on users. Rip LLama.cpp.
They were already doing that by making the build process phone home to Huggingface and also download huge swathes of indecipherable JavaScript (3 of those have been backdoored a month ago and only version pinning saved downstream).
Yes there is much more I forgot to mention (you nailed some of it), but as I already said, everyone has a price. Although it remains a piece of engineering that empowered users certainly.
Anyway we managed to eliminate all unpleasantries and have a clean CLI. I leave the server to whom may like it. Hit me up if you want to test it.
If you load an MoE model that slightly exceeds your GPU memory budget, llama.cpp and ggml v1.0.0 can cache less-frequently used experts in host memory rather than failing to load.
"""
Description
The main goal of ggml is to be a simple, portable, and efficient tensor library for machine learning with minimal setup.
Plain C/C++ implementation without any dependencies
Cross-platform - x86, ARM, RISC-V, LoongArch, PowerPC, s390x, and WebAssembly
SIMD-optimized kernels for x86, ARM, and RISC-V
Broad backend support - CPU, GPU, NPU, and browser
2- to 8-bit integer quantization, plus MXFP4 and NVFP4 microscaling formats
Zero memory allocations during runtime
"""
As a mainframer, love to see the s390x support and zdnn (C library with all the instructions for our on chip accelerators) support. I like that IBM and Broadcom et al. went with something open source and local that could be tested on our own machines so that it wouldn't be a big pain to implement on mainframe so to speak. Have had some fun playing around with zdnn on a z17. I've not tried the Spyre cards, but the on chip accelerator for Telum II is surprisingly capable with smaller models.
58 comments
[ 0.28 ms ] story [ 30.1 ms ] threadWishing you unlimited power and may you never see a day of low battery. Respect and Love. Keep it up.
> GGML is a C library for machine learning (ML) - the "GG" refers to the initials of its originator (Georgi Gerganov).
https://github.com/rustformers/llm/blob/main/crates/ggml/REA...
For example, DJ Delorie's djgpp project, built in 1989, was how I got into "real" programming on MS DOS.
And we can only hope that our modern robber barons will use their ill-gotten gains to create something as beneficial as Carnegie libraries.
Imagine trying to name a file format for yourself even if it wasn't yours to begin with...
But if you don't have time to evaluate other engines and you simply want a rock solid inference engine, then llama.cpp is the one of the best ones to go to.
Of course you will. But that's not the point. The point is that Llama runs just about every model on just about every hardware. You can always improve on any metric by dropping that constraint. But there are a lot of such metrics:
- ease of use
- range of hardware
- accuracy of results
- range of models
Probably others. Llama.cpp may not be best at any one of those. But across all of them it has been consistently good. They've shown they're willing to stay the course as well. This is a foundation level project much like Linux or Apache.
The result is I run my big model (glm 5.3 flash) on llama.cpp and my small utility model (Qwen3 vl 8b) on vllm. Qwen often has better vision than glm (despite the tiny size) and can respond to tool calls from glm and importantly it doesn't use gobs of prefill+kv and is super fast.
I have an Intel Arc Pro B70. Serves me well! I use Qwen 3.8 27B
Congrats for 1.0.0.
Please put a little more effort in for FFS, you have seed money?
Let your design show through; your product can still be professional, and you don't have over index on bland.
https://ggml.ai/
> Minimal
> We like simplicity and aim to keep the codebase as small and as simple as possible
I'm glad their philosophy extends to the website as well. Genuinely refreshing.
Spawning the cli to server routing everything to http it complicates it instead of keeping it simple. This is pure telemetry or in plain english Spying on users. Rip LLama.cpp.
holy fucking shit I just checked and they really did this LOL (https://github.com/ggml-org/llama.cpp/pull/17824)
good thing i'm never touching that repo ever again, or only to steal implementations
That seems borderline perfect to me.
Besides, some fancy AI generated landing page would signal even lower effort.