> not to be confused with the somewhat baffling llama_chat_apply_template exposed in the libllama API, which hardcodes a handful of chat formats directly in C++
As someone who is tinkering with a desktop-based inference app in FLTK[0], i wish this used the actual Jinja2 template parser llama.cpp uses (or there was another C function that did that since AFAICT for "proper" parsing you need to be able to pass a bunch of data to the template so it knows if you, e.g., do tool calling). Currently i'm using this adhocky function, but i guess i'll either write a Jinja2 interpreter or copy/paste the one from llama.cpp's code (depending on how i feel at the time :-P).
But yeah, GGUF's "all-in-one" approach is very convenient. And i agree that it feels odd to have the projection models as separate files - i remember when i first download a vision-capable model, i just grabbed whatever GGUF looked appropriate, then llama.cpp told me it couldn't do model and took me a bit to realize that i had to download an extra file. Literally my thought once i did was "wasn't GGUF supposed to contain everything?" :-P
> The really neat thing about GGUF is that it's just one file. Compare this to a typical safetensors repo on huggingface, where there's a pile of necessary JSON files scattered around [...]
Funny, to me AI models have "always" been single files, as that's what has been the norm in the local image gen business. Safetensors files allow stuffing all kinds of stuff inside them too, no GGUF needed for that. Though given that the text encoders of modern models are multi-gigabyte language models themselves, nobody includes redundant copies of those in every checkpoint.
IMO the biggest thing still missing is an actual way to define the model architecture outside of being hard coded into the current build. It doesn't need to be a 1:1 performance parity with the fully supported models. Having proper, vendor validated support for day 1 is what is the difference between people thinking a model is amazing vs horrible. See recent Gemma vs Qwen releases.
Not sure what the solution is, other than writing a DSL to describe the model graphs which you then embed in the GGUF. The other fallback is to just read the PyTorch modules from the official model releases and convert that to GGML ops somehow.
I regret that the projection models ended up separate, and I too would have preferred for them to be in a single file. I'm not entirely sure why that ended up happening, but it very much runs counter to the single-file ethos I had in mind when I designed GGUF.
Hoping that someone will shepherd the cause of merging the two; I think I'm too out of the loop to do it this time around :-)
Currently not many people would finetune mmproj, so mmproj is reusable. The mmproj for Qwen 3.6 27B can be reused on all its finetunes. While the MTP model usually needs to be finetuned with the main model to get the best performance, which is being studied in Heretic.
hi, first post. 56 year old world's foremost procedural audio programmer xoxos vst,
wrote the world's first procedural lyrical engine in 1994.
just about every field has documented cancellation of my egalitarian work for the apron brethren. it would be nice if this species could explain how to use a text to image without leaving which mess of 30g worth of downloads to try.
please, just someone SAY what things i need just once simply without going "you need 5G 5G 5G 5G 5G 5G"
your species doesn't work since the Emm Kay heterodyning from orbit. since rely, natural kinda ears to west papua FOR A REASON
GGML & GGUF have been extremely important to the open-source ML/AI space. Projects like llama.cpp, whisper.cpp, and stable-diffusion.cpp tend to just work perfectly, across a whole bunch of different platforms and hardware backends.
I mean, one if the big issues I've had is that it doesn't really store the compute graph. It only stores a string of the foundational architecture, along with parameter metadata to allow you to rebuild the compute graph.
That means that every foundational model architecture requires new code in whatever is consuming the gguf to support that model.
Fun lore, GGUFs were once called GGJTs until I caught the "JT" (Justine Tunney) stealing the memory map code from a user who did 99% of the work in a draft PR (slaren) and lying about it, and misrepresenting or not understanding how memory map worked. She wanted her initials in the file format for bragging rights because it was claimed that it caused 90% memory reduction (actually it was just lazy loading into memory). Gerganov was quite angry when he found out what happened. Jart (JT) was then banned from the llama.cpp repo but managed to get back in a year or so later.
I recall reading it and you mischaracterize and conflate the core issues with accusations of hate and mean words, which is an inappropriate deflection and an attempt to control the narrative. The core issues were always plagiarism, misrepresentation of another user's work, refusal to give proper credit to the real author, and you bragging about it when you thought you could get away with it with quotes like "great artists steal." You never took accountability nor grasped the seriousness of what you attempted to do. You don't seem to understand WHY you faced blowback. You never expressed regret that the person you were victimizing was deeply depressed and pushed moreso by your betrayal. It's always "me, me, me." Your behavior and response is toxic.
Stealing code and credit (and being unrepentant) is actually what gets you into situations like that one. I read your document and you still seem to think you're a victim.
Also I must've been mistaken about you having been unbanned from llama.cpp, as I'm sure you're aware that being so brazen and unrepentant about your plagiarism and victimization of another developer would have risked your access once again. It's a shame you've not grown as an open source contributor
Thanks, I learned something more about GGUF by seeing what's not there yet. Tool calling format makes so much sense. It's going to be a milestone transitioning from LLMs to agents.
I have always used safetensors + metadata files (similar to Huggingface repo) format.
It is not a major pain point by any means, but good that GGUF has a compact format and good support.
21 comments
[ 53.0 ms ] story [ 354 ms ] threadhmmm...
As someone who is tinkering with a desktop-based inference app in FLTK[0], i wish this used the actual Jinja2 template parser llama.cpp uses (or there was another C function that did that since AFAICT for "proper" parsing you need to be able to pass a bunch of data to the template so it knows if you, e.g., do tool calling). Currently i'm using this adhocky function, but i guess i'll either write a Jinja2 interpreter or copy/paste the one from llama.cpp's code (depending on how i feel at the time :-P).
But yeah, GGUF's "all-in-one" approach is very convenient. And i agree that it feels odd to have the projection models as separate files - i remember when i first download a vision-capable model, i just grabbed whatever GGUF looked appropriate, then llama.cpp told me it couldn't do model and took me a bit to realize that i had to download an extra file. Literally my thought once i did was "wasn't GGUF supposed to contain everything?" :-P
[0] https://i.imgur.com/GiTBE1j.png
Funny, to me AI models have "always" been single files, as that's what has been the norm in the local image gen business. Safetensors files allow stuffing all kinds of stuff inside them too, no GGUF needed for that. Though given that the text encoders of modern models are multi-gigabyte language models themselves, nobody includes redundant copies of those in every checkpoint.
Good lord, they managed to invent a format that is even less readable than XML.
Not sure what the solution is, other than writing a DSL to describe the model graphs which you then embed in the GGUF. The other fallback is to just read the PyTorch modules from the official model releases and convert that to GGML ops somehow.
Hoping that someone will shepherd the cause of merging the two; I think I'm too out of the loop to do it this time around :-)
just about every field has documented cancellation of my egalitarian work for the apron brethren. it would be nice if this species could explain how to use a text to image without leaving which mess of 30g worth of downloads to try.
please, just someone SAY what things i need just once simply without going "you need 5G 5G 5G 5G 5G 5G"
your species doesn't work since the Emm Kay heterodyning from orbit. since rely, natural kinda ears to west papua FOR A REASON
That means that every foundational model architecture requires new code in whatever is consuming the gguf to support that model.
If only you could see how much code I've stolen from Rich Felker, David Gay, Sun Microsystems, etc.
Also I must've been mistaken about you having been unbanned from llama.cpp, as I'm sure you're aware that being so brazen and unrepentant about your plagiarism and victimization of another developer would have risked your access once again. It's a shame you've not grown as an open source contributor