Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!
You can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size.
This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
I'm doing the same with a context of about 128-150k Surprisingly, I get subjectively better results with Unsloth's 3 bit quants (UD-Q3-XL something), than their 4 bit quants (S or M)
You can trivially run 131k on 24GB 4bit, and there are repos with tweaks that allow you to get the full 262k but idk if there's degradation with their approach.
The biggest/easiest tweak is kv quantization. It's not free, but it seems...sometimes worth it? You can quant them down to 8bit (from 16) without it _obviously_ going to hell.
I run qwen 27b on an old-ass 16gb gpu. It’s very possible using unsloth 2bit and 3bit quants, tho there are a bunch of interesting quants that let you run closer to 4bit on 16gb. This article that’s currently also on the front page mentions a bunch of them while discussing their own quant https://byteshape.com/blogs/Qwen3.8-27B/
Maybe I oversold the fun-ness of it. The most common failure mode is that it goes into loops and you come back to find it exhausted the output length without getting anywhere.
I'm hoping they release an 8B v2 based on the Qwen 3.8 series in the near future - that would give us a really powerful model that could be run directly on users phones.
Yeah just wanted to mention in case it explains the 2x lower throughout you are seeing on M5. To be fair their documentation is a bit inconsistent in some spots.
Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
They’re fairly useful as constrain domain classifiers due to the low memory requirements means you can stuff a lot of them into a less expensive GPU farm and get really decent throughout with pretty good results over all. At least that’s my experience. I wouldn’t bother using a tiny model for coding - but the world is full of abductive reasoning tasks that don’t involve coding.
If you want to download the gguf to your regular huggingface cache directory instead of to /tmp, you can download the model and run the server in one step:
export HF_TOKEN=xxx # optional, speeds up the download
./llama-prism-b10685-7dffb15/llama serve \
-hf prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0 \
--port 8331 -ngl 99 -fa on -c 32768
Thanks for all of your exploration in public Simon.
Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
I managed to run it with RTX 3070 (8GB VRAM) following the "Quickstart" on HF model card with minor modifications (modify the architecture 86 for your own hardware):
the max context size i could serve is 64K on GPU only (the -ngl 99 setting). tested with pi harness and it is very fast. the /thinking level always gets reset to off though and it is not very smart like this. haven't figure out a way to fix that.
I really wish people would stop saying N times smaller than something when making a comparison; that makes no sense - it's 1/9th (11.11%) the size. You don't get a smaller quantity by multiplying by a number greater than 1.0. You could instead reverse the subjects being compared - "the original model is 9x bigger than this new smaller, efficient model" or some such. That makes sense.
I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.
I agree it makes little sense in a literal mathematical take but "it's 9x smaller" is too much of linguistic advantage compared to "the original is 9x larger" to expect a change with. You don't have to invoke fractions, it keeps the thing in focus as the first subject, it matches the pattern of the inverse statement, and it's just plain shorter... so that's what people will do and adapt the meaning to be.
One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then smaller/larger tells you which end of that separation the subject is rather than specifying whether the multiplication builds up or down.
11% is also a really saying a fraction, 11 per-cent or 11/100, but there's nothing wrong with that feeling more natural to some and it is at least a nice shorthand way for the written form. "A ninth the size" is a similar alternative. All are really fine, there's always someone who has trouble with a given representation.
It's an idiom, much like "you are pulling my hair". Are you actually pulling my hair? Of course not; taken literally, many English phrases make no sense. But they are common elements of the language and everyone understands them, so they aren't problematic.
This is the same. Taken literally, "9x smaller" is nonsense, but everyone who hears that phrase knows exactly what mathematical operation you are referring to, thus it's a totally acceptable way to express that you mean to say 11.11% as large.
This is another pet peeve that I have with the way we phrase these things: It should be 9x "as large" and 8x "larger."
The way I'd usually phrase this, to avoid ambiguity, is "this new model is 11% the size (of the original)." Or, the other way, "the old model is 9x the size".
Eh, when I read smaller with an integer multiplier, I mentally switch to the reciprocal. Easier than convincing the world not to use "9x smaller". Do you feel the same way about "9x faster"? What you're actually measuring is time, and "faster" is the reciprocal of time, similarly to "smaller" being the reciprocal of size.
yes, actually I do think the phrase 'N times faster' is sensible and logical
if I say 'this Apple M5 chip is 3x faster' than this intel chip, it implies two things:
- the run time of most operations that runs on it is now reduced (so one quantity is smaller)
- but also: MORE WORK is being completed per unit of time compared to the intel chip (so this quantity is greater)
So yes, a greater quantity is being measured in the apple chip compared to the intel when you say apple is N times faster.
i guess this a quirk with the word 'faster' - it actually measures two things, time and work performed per unit of time. the word smaller just measures size.
like, if I give a customer a cup of coffee one day, then give them a SMALLER cup of coffee the next day, for the same price, but declare "It's now 2X more space efficient!" i.e. it's now half the size, I'm certain the customer is gonna be pissed.
Ah, but in this case the customer is buying the coffee for the caffeine content. And you've doubled the caffeine content per volume! Twice as efficient a delivery mechanism, similar to this model!
I feel like this becomes less clear if you're talking about "one-time" events. "Our website loads 2x faster now" doesn't mean that they are sending more data per second, and the user won't be doing more website loads. The website load time going from 1 second to 0.5 seconds is "2x faster" and I think that that is a very common way of communicating such performance increases too.
They probably rephrased it from some more technical form like "we compressed the model by a factor of 9" or "we've improved the packing efficiency of the model by 9x". Where these are measurements of the transformation the model is undergoing, not measurements of the resulting model.
This sounds like the same kind of error as writing "0.10 cents" because it's less than a dollar when the number is in dollars regardless of how big or small it is
10% smaller is 3*0.1=0.3 smaller than 3, which is 3-0.3=2.7
So 200% smaller is 3*2=6 smaller than 3, which is 3-6=-3
Which is why these kinds of statements usually don't make any sense at all. Usually something can't be more than 100% smaller, at which point it is completely gone.
Totally. Any MoE model can have experts swapped in and out from disk or system ram. I only framed it this way because the question was about the model fitting in vram.
The benchmarks they chose are rather cherry picked to not include long context or difficult ones that involve long horizon work or many agent turns, as I suspect this is where the model shows more differences compared to the full fat one
Yep more hops from the lower Q is likely going to skew the vectors further over time.
I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might even without this may be amazing for short lived agents that work via generations / changing task.
I just checked that model (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S) and it does much much worse on the "Please recite Jabberwocky" test than the original bf16 does.
The bf16 only misses "snicker-snack" and this quantization becomes confused after the first stanza.
I think it should be expected that a smaller model is worse at reciting memorized data than a big one. I also don't think it's a good use case of small local models. Can it find and recite Jabberwocky if given access to a web search tool?
It depends on what you mean by lossless. If both models can perform the same tasks it can be considered lossless for those tasks. That task might be more related to language understanding rather than memorizing, they have several benchmarks in the article.
> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weight
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what makes them better?
I think the general idea is naive quantization falls apart below 4bpw but you can go lower with more sophisticated QAT-adjacent methods. Bonsai's quantization method is proprietary though.
I wonder how well it performs in practice, because I can't help seriously doubting those benchmarks. That would put this 6GB model in Opus 4.6+ ballpark. Granted, that's mostly to Qwen 3.8's credit, but it's hard to believe that Qwen's already unbelievable capability density can still be compressed this much more.
(In case anyone remembers the compression post from yesterday[1], I checked and this one doesn't qualify for further compression - it's not zero-biased at all.)
Cautiously optimistic. The V1 was noticeably weak on world knowledge but here the 3.8 base model is geared more towards reasoning than world knowledge anyway so might not matter as much
GPT Astra did some benchmarking on the DGX Spark. Speed: 34.38 tokens/sec for generation.
Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.
Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.
That's a rough place to land on a spark. It seems unlikely to be memory bandwidth at this model size, but maybe just lack of tuned kernels? The chip is missing some CUDA features but with tuning you should be able to hit way more than that even without a drafter.
I was wondering if those ternary bits get expanded into full floats internally in the kernels - you’re probably right about lack of tuned kernels. I’m not sure you could just tune your way out of that easily though. Any suggestions on trying particular solutions?
I wonder how their talks with Apple went. Having this run on the TPU opposed to just the GPU, which drains a significant amount of battery life by comparison, is what I'm really interested in.
What is never totally clear with a lot of these releases is the scope of what it's good at. Models that can run with good speed on affordable consumer hardware for coding only is the dream. I am never going to use this for writing, images, or "general knowledge". Coding only
I tried their WebGPU version and it immediately started looping. Yeah "near lossless" my ass. Plus the reasoning that it looped on was clearly wrong and unlike the non quantized 27B
LLM quants seem to eerily converge to modern/not so modern graphics techniques. You wouldn't think it would apply but it's obvious in hindsight. In fact mining graphics ideas is probably a good inspiration for efficient LLM architecture.
For example, the Hadamard activation transform used here feels a lot like multiplying
Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).
I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].
173 comments
[ 5.4 ms ] story [ 111 ms ] threadThis will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
Prism's llama.cpp fork only has the kernels for CUDA, CPU and Vulkan. No SYCL at all :(
Remember to clear the downloaded weights afterward.
Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.
I really hope they stop using PowerVR in pixel 11.
https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.
For my "Please recite Jabberwocky" test the bf16 almost passes but the ternary and even fp8 versions fail badly.
https://gist.github.com/ctolsen/b2883e7cbf5e4357fa04366019e6...
They have a demo repo with a setup.sh script:
https://github.com/PrismML-Eng/Bonsai-demo
The release tag and weight file you suggest doesn’t match what they wrote.
If you have found better instructions and they work then use those instead!
Personally I prefer to download models directly rather than running some `./setup.sh` script where I need to then review what it does first.
Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
Here’s a link to the diff if you want to try and update that fork to take advantage of the prefill gains afforded by the hardware: https://github.com/ggml-org/llama.cpp/pull/27461/changes
Am I understanding correctly that no draft model exists (will never exist or just currently does not exist yet)?
There is no draft file in Huggingface's repository ( https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/tr... ) and in the file "scripts/download_models.sh" of the demo repository ( https://github.com/PrismML-Eng/Bonsai-demo/blob/main/scripts... ) I see this remark:
./llama-prism-b10685-7dffb15/llama-server \ -m Ternary-Bonsai-2-27B-PQ2_0.gguf \ --port 8331 -ngl 99 -fa on -c 65536 --jinja \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.
One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then smaller/larger tells you which end of that separation the subject is rather than specifying whether the multiplication builds up or down.
This is the same. Taken literally, "9x smaller" is nonsense, but everyone who hears that phrase knows exactly what mathematical operation you are referring to, thus it's a totally acceptable way to express that you mean to say 11.11% as large.
"9 times larger" => multiply the numerator of the fraction with 9
"9 times smaller" => multiply the denominator of the fraction with 9
Consider a bag of identical resistors with resistance R
Add 9 of them in series: the resistance is 9 times larger.
Add 9 of them in parallel: the resistance is 9 times smaller.
Its the series vs parallel dictionary wars all over again...
Check my other comment.
This is another pet peeve that I have with the way we phrase these things: It should be 9x "as large" and 8x "larger."
The way I'd usually phrase this, to avoid ambiguity, is "this new model is 11% the size (of the original)." Or, the other way, "the old model is 9x the size".
if I say 'this Apple M5 chip is 3x faster' than this intel chip, it implies two things:
- the run time of most operations that runs on it is now reduced (so one quantity is smaller)
- but also: MORE WORK is being completed per unit of time compared to the intel chip (so this quantity is greater)
So yes, a greater quantity is being measured in the apple chip compared to the intel when you say apple is N times faster. i guess this a quirk with the word 'faster' - it actually measures two things, time and work performed per unit of time. the word smaller just measures size.
like, if I give a customer a cup of coffee one day, then give them a SMALLER cup of coffee the next day, for the same price, but declare "It's now 2X more space efficient!" i.e. it's now half the size, I'm certain the customer is gonna be pissed.
If 9 is "9 times greater" than 1 then 1 must be "9 times smaller" than 9
It'd help if you read "9 times" with the operator which is what's being flipped instead of with the number
50 percent smaller means either half, or two-thirds the size, depending on Apple Marketing doing the math.
0.999 times smaller means size is nearly zero. 9 times smaller means you get negative memory from loading it.
I don't see what makes it hard to understand.
200% as large as 1 is 2.
200% larger than 1 is 3..
So 200% smaller is 3*2=6 smaller than 3, which is 3-6=-3
Which is why these kinds of statements usually don't make any sense at all. Usually something can't be more than 100% smaller, at which point it is completely gone.
a∥b=((a*b)/(a+b)
A normal sum has the associative property:
a+(b+c)=(a+b)+c which justifies dropping the parentheses
a+(b+c)=a+b+c=(a+b)+c
It is just an exercise for the reader that the parallel sum ∥ operation is associative:
a∥(b∥c) = (a∥b)∥c
multiplication is typically defined as repeatedly adding:
a x b = b+b+...+b+b (a times b)
similarily one can define parallel multiplication xx :
a xx b = b∥b∥...∥b∥b (a parallel-times b)
The reader can verify that 9 parallel-times size
9 xx size = size∥size∥size∥size∥size∥size∥size∥size∥size = size / 9
So I just think the Bonsai designers meant it was 9 parallel-times smaller, which checks out...
x∥x=(xx)/(x+x)= x/2
x∥x∥x =(x/2x)/(2/x+x)=(x/2)/(3/2)=x/3
x∥x∥x∥x = x/4
I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might even without this may be amazing for short lived agents that work via generations / changing task.
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
The bf16 only misses "snicker-snack" and this quantization becomes confused after the first stanza.
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what makes them better?
https://news.ycombinator.com/item?id=49611128
[1] https://news.ycombinator.com/item?id=49732931
So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)
Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.
Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.
That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.
Also, perf speedup?
So for this one, 27B * 1.76 / 8 = 5.94 GB
Yet, seems like there is still another year for improvements.
I like local models (but not mainly using them) for offline needs.
For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).
I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].
[1] https://arxiv.org/abs/2507.13575