Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!
You can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size.
This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
I'm doing the same with a context of about 128-150k Surprisingly, I get subjectively better results with Unsloth's 3 bit quants (UD-Q3-XL something), than their 4 bit quants (S or M)
You can trivially run 131k on 24GB 4bit, and there are repos with tweaks that allow you to get the full 262k but idk if there's degradation with their approach.
I run qwen 27b on an old-ass 16gb gpu. It’s very possible using unsloth 2bit and 3bit quants, tho there are a bunch of interesting quants that let you run closer to 4bit on 16gb. This article that’s currently also on the front page mentions a bunch of them while discussing their own quant https://byteshape.com/blogs/Qwen3.8-27B/
I'm hoping they release an 8B v2 based on the Qwen 3.8 series in the near future - that would give us a really powerful model that could be run directly on users phones.
Yeah just wanted to mention in case it explains the 2x lower throughout you are seeing on M5. To be fair their documentation is a bit inconsistent in some spots.
Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
If you want to download the gguf to your regular huggingface cache directory instead of to /tmp, you can download the model and run the server in one step:
export HF_TOKEN=xxx # optional, speeds up the download
./llama-prism-b10685-7dffb15/llama serve \
-hf prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0 \
--port 8331 -ngl 99 -fa on -c 32768
Thanks for all of your exploration in public Simon.
Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
I really wish people would stop saying N times smaller than something when making a comparison; that makes no sense - it's 1/9th (11.11%) the size. You don't get a smaller quantity by multiplying by a number greater than 1.0. You could instead reverse the subjects being compared - "the original model is 9x bigger than this new smaller, efficient model" or some such. That makes sense.
I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.
I agree it makes little sense in a literal mathematical take but "it's 9x smaller" is too much of linguistic advantage compared to "the original is 9x larger" to expect a change with. You don't have to invoke fractions, it keeps the thing in focus as the first subject, it matches the pattern of the inverse statement, and it's just plain shorter... so that's what people will do and adapt the meaning to be.
One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then smaller/larger tells you which end of that separation the subject is rather than specifying whether the multiplication builds up or down.
11% is also a really saying a fraction, 11 per-cent or 11/100, but there's nothing wrong with that feeling more natural to some and it is at least a nice shorthand way for the written form. "A ninth the size" is a similar alternative. All are really fine, there's always someone who has trouble with a given representation.
Eh, when I read smaller with an integer multiplier, I mentally switch to the reciprocal. Easier than convincing the world not to use "9x smaller". Do you feel the same way about "9x faster"? What you're actually measuring is time, and "faster" is the reciprocal of time, similarly to "smaller" being the reciprocal of size.
yes, actually I do think the phrase 'N times faster' is sensible and logical
if I say 'this Apple M5 chip is 3x faster' than this intel chip, it implies two things:
- the run time of most operations that runs on it is now reduced (so one quantity is smaller)
- but also: MORE WORK is being completed per unit of time compared to the intel chip (so this quantity is greater)
So yes, a greater quantity is being measured in the apple chip compared to the intel when you say apple is N times faster.
i guess this a quirk with the word 'faster' - it actually measures two things, time and work performed per unit of time. the word smaller just measures size.
like, if I give a customer a cup of coffee one day, then give them a SMALLER cup of coffee the next day, for the same price, but declare "It's now 2X more space efficient!" i.e. it's now half the size, I'm certain the customer is gonna be pissed.
Ah, but in this case the customer is buying the coffee for the caffeine content. And you've doubled the caffeine content per volume! Twice as efficient a delivery mechanism, similar to this model!
They probably rephrased it from some more technical form like "we compressed the model by a factor of 9" or "we've improved the packing efficiency of the model by 9x". Where these are measurements of the transformation the model is undergoing, not measurements of the resulting model.
This sounds like the same kind of error as writing "0.10 cents" because it's less than a dollar when the number is in dollars regardless of how big or small it is
The benchmarks they chose are rather cherry picked to not include long context or difficult ones that involve long horizon work or many agent turns, as I suspect this is where the model shows more differences compared to the full fat one
Yep more hops from the lower Q is likely going to skew the vectors further over time.
I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might even without this may be amazing for short lived agents that work via generations / changing task.
> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weight
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what makes them better?
I think the general idea is naive quantization falls apart below 4bpw but you can go lower with more sophisticated QAT-adjacent methods. Bonsai's quantization method is proprietary though.
(In case anyone remembers the compression post from yesterday[1], I checked and this one doesn't qualify for further compression - it's not zero-biased at all.)
Cautiously optimistic. The V1 was noticeably weak on world knowledge but here the 3.8 base model is geared more towards reasoning than world knowledge anyway so might not matter as much
GPT Astra did some benchmarking on the DGX Spark. Speed: 34.38 tokens/sec for generation.
Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.
Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.
That's a rough place to land on a spark. It seems unlikely to be memory bandwidth at this model size, but maybe just lack of tuned kernels? The chip is missing some CUDA features but with tuning you should be able to hit way more than that even without a drafter.
I wonder how their talks with Apple went. Having this run on the TPU opposed to just the GPU, which drains a significant amount of battery life by comparison, is what I'm really interested in.
What is never totally clear with a lot of these releases is the scope of what it's good at. Models that can run with good speed on affordable consumer hardware for coding only is the dream. I am never going to use this for writing, images, or "general knowledge". Coding only
I tried their WebGPU version and it immediately started looping. Yeah "near lossless" my ass. Plus the reasoning that it looped on was clearly wrong and unlike the non quantized 27B
LLM quants seem to eerily converge to modern/not so modern graphics techniques. You wouldn't think it would apply but it's obvious in hindsight. In fact mining graphics ideas is probably a good inspiration for efficient LLM architecture.
For example, the Hadamard activation transform used here feels a lot like multiplying
Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).
I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].
90 comments
[ 0.21 ms ] story [ 17.4 ms ] threadThis will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
Prism's llama.cpp fork only has the kernels for CUDA, CPU and Vulkan. No SYCL at all :(
Remember to clear the downloaded weights afterward.
Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.
https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.
They have a demo repo with a setup.sh script:
https://github.com/PrismML-Eng/Bonsai-demo
The release tag and weight file you suggest doesn’t match what they wrote.
If you have found better instructions and they work then use those instead!
Personally I prefer to download models directly rather than running some `./setup.sh` script where I need to then review what it does first.
Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
Here’s a link to the diff if you want to try and update that fork to take advantage of the prefill gains afforded by the hardware: https://github.com/ggml-org/llama.cpp/pull/27461/changes
I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.
One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then smaller/larger tells you which end of that separation the subject is rather than specifying whether the multiplication builds up or down.
if I say 'this Apple M5 chip is 3x faster' than this intel chip, it implies two things:
- the run time of most operations that runs on it is now reduced (so one quantity is smaller)
- but also: MORE WORK is being completed per unit of time compared to the intel chip (so this quantity is greater)
So yes, a greater quantity is being measured in the apple chip compared to the intel when you say apple is N times faster. i guess this a quirk with the word 'faster' - it actually measures two things, time and work performed per unit of time. the word smaller just measures size.
like, if I give a customer a cup of coffee one day, then give them a SMALLER cup of coffee the next day, for the same price, but declare "It's now 2X more space efficient!" i.e. it's now half the size, I'm certain the customer is gonna be pissed.
If 9 is "9 times greater" than 1 then 1 must be "9 times smaller" than 9
It'd help if you read "9 times" with the operator which is what's being flipped instead of with the number
I don't see what makes it hard to understand.
I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might even without this may be amazing for short lived agents that work via generations / changing task.
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what makes them better?
https://news.ycombinator.com/item?id=49611128
[1] https://news.ycombinator.com/item?id=49732931
So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)
Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.
Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.
That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.
Also, perf speedup?
So for this one, 27B * 1.76 / 8 = 5.94 GB
Yet, seems like there is still another year for improvements.
I like local models (but not mainly using them) for offline needs.
For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).
I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].
[1] https://arxiv.org/abs/2507.13575