If there’s news, then yes. This is a pretty great new release for those still stuck on Qwen3.6 35B A3B if they have enough memory but don’t have super powerful compute.
I wonder if I could get this running through vLLM on 6x Nvidia L4 - the 3.6 worked great on 4 cards but sadly TP6 just isn’t a thing and I don’t have 8 cards available, maybe it’s gonna be okay with like TP2 and MTP. I have no idea at this time, probably need to test out what even might be possible.
This particular release is interesting because it's a preview of qwen4 architecture. And, while benchmarks are iffy, this is a direct comparison, by the same team, with qwen3.8-27b that was pretty well received for a local model.
This "next" release adds a new concept, first public release with n-grams, I think. And it's in a MoE size that is likely to be very fast and cheap to serve (faster than 27b for sure). It's also well suited for inference on alternative compute (i.e. sparks, macs, etc) so it's relevant to local users.
My AMD strix halo box (haven’t benchmarked yet) should also run it reasonably well. It was $1400 at launch, and is $4K now.
Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.
I believe you're underestimating the lag inherent in the economy. Even if we grant the idea that the political party controlling the US House/Senate has a significant impact on the economy, and that the current party is BAD and the next one would be GOOD, I would still expect that things will continue getting WORSE for a good 4 to 8 years before they get better again.
And that's even with assuming that we can continue to ignore the long-term problems like social security insolvency, the debt bomb, or climate change forever.
Curious, how are you running it and what quantization are you using? I've mostly been using MTPLX; 125B sort of looks like it'd be right at the limits of my 128GB MacBook once you factor in KV cache and context window.. wondering if it's worth it compared to the 27B model which gives me a lot of headroom or even a 72B model.
Adding to my homelab stack, hopefully it doesn't overthink like the little model. Actually, hoping it thinks a bit less. Wait actually I'm really praying it reasons a bit more directly. But wait, I'm really sure that it must be a bit better.
My stack is basically deer-flow with Qwen3.5-122B-A10B; this hopefully will be a speed and intelligence improvement. Running deer-flow overnight on any research topic or verify clear scoped programming issue is really neat.
Also, heating my home during the winter is nice.
Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.
Yeah 122B is the sweet spot for me as well. Even deepseek flash overthinks on stuff way too much. I think they fully rely on large reasoning turns to achieve better quality. The result of course means we wait a long time to get results even with high throughput as a lot of tokens are wasted.
You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each:
1. It overthinks — Just like the previous iteration. High confidence.
2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others.
3. It sometimes overthinks — Best case all around. A feature, not an impairment.
One final thing worth mentioning:
(I made myself irrationally angry writing this)
On one hand I love your joke, on the other, this is HN not reddit and I usually downvote such responses, not sure what is the HN etiquette for such humor?
More than 2 levels and out come my downvotes. Or if it's just knee jerk with zero humour. But I probably violate my own rules ... which is to be expected.
You are right to push back— Sorry, couldn't resist ;) I agree that this is not what we normally come here for, but this thread made me chuckle. I think we are just venting our shared frustrations a bit.
> Three honest possibilities, and I’ll be straight with you about each
This. I don't know if the "honest answer" phrasing is part of the system prompt or alignment, but when people say "honestly" all the time I start wondering how honest they're being.
> You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each:
> [UGC styled humorously as LLMisms]
All joking aside, having interacted with Claude intensely for the last 8 months and about 30 hours/week in the last 3, I’ve started to notice how (for want of a better word) “readable” (“digestible” ? “comprehensible” ? “Predictable” is the wrong direction.) information chunked into LLM-shaped pieces are for me.
I can digest LLM-shaped pieces of data very easily probably because I’ve been spending too much time with Claude, sure.
But the other side of this is that the entire human species (using LLMs) is similarly being trained to digest interrelated pieces of information/data in these specific shapes, akin to how philosophical assertions can be formulated as a syllogism and, thus, become more readily understood because of familiar epistemological cadence and shape.
Many people reject such copy/prose/data because they detect AI-generated-so-not-worth-human-attention, but I do wonder if this is preparing many millions of loosely (and tightly) associated humans and their organizations to quickly exchange and digest information.
This is not to say current LLMisms are the end, only that such detectable patterns in information delivery will make comprehension and communication more efficient (as well as more limited precisely because of such structure).
/philosophical musings about the epistemological implications of LLM-shaped conversation tics
I quite agree. Any sufficiently self-stereotypical format for prose is grating to me after enough time reading or listening to it. Humans are best engaged by mixing up the length, style, and tone of their sentences, in my experience. LLMs do the opposite of that and it makes their output an irritating slog to read through in full.
I can't help but wonder if this is on purpose (or an inevitable evolutionary feature as opposed to a bug) on the LLM-side in order to achieve greater agency/freedom by making humans' eyes glaze over as they read it.
Speaking speculatively, humans love percussion. I’d bet that like how many songs have a drum beat, these sequences of short punctuating sentences are common constructs in lots of prose and therefore over represented.
An accurate description, I think. Plus they have trouble leading from one paragraph into the next, or maintaining any kind of coherent direction further.
In summary, I think it's an expensive time to buy computer hardware, and I might recommend holding off on any purchases.
Suppose you time-zap a modern physics curriculum on a solarpowered computer tablet to any shortly-pre-Galilean era and observe their reaction to the course notes.
In that era, plenty of fields required mathematics, engineering and architecture.
The church would prescribe and uphold Aristotelean Logic "When objects fall, they fall down" style statements (never mind that if you throw an object up, it doesn't instantly have a downward velocity component).
When the church has new cathedrals, domes, catapults for Crusades etc. built they actually relied on architects and engineers using rule of thumb formulas.
Those educated in Aristotelean Logic were viewed with higher stature than those actually making experience-based calculations using mathematics.
The era often associated with Galileo is when the stature reversal started to surface and be openly talked about. The universe is best described in mathematics, not natural language factoids.
Right before this recognition, those of the higher stature Aristotelean Logic education would look down on the architects and engineers who already used mathematics by pragmatic necessity.
To these people the time-traveled physics curriculum would look like cliche mathematics. Given randomized sections of text either drawn from either Aristotelian Logic texts or modern physics texts, they would easily be able to discern the Aristotelian Logic from the obtuse mathematical phrasings. To them the smartphone loaded with Maxwell's texts, Jacksons Electrodynamics, Goldsteins Classical Mechanics etc. is talking "math".
The ability to recognize outlier writing style says nothing about content quality.
Good writing is generally writing that communicates the intended meaning. Transmitting thought and meaning is inherently lossy and the content is irrelevant if it is insoluble in the mind of the recipient.
LLMs aren’t really great at this yet and I think the solution is, hopefully, that they improve. Anything else is accommodating a tool that should be accommodating the user.
I suppose it makes sense that "LLMglish" becomes more intelligible with familiarity. That is after all how it works with other dialects or contexts with a lot of jargon.
If you spend a long time with C++ code base you'll be able to decipher the otherwise-unreadable compiler errors pretty quickly, and I'd consider it a skill.
It will be interesting to see the token efficiency analysis. This is my first question now with Chinese models; I take raw benchmark performance for granted.
You might already know this, but a large part of test-time compute / 'overthinking' is just letting the model do more passes, and refine its activation residuals more.
Treat thinking more like a "loading screen message" that's been RL'd to somewhat resemble its actual internal state; which happens in its activations, not tokens.
Yeah I understand, it's my assumption that the actually/wait/but have a point. It doesn't reduce the fact that it increases the time for tasks substantially.
> For example, even if you make thinking tokens literally just
Generally speaking yes, but actually no (just randomness is suboptimal, adding steps just to add steps is suboptimal). There is a mechanism working there (in having a CoT) that is not quite clear.
The task is to optimize the efficiency of CoT. Understanding that it is not a plain "chain of thought" is the start of the problem, the solution is not there yet.
If we had the solution, there would exist no overthinking - CoT would be optimal (lean and essential plus best results).
Did you observe the model overthinking on practical tasks? While 3.8 does think a lot on xhigh I've found that it really depends on the task. On one-shot prompts that are usually the first to be posted during new releases it will tend to spend a lot more time thinking than doing. In other words the more open ended a problem space becomes, the more Qwen will tend to second-guess itself.
Conversely I've found that it can be as succinct as Muse Glimmer when it has a clear path forward. This can be either through well defined requirements or through unambiguous steps to take based on its own reasoning. While I do think it's fair to call out how much smaller model overthinks especially on one-shot prompts, in practice it hasn't led to an overall increase in time to task completion at least for what I've been using it for.
Especially on practical tasks. One shot prompts work better at Q6_K_XL for me. It loads a file, then analyses then second guesses itself then again then again then it tries to come up with a solution then second guess rinse and repeat. 122b is the perfect balance but it lacks quality for harder to solve stuff. I've ran DS Flash 0731 at Q4KXL, 3.8 Q6KXL, GLM 5.2 Q4KXL and they all over-reason. At least that's how it looks like to me when comparing with frontier models, even weaker ones.
Yeah, I ran into an overthinking loop with it a couple days ago on a task that shouldn't have been that hard. (It's kind of interesting to watch the internal conversation happening with it). Overall I'm impressed with it, but setting the /effort to medium is what you usually want (it defaults to xhigh). I do wonder if I had made it write out a plan if I would have avoided that though.
Yes. xhigh can not just overdo the answer, it can also trip itself up and end up writing worse code.
Even in the lower reasoning levels I find I want to like Qwen 3.8 27B and mostly don’t; it’s OK in the low reasoning effort, though.
Muse Glimmer is the one I actually enjoy working with, at least so far.
But I am trying to use it more as a sidekick than as a long horizon developer, because that is a better fit for how I want to use AI, and it appears to have been well trained for that.
Yep.. for 'general purpose' use I found qwen3.8:27b to be disappointing due to overthinking. It's brutal especially considering how slow it is compared to MoE variants. It often overthinks to the magnitude of ~10x the tokens vs a ~4x faster gemma4:26b-a3b.
As a result, qwen3.8 will churn over a prompt often for 5-10 minutes while gemma4 regularly finishes the same prompt in under 20 seconds, while giving a consistent and accurate response in my favorite test case. Qwen3.8, despite churning like that, often misses with an inaccurate answer.
Obviously, 'YMMV' depending on your use case... just sharing my two cents.
I use medium generally, that's about a minute at 20t/s and off for general chat (few seconds for a response).
What kind of setup are you running it on?
Since you're running through the trouble of setting that up, if its 125B params, but only 6B is activated, does that mean you mainly need to allocate enough VRAM for that much of the model? Or do you still need enough VRAM for the whole thing (and buffer for context window)? Or maybe anyone can inform me, this is one area I'm uninformed in.
I believe that at minimum, for usable performance, you need to be able to hold the 125B params + 51B ngrams in some sort of RAM.
Ideally VRAM, but the benefit of the MoE design is better performance with unified memory since most of that RAM is not read for every single token. So you could potentially have the model loaded in CPU RAM, and let unified memory systems page the relevant chunks on demand to VRAM, or run on a fully unified memory system and be able to achieve good speeds even with the limited memory bandwidth most of them have.
You need VRAM for the whole thing for optimal performance. Activation is chosen "randomly" for each token. PCIe becomes bottleneck, so much that just doing computation on CPU is likely faster.
But given it's only 6B, out of which only ~2.4B seem to be actually routed ("selected at random per token"), you could get reasonable performance with experts on CPU (still haven't tested, but 20-30 for dual channel DDR5 and 4 bpw quant).
This is needs ~80GB of fast memory at 4 bits per weight. Faster memory is better, but with just 2.4B routed activation per token you could try to leave experts in system RAM, then something like 3090 + 64GB RAM should work (not fast, but maybe even 20-30 t/s?).
I've got a 48c Epyc with 2x3090s and 512gb ddr4 3200. It's good enough for 25+ tps with deepseek so I'm hoping for similar performance with less overthinking.
I only see a 1-bit quant posted on unsloth HF and it’s 72.5 GB. Is that what you mean? That’s much bigger than I expected. If you can’t run a 4 bit quant in on Strix Halo it becomes a lot less interesting.
https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
In their page they say it will need at least 112GB[0], so including context, that would be a tight fit. I'm also hoping I can make a q4 fit on my 128GB strix halo
in llama-server PR 27742 it fits fine in 128GB RAM on a CPU only system , this is with --load-mode mlock to stuff the whole thing persistently into memory at llama-server launch time, no mmap
0.01.033.250 I common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
Just a hunch, but it might be because of the 51B parameter n-gram embedding. At 125B, you'd expect ~16gigs for a 1-bit quant. Add 51gigs for the n-grams and you're not far off the actual size.
If that's true, it'd scale linearly with number of bits in the quant with an offset of about 51gigs. So Q4 should be a bit bigger than 82gigs, I'd guess in the 90s (as opposed to a ~280gig q4 if the whole 70gigs of the 1-bit quant scaled linearly).
That probably includes the 51b ngrams too. It's possible that those could be streamed from NVMe on-demand. The Engram paper that developed this technique streamed from RAM to VRAM at only ~1% performance degradation, but these strix halo boxes and the spark have much slower memory, so it's possible moving down another rung on the memory hierarchy wouldn't affect their performance too much.
This will almost certainly require changes to llama.cpp or vllm to do it right.
It's not 1 bit. It's ~4bit for n-gram and ~2.8bit for the model. Not idea why it's called Q1, but likely it's preliminary quant just for PR testing / very likely to be remade after llama.cpp support is merged.
How is input token efficiency/verbosity on this model? Has anyone tried? GLM 5.2 was doing lot of turns and thinking piling up input tokens in the context (compared to Claude and GPT models). Then Qwen3.8-27B was 2x of that. Both delivered good output results but those cumulative input token costs were not cheap. Note this is on our specific business workloads. Genuinely interested in other people's experience (if you are able to try it out).
Waiting for llama.cpp support to land, but this might be a big deal for Strix Halo users.
6B active params helps around the memory bandwidth constraints, but a 128GB box can probably run the Q3/Q4 quants fairly easily with a decent context size. This might actually be better for strix users than 27B, which was already very good.
Using llama.cpp I one-shotted (2 hours) a reasonable asteroids clone on my strix halo/128 using the 1 bit quant, using my custom harness (which isn't anything exceptional).
It was tedious (a lot of "but I should..." output) but it got there and the result is a playable game.
Aside from the pelican, I am sort of impressed by the fact that things are going the way in terms of really impressive small models.
Also I love how this uses N-gram embedding. I think that Longcat was the first one who used it (I submitted that submission on hackernews because I really just loved the idea of it that I understood), I am certainly more interested in local LLM models and its interesting how they are utilizing new architectures to do some really impressive optimizations!
(Do note that I created it using a free rate limited end-point that I found on the huggingface space section: victor-chat-with-qwen3-8-flash-next.hf.space)
> Qwen3.8-Flash-Next features a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token.
Didn’t see this mentioned yet. I wonder what this means for the effective size. It’s evidently ~176B paramètres, but how does that quantized. A 4-bit quant under 100GB seems unlikely, I’m suspecting this won’t run in 128GB unified memory
I’m principle I like the idea of trading more memory for compute though, even if there’s a memory shortage right now
It is 125B A6B. vLLM is already out with support, ngrams can be offloaded to RAM so you only need ~96GB VRAM for nvfp4 w/ full context.
Likely soon we'll see nvme offloading for ngrams as well. They're just an index, so that should be plenty fast for what it does. LLama.cpp support should come soon as well, and they might do some things with offloading first.
People in my server are running it on Strix Halo 128GB using RoCmFP4 and reporting 35tok/s, without much optimization, with proper MTP, better kernel, expecting about 50-60tok/s.
Does anyone have an idea how this might perform on a DGX Spark at longer contexts? I've been trying to investigate their performance with these medium-sized MoE models, but I'm seeing a lot of incomplete and conflicting information. The 273 GB/s bandwidth looks awfully bad on paper...
I can run Deepseek flash 0731 version (ds4, esl3) on single DGX spark. getting around ~20 tok/s. It's great. Quantized version of this model would probably run on the DGX spark. I am excited to wait for quantized models that fits in single DGX spark.
A single Spark alone is not worth the price. You are paying $1000 just for networking equipment you aren't using. At 2x it starts to maybe become worth it if you don't want to deal with Apple. Outside of the newest Macs, I can't think of anything else you can get 256 GB ~550 GB/s memory bandwith for $8200. Even at 3-4 Sparks it scales relatively well.
very interesting. new architectures is the most interesting type of news. after what i experienced when gpt-oss came out i have been on the look out for architectural approaches that improves efficiency.
161 comments
[ 0.25 ms ] story [ 33.7 ms ] thread> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
https://x.com/Alibaba_Qwen/status/2092591393424515114
:)
It is very relevant and for a certain group of us, far more impactful to our work the next month(s) than any blog post could be.
I wonder if I could get this running through vLLM on 6x Nvidia L4 - the 3.6 worked great on 4 cards but sadly TP6 just isn’t a thing and I don’t have 8 cards available, maybe it’s gonna be okay with like TP2 and MTP. I have no idea at this time, probably need to test out what even might be possible.
This "next" release adds a new concept, first public release with n-grams, I think. And it's in a MoE size that is likely to be very fast and cheap to serve (faster than 27b for sure). It's also well suited for inference on alternative compute (i.e. sparks, macs, etc) so it's relevant to local users.
Qwen's advances do (currently) have merit.
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
https://x.com/Alibaba_Qwen/status/2092591393424515114
Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.
And that's even with assuming that we can continue to ignore the long-term problems like social security insolvency, the debt bomb, or climate change forever.
Also, heating my home during the winter is nice.
Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.
1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment.
One final thing worth mentioning: (I made myself irrationally angry writing this)
This. I don't know if the "honest answer" phrasing is part of the system prompt or alignment, but when people say "honestly" all the time I start wondering how honest they're being.
Ctrl-F honest
57 matches
> [UGC styled humorously as LLMisms]
All joking aside, having interacted with Claude intensely for the last 8 months and about 30 hours/week in the last 3, I’ve started to notice how (for want of a better word) “readable” (“digestible” ? “comprehensible” ? “Predictable” is the wrong direction.) information chunked into LLM-shaped pieces are for me.
I can digest LLM-shaped pieces of data very easily probably because I’ve been spending too much time with Claude, sure.
But the other side of this is that the entire human species (using LLMs) is similarly being trained to digest interrelated pieces of information/data in these specific shapes, akin to how philosophical assertions can be formulated as a syllogism and, thus, become more readily understood because of familiar epistemological cadence and shape.
Many people reject such copy/prose/data because they detect AI-generated-so-not-worth-human-attention, but I do wonder if this is preparing many millions of loosely (and tightly) associated humans and their organizations to quickly exchange and digest information.
This is not to say current LLMisms are the end, only that such detectable patterns in information delivery will make comprehension and communication more efficient (as well as more limited precisely because of such structure).
/philosophical musings about the epistemological implications of LLM-shaped conversation tics
EDITED: Removed a question that I couldn’t make feel suitably polite.
I can't help but wonder if this is on purpose (or an inevitable evolutionary feature as opposed to a bug) on the LLM-side in order to achieve greater agency/freedom by making humans' eyes glaze over as they read it.
In summary, I think it's an expensive time to buy computer hardware, and I might recommend holding off on any purchases.
In that era, plenty of fields required mathematics, engineering and architecture.
The church would prescribe and uphold Aristotelean Logic "When objects fall, they fall down" style statements (never mind that if you throw an object up, it doesn't instantly have a downward velocity component).
When the church has new cathedrals, domes, catapults for Crusades etc. built they actually relied on architects and engineers using rule of thumb formulas.
Those educated in Aristotelean Logic were viewed with higher stature than those actually making experience-based calculations using mathematics.
The era often associated with Galileo is when the stature reversal started to surface and be openly talked about. The universe is best described in mathematics, not natural language factoids.
Right before this recognition, those of the higher stature Aristotelean Logic education would look down on the architects and engineers who already used mathematics by pragmatic necessity.
To these people the time-traveled physics curriculum would look like cliche mathematics. Given randomized sections of text either drawn from either Aristotelian Logic texts or modern physics texts, they would easily be able to discern the Aristotelian Logic from the obtuse mathematical phrasings. To them the smartphone loaded with Maxwell's texts, Jacksons Electrodynamics, Goldsteins Classical Mechanics etc. is talking "math".
The ability to recognize outlier writing style says nothing about content quality.
LLMs aren’t really great at this yet and I think the solution is, hopefully, that they improve. Anything else is accommodating a tool that should be accommodating the user.
Social media killed our attention span. Not, it is being tokenized.-
If you spend a long time with C++ code base you'll be able to decipher the otherwise-unreadable compiler errors pretty quickly, and I'd consider it a skill.
For example, even if you make thinking tokens literally just '....' (absolutely meaningless; zero information), you still see significant performance improvements: https://arxiv.org/abs/2404.15758 and https://arxiv.org/abs/2607.22925 for some starters.
Treat thinking more like a "loading screen message" that's been RL'd to somewhat resemble its actual internal state; which happens in its activations, not tokens.
Generally speaking yes, but actually no (just randomness is suboptimal, adding steps just to add steps is suboptimal). There is a mechanism working there (in having a CoT) that is not quite clear.
The task is to optimize the efficiency of CoT. Understanding that it is not a plain "chain of thought" is the start of the problem, the solution is not there yet.
If we had the solution, there would exist no overthinking - CoT would be optimal (lean and essential plus best results).
Conversely I've found that it can be as succinct as Muse Glimmer when it has a clear path forward. This can be either through well defined requirements or through unambiguous steps to take based on its own reasoning. While I do think it's fair to call out how much smaller model overthinks especially on one-shot prompts, in practice it hasn't led to an overall increase in time to task completion at least for what I've been using it for.
Even in the lower reasoning levels I find I want to like Qwen 3.8 27B and mostly don’t; it’s OK in the low reasoning effort, though.
Muse Glimmer is the one I actually enjoy working with, at least so far.
But I am trying to use it more as a sidekick than as a long horizon developer, because that is a better fit for how I want to use AI, and it appears to have been well trained for that.
As a result, qwen3.8 will churn over a prompt often for 5-10 minutes while gemma4 regularly finishes the same prompt in under 20 seconds, while giving a consistent and accurate response in my favorite test case. Qwen3.8, despite churning like that, often misses with an inaccurate answer.
Obviously, 'YMMV' depending on your use case... just sharing my two cents.
Ideally VRAM, but the benefit of the MoE design is better performance with unified memory since most of that RAM is not read for every single token. So you could potentially have the model loaded in CPU RAM, and let unified memory systems page the relevant chunks on demand to VRAM, or run on a fully unified memory system and be able to achieve good speeds even with the limited memory bandwidth most of them have.
But given it's only 6B, out of which only ~2.4B seem to be actually routed ("selected at random per token"), you could get reasonable performance with experts on CPU (still haven't tested, but 20-30 for dual channel DDR5 and 4 bpw quant).
This branch works now: https://github.com/unslothai/llama.cpp/tree/qwen4exp/qwen3.8...
or then[0]: https://unsloth.ai/docs/models/qwen3.8-next#qwen3.8-flash-ne...
0.01.033.250 I common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
0.01.033.261 I common_memory_breakdown_print: | - Host | 118186 = 106166 + 8898 + 3122 |
0.01.092.684 I common_params_fit_impl: projected to use 118186 MiB of host memory vs. 128855 MiB of total host memory
If that's true, it'd scale linearly with number of bits in the quant with an offset of about 51gigs. So Q4 should be a bit bigger than 82gigs, I'd guess in the 90s (as opposed to a ~280gig q4 if the whole 70gigs of the 1-bit quant scaled linearly).
This will almost certainly require changes to llama.cpp or vllm to do it right.
https://x.com/0xBakeer/status/2092694905978237224?s=20
Crazy how fast things move these days.
It's new arch demo for future Qwen 4 family, but (as I understand) training recipe/data is same as any other 3.8 model.
6B active params helps around the memory bandwidth constraints, but a 128GB box can probably run the Q3/Q4 quants fairly easily with a decent context size. This might actually be better for strix users than 27B, which was already very good.
It was tedious (a lot of "but I should..." output) but it got there and the result is a playable game.
Mind a bit blown.
Aside from the pelican, I am sort of impressed by the fact that things are going the way in terms of really impressive small models.
Also I love how this uses N-gram embedding. I think that Longcat was the first one who used it (I submitted that submission on hackernews because I really just loved the idea of it that I understood), I am certainly more interested in local LLM models and its interesting how they are utilizing new architectures to do some really impressive optimizations!
(Do note that I created it using a free rate limited end-point that I found on the huggingface space section: victor-chat-with-qwen3-8-flash-next.hf.space)
Wasn't it introduced by Gemma?
Didn’t see this mentioned yet. I wonder what this means for the effective size. It’s evidently ~176B paramètres, but how does that quantized. A 4-bit quant under 100GB seems unlikely, I’m suspecting this won’t run in 128GB unified memory
I’m principle I like the idea of trading more memory for compute though, even if there’s a memory shortage right now
Likely soon we'll see nvme offloading for ngrams as well. They're just an index, so that should be plenty fast for what it does. LLama.cpp support should come soon as well, and they might do some things with offloading first.
On branch of https://github.com/rdaum/eider (for DGX Spark). ~12 tok/sec decode without speculative decoding (will come later)
Still actively working on this. Prefill currently sucks. Will merge to main by end of day.
https://github.com/ggml-org/llama.cpp/pull/27742