Here's the thing: life also imitates art. If you invert your load-bearing assumption, it could be that he just reads too much slop. But my honest take? You might be right.
Have you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes
I have (192GB machine with two CPUs), pretty much does the trick. It just runs some small models used for embedding, etc. and has those on one CPU / memory node and all the Docker containers on the other one.c
I have a dual xeon also, same as OP: Ivy Bridge + 128GB DRAM, and was never really able to get decent LLM performance out of it. So I ended up biting the bullet and adding a "budget tier" A4000 20GB GPU. Too bad all my DRAM is wasted now--not sure if there is a way to take advantage of lots of DRAM once you move over to having inference happening on the GPU.
Have you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.
Yes and a 10 year old Xeon is going to be a v4 (not a v2 as in TFA) and it's going to have DDR4 ECC, not DDR3 ECC.
I've got a 14 cores / 28 threads Xeon from 2015 that I use as a server at home (ZFS / VMs).
It's really a sweet machine.
For ricing I've got a semi-recent AMD 7700X / DDR5 RAM (from 2023 ?) which is my main machine but the real deal is my old and trusty 10 years old Xeon server.
DDR4 ECC is pricey too atm but a 10 years old Xeon is basically free now.
A 20 cores / 40 threads costs maybe 20 USD (for just the CPU). Slap that in a $100 old HP Z440 workstation and you're good to go for quite a few workloads.
Mine is only on when I'm at my computer: it's not turned on 24/7 but more like 8/7 so the entire "but it consumes energy" point is moot.
If the local model is served via ollama, there's a default timeout of 10 minutes , which can be adjusted either per-call , or (as I did) in the systemd service environment variables
Thank you for your reply. I use LM studio (local server), but can switch to a different tool.
Do you know how to switch it in LM studio?
What I see is that: android studio gives "Error: stream failed" and in LM studio server I see it is still working, then says that client (=android studio) disconnected.
Dunno, I have not used either of those. (Had been using zed and ollama, and ollama had plenty of odd defaults that needed fixing)
Glancing through the docs, I would be digging down in the config of both Android studio and lm studio for either a TTL or jit auto evict setting, and if you find it, set it to some large number measured in hours?
Is it just me or does this post not mention how much RAM they had? I would love to know - I have a dual-Xeon 1U screamer with 96GB of DDR4 RDIMM just sitting around...
Such a system is RAM bandwidth limited and not compute limited Switching to q4 from q8 would decrease the amount of data needing to be loaded by half. The token generation rate would nearly double. But generally if you can do q6 or q8 and you have enough RAM you really should. Even if it's slower.
Token generation is nominally bandwidth limited. Prefill/prompt processing is nominally compute limited.
For CPU inference on old hardware I don't think q4 offers any benefit over q8 since the AVX unit doesn't support such small floats. I don't even think AVX supports 4-bit int math. IIRC AVX2 does.
I think I was just following along with the previous post about running Gemma on a Xeon. Next I’m going to see which model can give the highest tokens/sec
To me context means everything.
Tokens per second is a great metric but in the real world context window is the deal breaker when a real use case is on the table.
i have been optimizing for that. for now samosa is capped at using half of the avaiable cores and switching between them, which keeps the system 'less hot' as it would have been. i will also release better thermal control in the next release. at this point its basically sacrificing about 20% of the speed to keep the hardware less stressed (and hot).
I'm not lamenting that they aren't close, I'm saying Qwen will frequently output code that isn't even syntactically correct, even when the syntax is simple. Which makes it unusable for coding.
Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one.
It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.
We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?
But do you need to run every small problem through a 10B-30B model?
We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code.
Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for the end user.
I do about 10 google search queries for every 1 opus/gpt prompt. For google, I don't actually open pages anymore 9 out 10 times; I rely on the AI summary. It's fast and accurate; the trick is that you learn where the boundary is of what you can ask it. Querying information the small model is great at.
Then there might be slow, batch tasks. I can see myself getting 1T of slow RAM one day (in a few years?) and having a slow onsite GLM5.2 doing batch jobs that would be wasteful of my subscription limits, plus sensitive but boring things, such as bookeeping and general admin.
I'd like to to read all my email and al quarterly reporting. But that would have to be a good local model, probably a model simmilar to whatever google search uses, which seems just correct unless you throw serious challenges at it.
> do you need to run every small problem through a 10B-30B model? ... We're asking frontier Opus/Fable models to classify text
Actually probably yes: text analysis (magazine articles) by LLMs in the ~30b .. ~120b range failed miserably (and also randomly - the rare cases of proper interpretation* occurred among the failure cases) with the main public models of around one year ago, tried extensively.
So, yes, you can employ an ~80IQ only if you will expect the related quality.
The big hammers buy you more confidence and need less supervision. In pure task execution you _might_ smash the ant with a small surgical hammer, but if you absolutely need it smashed, that's when people still reach for the big hammer. It buys more confidence.
Its somewhat good, the prism team’s webgpu demo gave it a couple dozen “kernels” written in Fable 5 and it calls them for almost everything procedural
I feel like these things are experiencing convergent evolution to be like biological brains. The large parameters are merely potentially large parameters and they keep having more and more and smaller active layers, which are themselves quantized down. This is seems analogous to the chemical spiking of neurons and inactive layers of a brain in power and efficiency.
1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model.
2. quantization != native low precision training. a model trained in native ternary should generally outperform a full-precision model quantized after the fact.
even if a ternary model only retains 90-95% of the performance of its fp16 equivalent, who cares? if a 200b ternary model retains most of the capability of the 200b fp16 model while using a fraction of the memory and bandwidth, it can be substantially less efficient per parameter and still dominate a smaller fp16 model under the same hardware budget.
I know that's what the paper says the benchmarks say, but these models feel significantly worse than the base model when you start using them for real tasks.
Even the Q4 quant which they put in between their Bonsai models and the FP16 in the benchmarks has a tendency to go into doom loops and get lost compared to even Q5 or Q6.
I don't know how much of this is due to benchmaxxing (putting the benchmarks into the post-training loop) or cherry picking benchmarks to look good. If you spend a lot of time using local models you learn to take vendor provided benchmarks with a huge heap of doubt. Everything looks amazing in the benchmarks these days.
Models are not like a fps counter where losing a few percentage has no actual impact. These percentages may be the difference between a model that writes code and one that goes into loops, and think rm -rf . is a good idea.
There is a reason why most models try to stay in the FP4 or higher range, because the reduced accuracy can have major consequences.
You are better off with a 8b FP4+ model then a 27b Q2 model.
There’s a good eval floating around somewhere and tl;dr they’re awesome but the benchmarks are cooked, you’re better off with Qwen 8B Q4 than 27B 1b or ternary.
Thanks for being skeptical, I maintain a llama.cpp-based client and it’s frustrating how high expectations are for local AI bc the median effort level means people mostly assemble their expectations and understanding via marketing soundbites
I mostly agree with the prediction though maybe a bit more pessimistic about the timeline. Also I'm not sure our current usage of parameter count would make sense in this scenario, such a feat would require compressing current parameters in a manner much different then something producing a bit count per parameter. A hypothetical example would be using a single seed parameter per layer which then passed into a noise function produces the functional weights for that layer, able to reduce per weight size to sub bit levels (256 bit seed, producing 16K weights).
> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.
This prediction alone isn’t useful at all without a bound on speed and maybe quantization.
You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.
If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.
> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.
You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.
You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.
Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
>>Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
What are the LLM standards?
Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone might find this beneficial.
I have a free perplexity account from some promotion. Not sure what comparison you’re trying to make because Perplexity’s whole thing is that it’s really fast. It launches the search with parallel agents and then even seems to render some of the output paragraphs with parallel sessions to get the results.
Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and subsequent processing.
It wouldn’t compare in any meaningful way, because perplexity delivers instant results. That’s what I meant by modern standards of LLM usefulness.
Its really easy to argue against local models because when it comes to quality, you can argue using the tokens/sec. and when it comes to speed, you can argue using the parameter count. This is not compared to the frontier stuff but it is the frontier of last year that now runs on a local machine. It was impossible to do this last year.
the local models open source harnesses are really improving quite fast; just a few months ago i couldn't get any tool calling to work, and responses were very slow (thinking going on too long etc) but now with some newer models on my macbook air, tool calling works and depending on the model, it returns from thinking fairly swiftly....
>If you’re saying that hardware will catch up by mid-2027, I disagree.
If DDR6 comes out that will actually double the effective memory bandwidth on next generation computers. Most computers will reach a good fraction of a Strix Halo system's memory bandwidth and the next generation of Strix Halo will reach Macbook Pro levels of memory bandwidth. Of course there is no guarantee this will happen by 2027 but DDR6 will probably exist in some form by the end of 2027.
> You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.
On the contrary, that's ultimately an excellent baseline capability for most casual LLM users. Especially if you add the ability to fire off multiple requests over time and work on them concurrently (at least for not-very-long contexts), which is quite natural if one can implement some sort of continuous batching.
Even quantizing the model (with the ensuing loss in quality) is not an absolute requirement, quite unlike e.g. ensuring that a local model can run fully from VRAM on a typical consumer dGPU.
In general, ISTM that people overemphasize real-time or near real-time response which is a rather terrible fit if your goal is efficient LLM inference of near-SOTA models on typical local hardware.
Unless there are major improvements to how much hardware it takes to run a 1T model, this is deeply unrealistic. First because why release hardware that puts your biggest customers (data centers) out of business. Second because as I understand it the data centers have bought up all the high end chip production capacity for at least the next year and unless the bubble pops that'll continue for a while.
First off the math doesn’t math. Datacenters are willing to pay $50k for a single high end GPU. If you have unlimited capacity, yeah sell millions for $100 a pop or $10 a pop or whatever the bom cost of a phone GPU would be - but if you have limited capacity, you’re gonna sell all of that to the customer who is willing to pay the most PER UNIT.
Second off, this doesn’t work from a power consumption standpoint. When I run qwen3.6-35b, a far smaller model than you are suggesting, power usage spikes to 150-200W during inference. To fit a 1T model in the palm of my hand, the amount of processing required doesn’t fit the amount of power available.
Now I’m not saying this will never happen - there are some great leads, e.g. burning models directly on to a chip - but your scenario is definitely not happening in two years. Maybe 5, a lot more likely 10, unless of course local ai is made illegal
Even if the bubble pops and anthropic and openai et al implode - genie doesn’t go back in the bottle. The usefulness of LLMs for coding is proven, and a chip in a datacenter running 24/7 is always going to be more valuable than in a personal device running occasionally.
That doesn’t change until production capacity exceeds the datacenter demand. When that happens, they’ll start selling them down the market until it eventually reaches phones and toasters and whatever. But not in two years.
I agree it’s too small a benefit to justify the investment, and I agree the bubble will pop. I just don’t think that means hardware prices become sane again for quite a while. I think if you half the price of a server GPU because demand from the big ai companies drops out, we’ll still have a shortage - it’ll just being going into commodity data centers to run open weight models.
Absolutely. I'd be surprised if they couldn't 2x performance in the next year. Still doesn't make a 1T model fit on your phone.
> * Heavy quantization
I think this is a dead end if you're trying to fit a 1T model into a phone. Makes much more sense to train a model that's designed to be small, than train a model that's smart and then quantize it into stupidity.
> * Chips with hardcoded transformer architecture
Totally, this will probably work great. Now good luck booking fab time any time in the next 2 years.
> * Much cheaper HBM
Totally, this will probably work great. Now good luck booking fab time any time in the next two years.
> * Much sparser models - 1T total with ~1-10B active params e.g.
Fewer active params helps with the speed of token generation, but if the whole model doesn't fit into ram it doesn't solve the issue of having to constantly stream portions of the model from disk to ram.
> * Not to mention - 2 years of today's frontier models writing RTL and kernels at superhuman levels.
IMO this is a delusional myth-making idea being sold to us by ai companies. Machines that generate output based on statistical averages won't generate genuinely new ideas. They can help us try out ideas faster, but they're simply not capable of the kind of creativity and understanding required to push a field forward, except incrementally.
It will, but the process at this point is SSD bound rather than compute bound. On a bigger machine, Apple silicon must help but I don't have a bigger machine. I can think about this more and will make changes if that helps.
I sense the same. Running Ornith 35b with Pi and hitting 50+toks, and now that I learned Pi can do search and fetch, I have not needed to visit any of the large models as a search function. Soon we will be seeing new models drop this month and next that will change some of the landscape. Exciting times. P.s. Try Ornith, well worth it.
I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available
I also replaced my crap implementation of the oai server following yours, I hope it's all right. I did add a mention to your project and this fact in the README. It is a lot more barebones than what you have, but I have to admit that it works really nicely for me
I bought a trashcan Mac Pro on a whim last week ($120 in eBay!) and did some reading about them—it turns out people recently started using the GPUs to run Qwen4 @ 20-30 tok/s.
I'm excited to get my mitts on it on Friday when it finally arrives.
Here's some of the resources I came across if you're interested in reading.
oh very exciting! Thanks for sharing these sjs382! A bummer the models can't be run on both the GPUs and CPU so you could run much larger models. A year ago I was running gpt120b as it easily fit in 128gb ram. But now I'm only running gemma 26b and wishing I had stuck with 64gb ram since 128gb gets throttled. Not regretting buying 128gb 1.5 years ago though!
Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.
If we just take into account output token generation for simplicity. With 5tps u get 180k tokens an hour. That would costs around 0.05USD from an inference provider.
I estimate that the server consumes probably around 500W during inference.
In Germany where 1kwh cost around 0.3USD, 180k tokens inferred locally would therefore cost 0.15USD which is 3x the costs of using an inference provider.
But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.
The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.
Of course efficiency matters, but a lot of people either have cheap electricity or efficient hardware. My AMD strix halo home server can serve Gemma4-26B at like 70 TPS (rough estimate, I don’t remember the exact speed buts its fast af) while only using 100W.
I don't pay anywhere near .30usd in the US - I pay half that off peak and can buy 1000$ worth of batteries to load off on super off peak (0.11usd). Also the inference providers are fighting over market share with huge debt loads so they are def going to go up in price.
Inference costs will go down massively once they use the upcoming GPUs. I estimated that a model like GLM5.2 will be around 0.03USD/M output tokens in 2 years when the Feynman GPUs will be available in 2028. And this did not even consider architectural efficiency improvements. In mid 2027 we will already see a 10x reduction once everyone has switched to the Ruby architecture.
It will be feasible for everyone to have 20 different agents running at all times. A new world is coming
don't care, and yeah i don't like to hear it. we don't run local because it's cheaper money wise. we do it for freedom, for privacy and having option makes it cheaper in the long run. if there was no local options, your cloud model would cost much more!
I run qwen 27b at home when working it pulls around 400W. I get 40ish tokens per second generation and more importantly about 1000 tokens per second prompt processing.
In an hour it can process 3.6 million tokens or generate 144000 tokens. This costs me about 15 cents given my electricity prices.
For sonnet the token costs are 7.2 dollars for the prompt processing or 1.4 dollars for the generation. The cloud is 10x more expensive for generation and
close to 50 times more expensive for processing.
Try dropping the power cap on your GPU if it supports it; you can often get much lower energy usage with minimal loss of tok/s (particularly during generation) than whatever the GPU defaults to. There's a sweet spot around 200W on the GPU I'm currently testing that gives me about ~75% of the max pp and 97% of tg while using 100W less than the default/max 300W power cap -- and the card runs much quieter as a result.
The correct comparison is not sonnet, but qwen3.5-27b on a cloud. Alibaba's pricing [0] is $0.20/m input $1.56/m output, so $0.72 for the prompt processing or $0.22 for the generation. Yours is still cheaper but the margins are less.
My guess is that this math gets less good with MoE (because you will be limited by VRAM, but clouds won't).
From what I've seen, most inference providers are running at a loss, so it wouldn't be at all surprising if using their services costs less that running the same software locally.
The commodification of the hardware needed is probably a larger factor, because by the time a baseline computer has enough RAM and processing power to run a desired LLM, that hardware will be efficient enough that the extra electricity usage is nominal.
To this end, I've been thinking that it would be cool to create a solar-battery powered ("off grid") server providing a self-hosted LLM service. Offline when the sun doesn't shine enough (like Low Tech Magazine[1]). At whatever size is required for every day, community-scale use (a friend group, a street, a club). Fix the data centre issue by democratising AI to such an extent that we can bring it into the hands of communities to actually control (and democratically decide their own level of censorship / alignment). Along the lines of some of Geohotz' writing.
Within a short time I think open source models will all be getting good and efficient enough to make it viable to serve this on 2nd hand hardware for cheap. All it will take is a nerd in every small community to pool together a few hundred bucks initial outlay, and then ongoing costs are near free without electricity to pay for.
The transformer architecture is fundamentally unsuitable for local inference, while being efficient at scale. It's a fun experiment to try, but that's about it.
A dual Xeon of this era is probably pulling 300W or more when loaded.
At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.
If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.
That’s about $0.30 per million output tokens.
Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.
There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.
Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.
I love my little dual core X99 board with Xeon E5 2673 V3. It's not power efficient, but I just leave it in my basement for local Jupyter Notebook stuff. Much faster than everything cloud-based for a reasonably price at my scale.
148 comments
[ 0.23 ms ] story [ 69.4 ms ] threadI don't think the post itself reads like AI at all, but that's just me.
https://www.pangram.com/history/61fbc90e-180d-4d91-a85d-16ff...
https://www.pangram.com/history/61fbc90e-180d-4d91-a85d-16ff...
https://news.ycombinator.com/item?id=48354801
https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
A 10 year old Xeon is all you need
https://news.ycombinator.com/item?id=48353348
I've got a 14 cores / 28 threads Xeon from 2015 that I use as a server at home (ZFS / VMs).
It's really a sweet machine.
For ricing I've got a semi-recent AMD 7700X / DDR5 RAM (from 2023 ?) which is my main machine but the real deal is my old and trusty 10 years old Xeon server.
DDR4 ECC is pricey too atm but a 10 years old Xeon is basically free now.
A 20 cores / 40 threads costs maybe 20 USD (for just the CPU). Slap that in a $100 old HP Z440 workstation and you're good to go for quite a few workloads.
Mine is only on when I'm at my computer: it's not turned on 24/7 but more like 8/7 so the entire "but it consumes energy" point is moot.
Android studio connected to a local model disconnects automatiacally after 10 minutes. How set this limit to 12 hours or remove it completely?
I could run my LM studio model all night... but I cant, since Android studio times out after a hard limit of 10M.
This is not related to number of tokens. I do 130 sec.
https://docs.ollama.com/faq#how-do-i-keep-a-model-loaded-in-...
You didn't specify what was serving your local model.
Do you know how to switch it in LM studio?
What I see is that: android studio gives "Error: stream failed" and in LM studio server I see it is still working, then says that client (=android studio) disconnected.
So I assumed it was a setting in android studio.
Glancing through the docs, I would be digging down in the config of both Android studio and lm studio for either a TTL or jit auto evict setting, and if you find it, set it to some large number measured in hours?
https://developer.android.com/studio/gemini/use-a-local-mode...
https://lmstudio.ai/docs/developer/core/ttl-and-auto-evict
I found this one:
https://www.jetbrains.com/help/toolbox-app/remote-agent-time...
But it requires me to reinstall android studio because jetbrains toolbox cannot find it... because I installed ir on D:
On a side note Android studio atill creates hundred of megabutea of hidden stuff on C:
For CPU inference on old hardware I don't think q4 offers any benefit over q8 since the AVX unit doesn't support such small floats. I don't even think AVX supports 4-bit int math. IIRC AVX2 does.
Whether the writer's setup affects that choice I don't know.
I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat
This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
That is no where near decent at all.
It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.
But do you need to run every small problem through a 10B-30B model?
We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code.
Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for the end user.
Then there might be slow, batch tasks. I can see myself getting 1T of slow RAM one day (in a few years?) and having a slow onsite GLM5.2 doing batch jobs that would be wasteful of my subscription limits, plus sensitive but boring things, such as bookeeping and general admin.
I'd like to to read all my email and al quarterly reporting. But that would have to be a good local model, probably a model simmilar to whatever google search uses, which seems just correct unless you throw serious challenges at it.
Actually probably yes: text analysis (magazine articles) by LLMs in the ~30b .. ~120b range failed miserably (and also randomly - the rare cases of proper interpretation* occurred among the failure cases) with the main public models of around one year ago, tried extensively.
So, yes, you can employ an ~80IQ only if you will expect the related quality.
I feel like these things are experiencing convergent evolution to be like biological brains. The large parameters are merely potentially large parameters and they keep having more and more and smaller active layers, which are themselves quantized down. This is seems analogous to the chemical spiking of neurons and inactive layers of a brain in power and efficiency.
2. quantization != native low precision training. a model trained in native ternary should generally outperform a full-precision model quantized after the fact.
even if a ternary model only retains 90-95% of the performance of its fp16 equivalent, who cares? if a 200b ternary model retains most of the capability of the 200b fp16 model while using a fraction of the memory and bandwidth, it can be substantially less efficient per parameter and still dominate a smaller fp16 model under the same hardware budget.
I know that's what the paper says the benchmarks say, but these models feel significantly worse than the base model when you start using them for real tasks.
Even the Q4 quant which they put in between their Bonsai models and the FP16 in the benchmarks has a tendency to go into doom loops and get lost compared to even Q5 or Q6.
I don't know how much of this is due to benchmaxxing (putting the benchmarks into the post-training loop) or cherry picking benchmarks to look good. If you spend a lot of time using local models you learn to take vendor provided benchmarks with a huge heap of doubt. Everything looks amazing in the benchmarks these days.
There is a reason why most models try to stay in the FP4 or higher range, because the reduced accuracy can have major consequences.
You are better off with a 8b FP4+ model then a 27b Q2 model.
Thanks for being skeptical, I maintain a llama.cpp-based client and it’s frustrating how high expectations are for local AI bc the median effort level means people mostly assemble their expectations and understanding via marketing soundbites
https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit
This prediction alone isn’t useful at all without a bound on speed and maybe quantization.
You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.
If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.
> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.
You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.
You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.
Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
What are the LLM standards?
Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone might find this beneficial.
Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and subsequent processing.
It wouldn’t compare in any meaningful way, because perplexity delivers instant results. That’s what I meant by modern standards of LLM usefulness.
If DDR6 comes out that will actually double the effective memory bandwidth on next generation computers. Most computers will reach a good fraction of a Strix Halo system's memory bandwidth and the next generation of Strix Halo will reach Macbook Pro levels of memory bandwidth. Of course there is no guarantee this will happen by 2027 but DDR6 will probably exist in some form by the end of 2027.
On the contrary, that's ultimately an excellent baseline capability for most casual LLM users. Especially if you add the ability to fire off multiple requests over time and work on them concurrently (at least for not-very-long contexts), which is quite natural if one can implement some sort of continuous batching.
Even quantizing the model (with the ensuing loss in quality) is not an absolute requirement, quite unlike e.g. ensuring that a local model can run fully from VRAM on a typical consumer dGPU.
In general, ISTM that people overemphasize real-time or near real-time response which is a rather terrible fit if your goal is efficient LLM inference of near-SOTA models on typical local hardware.
Second off, this doesn’t work from a power consumption standpoint. When I run qwen3.6-35b, a far smaller model than you are suggesting, power usage spikes to 150-200W during inference. To fit a 1T model in the palm of my hand, the amount of processing required doesn’t fit the amount of power available.
Now I’m not saying this will never happen - there are some great leads, e.g. burning models directly on to a chip - but your scenario is definitely not happening in two years. Maybe 5, a lot more likely 10, unless of course local ai is made illegal
That doesn’t change until production capacity exceeds the datacenter demand. When that happens, they’ll start selling them down the market until it eventually reaches phones and toasters and whatever. But not in two years.
* Software inference optimizations
* Heavy quantization
* Chips with hardcoded transformer architecture
* Much cheaper HBM
* Much sparser models - 1T total with ~1-10B active params e.g.
* Not to mention - 2 years of today's frontier models writing RTL and kernels at superhuman levels.
Absolutely. I'd be surprised if they couldn't 2x performance in the next year. Still doesn't make a 1T model fit on your phone.
> * Heavy quantization
I think this is a dead end if you're trying to fit a 1T model into a phone. Makes much more sense to train a model that's designed to be small, than train a model that's smart and then quantize it into stupidity.
> * Chips with hardcoded transformer architecture
Totally, this will probably work great. Now good luck booking fab time any time in the next 2 years.
> * Much cheaper HBM
Totally, this will probably work great. Now good luck booking fab time any time in the next two years.
> * Much sparser models - 1T total with ~1-10B active params e.g.
Fewer active params helps with the speed of token generation, but if the whole model doesn't fit into ram it doesn't solve the issue of having to constantly stream portions of the model from disk to ram.
> * Not to mention - 2 years of today's frontier models writing RTL and kernels at superhuman levels.
IMO this is a delusional myth-making idea being sold to us by ai companies. Machines that generate output based on statistical averages won't generate genuinely new ideas. They can help us try out ideas faster, but they're simply not capable of the kind of creativity and understanding required to push a field forward, except incrementally.
I absolutely agree that models are going to advance on to “edge” hardware over the next few years by becoming small + specialized.
I am curious about the decision to not use GPU since this is for Apple Silicon.
Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?
Some throughput performance numbers (sequential and random) for RAM and SSD to put things into perspective would be awesome.
I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available
I also replaced my crap implementation of the oai server following yours, I hope it's all right. I did add a mention to your project and this fact in the README. It is a lot more barebones than what you have, but I have to admit that it works really nicely for me
I am getting [ Prompt: 91.3 t/s | Generation: 171.8 t/s ]
This is on a GPU (RTX 4060)
Is this decent?
I'm excited to get my mitts on it on Friday when it finally arrives.
Here's some of the resources I came across if you're interested in reading.
https://echalupa.com/blog/mac-pro-6-1-llama-cpp-firepro-d300...
https://matthewgribben.com/blog/mac-pro-6-1-llama-cpp-firepr...
If we just take into account output token generation for simplicity. With 5tps u get 180k tokens an hour. That would costs around 0.05USD from an inference provider.
I estimate that the server consumes probably around 500W during inference.
In Germany where 1kwh cost around 0.3USD, 180k tokens inferred locally would therefore cost 0.15USD which is 3x the costs of using an inference provider.
But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.
The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.
It will be feasible for everyone to have 20 different agents running at all times. A new world is coming
Maybe, but for how long? Prices keep going up, and every new model eats more and more tokens...
In an hour it can process 3.6 million tokens or generate 144000 tokens. This costs me about 15 cents given my electricity prices.
For sonnet the token costs are 7.2 dollars for the prompt processing or 1.4 dollars for the generation. The cloud is 10x more expensive for generation and close to 50 times more expensive for processing.
Try dropping the power cap on your GPU if it supports it; you can often get much lower energy usage with minimal loss of tok/s (particularly during generation) than whatever the GPU defaults to. There's a sweet spot around 200W on the GPU I'm currently testing that gives me about ~75% of the max pp and 97% of tg while using 100W less than the default/max 300W power cap -- and the card runs much quieter as a result.
My guess is that this math gets less good with MoE (because you will be limited by VRAM, but clouds won't).
[0]: https://openrouter.ai/qwen/qwen3.5-27b
The commodification of the hardware needed is probably a larger factor, because by the time a baseline computer has enough RAM and processing power to run a desired LLM, that hardware will be efficient enough that the extra electricity usage is nominal.
Within a short time I think open source models will all be getting good and efficient enough to make it viable to serve this on 2nd hand hardware for cheap. All it will take is a nerd in every small community to pool together a few hundred bucks initial outlay, and then ongoing costs are near free without electricity to pay for.
[1]: https://solar.lowtechmagazine.com/
At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.
If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.
That’s about $0.30 per million output tokens.
Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.
There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.
Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.